Adjoint variable method in combination with gradient descent optimization has been widely used for the inverse design of nanophotonic devices. In many of such optimizations, the design region is only a small fraction of the total computational domain. Here we show that the adjoint variable method can be combined with the Schur complement domain decomposition method. With this combination, in each optimization step, the simulation only involves the degrees of freedom that are inside the design region. Our approach should significantly improve the computational efficiency of adjoint variable method based optimization of photonic structures.