This study investigates the photoluminescence for self-assembled InAs quantum dots embedded in photonic crystal nanocavities as two of the air holes nearest the H1 cavity were shifted. A rapid decrease of resonant wavelength and quality factor for the cavity modes, in which the electric field patterns extended in the shifting direction, were found as the shift increased from 0.2 to 0.4 lattice constants. This phenomenon is interpreted as being caused by the formation of two point defects between the nearest and second nearest air holes.