Altering the local symmetry of an activator by lattice tuning is considered an effective strategy to optimize the luminescence performance of phosphors. Herein, the novel Mn4+-activated double perovskite phosphor La1.67MgTaO6 (LMTO) with A-site defects was successfully prepared. Benefiting from the random occupation of the nearest A-site by cations and vacancies, the distorted [MnO6] octahedra lack the inversion center. The LMTO:0.4 mol%Mn4+ phosphor has a significant zero-phonon line intensity with a high internal quantum efficiency (IQE) value of 62.97% after the Laporte selection rule is broken. Furthermore, local lattice tuning was performed by increasing the randomness of the A-site and the distortion index of the [MnO6] octahedron by co-doping with Ca2+ or Sr2+ ions. Importantly, the IQE value of the samples was enhanced from 62.97% to 72.65% and the activation energy increased from 0.497 eV to 0.548 eV, which can be well applied in the fields of plant cultivation and warm white light-emitting diodes. These studies provide valid fundamental insights for the selection of excellent luminescent matrices to obtain efficient Mn4+-activated phosphors via local lattice tuning.