Low-toxicity InP-based quantum dots (QDs) exhibit potential for replacing Cd/Pb-containing QDs in the visible and near-infrared regions. Despite advancements, further improvement relies on synthesizing homogeneous InP QDs to achieve a high color purity. In a commonly employed two-step "seed-mediated" synthetic approach, we demonstrate the high sensitivity of InP seed sizes and size distribution to the quantities of trioctylphosphine (TOP) and tris(trimethylsilyl)phosphine [(TMS)3P], attributed to the process of "self-focusing of size distribution" and enhanced reactivity of In-oleate through coordination with TOP. During growth, the processes of size focusing and defocusing are modulated by the accumulation of oleic acid and TOP molecules, as well as the amount of (TMS)3P in the growth precursor, which may relate to the dissolution process of InP magic size clusters. Through precise control, the best valley/depth ratio of InP QDs was 0.52 at the first absorption peak at 571 nm, resulting in luminescence with a full width at half-maximum of 35 at 620 nm with an absolute photoluminescence quantum yield around 90% after heteroepitaxial growth with ZnSe and ZnS shells.