Agricultural usage of phosphorus (P) is largely driven by the amount of P removed from fields in harvested plant matter as offtake needs to be balanced by P fertilizer application. Reducing P concentration in grains is a way to decrease P offtake and reduce P fertilizer requirements or soil P mining where insufficient P is applied. Our objective was to assesses the genotypic variation for grain P concentration present within the rice gene pool and resolve to what extent it is affected by environment (P supply) or associated with genetic factors. About 2-fold variation in grain P concentrations were detected in two rice diversity panels, however, environmental effects were stronger than genotype effects. Genome wide association studies identified several putative loci associated with grain P concentrations. In most cases this was caused by minor haplotype associations with high grain P concentrations while associations with reduced P concentrations were identified on chromosomes 1, 6, 8, 11 and 12. Only the latter type of locus is of interest in breeding for reduced P concentrations and the most promising locus was at 20.7 Mb on chromosome 8, where a rare haplotype that was absent from all modern varieties studied reduced grain P concentration by 9.3%. This and all other loci were not consistently detected across environments or association panels, confirming that genetic effects were small compared to effects of environment. We conclude that the genetic effects detected were not sufficiently large or consistent to be of utility in plant breeding. Instead breeding efforts may have to rely on small to medium effect mutants already identified and attempt to achieve a more pronounced reduction in grain P concentration through the introgression of these mutants into a single genetic background.