The interplay of genetic and environmental factors behind cognitive development has preoccupied multiple fields of science and sparked heated debates over the decades. Here we tested the hypothesis that developmental genes rely heavily on cognitive challenges-as opposed to natural maturation. Starting with a polygenic score (cogPGS) that previously explained variation in cognitive performance in adults, we estimated its effect in 344 children and adolescents (mean age of 12 years old, ranging from 6 to 25) who showed changes in working memory (WM) in two distinct samples: (1) a developmental sample showing significant WM gains after 2 years of typical, age-related development, and (2) a training sample showing significant, experimentally-induced WM gains after 25 days of an intense WM training. We found that the same genetic factor, cogPGS, significantly explained the amount of WM gain in both samples. And there was no interaction of cogPGS with sample, suggesting that those genetic factors are neutral to whether the WM gains came from development or training. These results represent evidence that cognitive challenges are a central piece in the gene-environment interplay during cognitive development. We believe our study sheds new light on previous findings of interindividual differences in education (rich-get-richer and compensation effects), brain plasticity in children, and the heritability increase of intelligence across the lifespan.