We examined whether birth weight (BW) predicts changes in body composition over a 6-year period in Swedish children and adolescents. For this purpose, a total of 247 children (55.5% girls) and 162 adolescents (60.5% girls) were included in the study and were followed up 6 years later. BW was obtained from parental records. We measured weight, height, waist circumference, and the bicep, tricep, subscapular, suprailiac, and medial calf skinfolds, and we calculated BMI, fat-free mass (FFM), and the sum of five skinfolds. Physical activity was assessed by accelerometry. Changes in pubertal status and baseline anthropometric estimates were used as confounders in all analysis. In the children cohort, we observed that BW was inversely associated with changes in BMI (β = -0.736, P = 0.002) and the sum of five skinfolds (β = -6.381, P = 0.009) regardless of confounders and physical activity, only in girls. We did not find any significant association between BW and adiposity gain estimates in the adolescent cohort. These findings give further support to the concept that low BW may have a programming effect of subsequent adiposity gain from childhood to adolescence. We also confirm the sex-related differences in the programming effect of body composition.