Levels of sex differences for human body size and shape phenotypes are hypothesized to have adaptively reduced following the agricultural transition as part of an evolutionary response to relatively more equal divisions of labor and new technology adoption. In this study, we tested this hypothesis by studying genetic variants associated with five sexually differentiated human phenotypes: height, body mass, hip circumference, body fat percentage, and waist circumference. We first analyzed genome-wide association (GWAS) results for UK Biobank individuals (~194,000 females and ~167,000 males) to identify a total of 114,199 single nucleotide polymorphisms (SNPs) significantly associated with at least one of the studied phenotypes in females, males, or both sexes (P<5x10-8). From these loci we then identified 3,016 SNPs (2.6%) with significant differences in the strength of association between the female- and male-specific GWAS results at a low false-discovery rate (FDR<0.001). Genes with known roles in sexual differentiation are significantly enriched for co-localization with one or more of these SNPs versus SNPs associated with the phenotypes generally but not with sex differences (2.73-fold enrichment; permutation test; P = 0.0041). We also confirmed that the identified variants are disproportionately associated with greater phenotype effect sizes in the sex with the stronger association value. We then used the singleton density score statistic, which quantifies recent (within the last ~3,000 years; post-agriculture adoption in Britain) changes in the frequencies of alleles underlying polygenic traits, to identify a signature of recent positive selection on alleles associated with greater body fat percentage in females (permutation test; P = 0.0038; FDR = 0.0380), directionally opposite to that predicted by the sex differences reduction hypothesis. Otherwise, we found no evidence of positive selection for sex difference-associated alleles for any other trait. Overall, our results challenge the longstanding hypothesis that sex differences adaptively decreased following subsistence transitions from hunting and gathering to agriculture.