Vitamin D receptor (VDR) gene polymorphisms may influence risk for adenomatous polyps (AP), a benign precursor to colon cancer, via modulation of vitamin D sensitive pathways, including cell proliferation and differentiation. However, results have been mixed and any association remains contentious. Failure to clinically exclude the presence of (AP in control cohorts may contribute to the lack of consensus. Therefore, we assessed the role of the FokI, BsmI, ApaI, and TaqI VDR polymorphisms in modifying risk for AP, adjusting for a range of dietary and lifestyle variables. Blood was collected from colonoscopy patients (n = 258) and VDR polymorphisms assessed by restriction fragment length polymorphism. Dietary habits were estimated from food frequency questionnaires. Odds ratios for AP were calculated by genotype, stratified by sex, and adjusted for age, lifestyle, and dietary factors. FokI was associated with modified risk for AP in males, whereas the BsmI/ApaI/TaqI haplotype was associated with modified risk in females. No interaction was found between VDR variants and vitamin D intake. This study offers novel insight into the potential for VDR genetics to contribute to risk for AP and is the first to demonstrate a sex-specific relationship between these polymorphisms and risk for AP.