Xeroderma pigmentosum (XP) is a rare autosomal recessive disease that is associated with a severe deficiency in nucleotide excision repair. Genetic polymorphisms in XP genes may be associated with a change in DNA repair capacity, which could be associated with colorectal cancer development. We assessed the association between 94 single nucleotide polymorphisms (SNPs) within seven XP genes (XPA-XPG) and the colorectal cancer risk in the Polish population. We genotyped 758 unselected patients with colorectal cancer and 1,841 healthy adults. We found that a significantly decreased risk of colorectal cancer was associated with XPC polymorphism rs2228000_CT genotype (OR 0.59; p < 0.0001) and the rs2228000_TT genotype (OR 0.29; p < 0.0001) compared to the reference genotype (CC). And an increased disease risk was associated with the XPD SNP, rs1799793_AG genotype (OR 1.44, p = 0.018) and rs1799793_AA genotype (OR 3.31, p < 0.0001) compared to the reference genotype. Haplotype analysis within XPC, XPD and XPG revealed haplotypes associated with an altered colorectal cancer risk. Stratified analysis by gender showed differences between the association of three SNPs: XPC rs2228000, XPD rs1799793 and XPD rs238406 in females and males. Association analysis between age of disease onset and polymorphisms in XPD (rs1799793) and XPC (rs2228000) revealed differences in the prevalence of these variants in patients under and over 50 years of age. Our results confirmed that polymorphisms in XPC and XPD may be associated with the risk of colorectal cancer.