Polymorphisms contribute to inter-individual differences and show a promising predictive role for chemotherapy-related toxicity in colon cancer (CC). TOSCA is a multicentre, randomized, non-inferiority, phase III study conducted in high-risk stage II/stage III CC patients treated with 6 vs 3 months of FOLFOX-4 or XELOX adjuvant chemotherapy. During this post-hoc analysis, 218 women and 294 men were genotyped for 17 polymorphisms: TYMS (rs34743033, rs2853542, rs11280056), MTHFR (rs1801133, rs1801131), ERCC1 (rs11615), XRCC1 (rs25487), XRCC3 (rs861539), XPD (rs1799793, rs13181), GSTP1 (rs1695), GSTT1/GSTM1 (deletion +/-), ABCC1 (rs2074087), and ABCC2 (rs3740066, rs1885301, rs4148386). The aim was to assess the interaction between these polymorphisms and sex, on safety in terms of time to grade ≥3 haematological (TTH), grade ≥3 gastrointestinal (TTG) and grade ≥2 neurological (TTN) toxicity. Interactions were detected on TTH for rs1801133 and rs1799793, on TTG for rs13181 and on TTN for rs11615. Rs1799793 GA genotype (p = 0.006) and A allele (p = 0.009) shortened TTH in men. In women, the rs11615 CC genotype worsened TTN (co-dominant model p = 0.008, recessive model p = 0.003) and rs13181 G allele improved the TTG (p = 0.039). Differences between the two sexes in genotype distribution of rs1885301 (p = 0.020) and rs4148386 (p = 0.005) were found. We highlight that polymorphisms could be sex-specific biomarkers. These results, however, need to be confirmed in additional series.