Here, we aim to investigate the independent and combined associations of serum 25-hydroxyvitamin D (25(OH)D) and cardiorespiratory fitness (CRF) with glucose metabolism. Fasting blood samples of 107 men aged 40-79 years were analyzed for 25(OH)D, glucose, insulin, glycated hemoglobin, and lipid profile. Homeostasis model assessment of insulin resistance index (HOMA-IR) was calculated from the fasting concentrations of glucose and insulin. Visceral fat area (VFA) was determined by magnetic resonance imaging and CRF by measuring maximal oxygen uptake. Median 25(OH)D concentration was 36.3 nmol/L, while the prevalence of 25(OH)D deficiency was 74.8%. Participants with high CRF had significantly lower HOMA-IR, glycated hemoglobin, and insulin values than participants with low CRF (p < 0.05). Higher 25(OH)D concentration was strongly correlated with lower HOMA-IR and insulin values independent of VFA (p < 0.01) but significantly affected by CRF. In the high CRF group, participants with higher 25(OH)D concentration had lower HOMA-IR values than participants with low 25(OH)D concentration (p < 0.05). Higher 25(OH)D and CRF are crucial for reducing insulin resistance regardless of abdominal fat. In addition, higher 25(OH)D concentration may strengthen the effect of CRF on reducing insulin resistance in middle-aged and elderly Japanese men with high CRF.