To compare the outcome of low-dose rate brachytherapy (LDR-BT) and image-guided intensity-modulated radiotherapy (IG-IMRT) for localized prostate cancer, we examined 488 LDR-BT and 269 IG-IMRT patients. IG-IMRT treated older and advanced disease with more hormonal therapy than LDR-BT, which excluded T3b-T4 tumor and initial PSA > 50 ng/ml. The actuarial five-year biochemical failure-free survival rate was 88.7% and 96.7% (p = 0.0003) in IG-IMRT and LDR-BT, respectively; it was 88.2% (85.1% for IG-IMRT and 94.9% for LDR-BT, p = 0.0578) for the high-risk group, 95.2% (91.6% and 97.0%, p = 0.3361) for the intermediate IG-IMRT and 96.8% (95.7% and 97%, p = 0.8625) for the low-risk group. Inverse probability of treatment weighting (IPTW) involving propensity scores was used to reduce background selection bias. IPTW showed a statistically significant difference between LDR-BT and IG-IMRT in high risk (p = 0.0009) and high risk excluding T3-4/initial PSA > 50 ng/ml group (p = 0.0073). IG-IMRT showed more gastrointestinal toxicity (p = 0.0023) and less genitourinary toxicity (p < 0.0001) than LDR-BT. LDR-BT and IG-IMRT showed equivocal outcome in low- and intermediate-risk groups. For selected high-risk patients, LDR-BT showed more potential to improve PSA control rate than IG-IMRT.