Deep learning models for variant pathogenicity prediction can recapitulate expert-curated annotations, but their performance remains unexplored on actual disease phenotypes in a real-world setting. Here, we apply three state-of-the-art pathogenicity prediction models to classify hereditary breast cancer gene variants in the UK Biobank. Predicted pathogenic variants in BRCA1, BRCA2 and PALB2, but not ATM and CHEK2, were associated with increased breast cancer risk. We explored gene-specific score thresholds for variant pathogenicity, finding that they could improve model performance. However, when specifically tasked with classifying variants of uncertain significance, the deep learning models were generally of limited clinical utility.