Optical coherence tomography (OCT) is being investigated in breast cancer diagnostics as a real-time histology evaluation tool. We present a customized deep convolutional neural network (CNN) for classification of breast tissues in OCT B-scans. Images of human breast samples from mastectomies and breast reductions were acquired using a custom ultrahigh-resolution OCT system with 2.72 µm axial resolution and 5.52 µm lateral resolution. The network achieved 96.7% accuracy, 92% sensitivity, and 99.7% specificity on a dataset of 23 patients. The usage of deep learning will be important for the practical integration of OCT into clinical practice.