In brain imaging research, it is becoming standard practice to remove the face from the individual's 3D structural MRI scan to ensure data privacy standards are met. Face removal - or 'defacing' - is being advocated for large, multi-site studies where data is transferred across geographically diverse sites. Several methods have been developed to limit the loss of important brain data by accurately and precisely removing non-brain facial tissue. At the same time, deep learning methods such as convolutional neural networks (CNNs) are increasingly being used in medical imaging research for diagnostic classification and prognosis in neurological diseases. These neural networks train predictive models based on patterns in large numbers of images. Because of this, defacing scans could remove informative data. Here, we evaluated 4 popular defacing methods to identify the effects of defacing on 'brain age' prediction - a common benchmarking task of predicting a subject's chronological age from their 3D T1-weighted brain MRI. We compared brain-age calculations using defaced MRIs to those that were directly brain extracted, and those with both brain and face. Significant differences were present when comparing average per-subject error rates between algorithms in both the defaced brain data and the extracted facial tissue. Results also indicated brain age accuracy depends on defacing and the choice of algorithm. In a secondary analysis, we also examined how well comparable CNNs could predict chronological age from the facial region only (the extracted portion of the defaced image), as well as visualize areas of importance in facial tissue for predictive tasks using CNNs. We obtained better performance in age prediction when using the extracted face portion alone than images of the brain, suggesting the need for caution when defacing methods are used in medical image analysis.