A machine learning approach to graduate admissions and the role of letters of recommendation

Abstract

The graduate admissions process is time-consuming, subjective, and complicated by the need to combine information from diverse data sources. Letters of recommendation (LORs) are particularly difficult to evaluate and it is unclear how much impact they have on admissions decisions. This study addresses these concerns by building machine learning models to predict admissions decisions for two STEM graduate programs, with a focus on examining the contribution of LORs in the decision-making process. We train our predictive models leveraging information extracted from structured application forms (e.g., undergraduate GPA, standardized test scores, etc.), applicants' resumes, and LORs. A particular challenge in our study is the different modalities of application data (i.e., text vs. structured forms). To address this issue, we converted the textual LORs into features using a commercial natural language processing product and a manual rating process that we developed. By analyzing the predictive performance of the models using different subsets of features, we show that LORs alone provide only modest, but useful, predictive signals to admission decisions; the best model for predicting admissions decisions utilized both LOR and non-LOR data and achieved 89% accuracy. Our experiments demonstrate promising results in the utility of automated systems for assisting with graduate admission decisions. The findings confirm the value of LORs and the effectiveness of our feature engineering methods from LOR text. This study also assesses the significance of individual features using the SHAP method, thereby providing insight into key factors affecting graduate admission decisions.