This study assessed the impact of individual social risk factor variables and social determinants of health (SDOH) measures on hospital readmission rates and penalties used in the Centers for Medicare and Medicaid Services (CMS) Hospital Readmissions Reduction Program (HRRP). Using 2012-16 hospital discharge data from New York City, we projected HRRP penalties by augmenting CMS's readmission model for heart attack, heart failure, and pneumonia with SDOH scores constructed at each of four geographic levels and a measure of individual-level social risk. Including additional SDOH scores in the model, especially those constructed with the most granular geographic data, along with social risk factor variables substantially affects projected penalties for hospitals treating the highest proportion of patients with high SDOH scores. Improved performance occurred even after we included peer-group stratification in the HRRP model pursuant to the 21st Century Cures Act. Small improvements in model accuracy were associated with substantial shifts in projected performance. Our results suggest that CMS's continued omission of relevant patient and geographic data from the HRRP readmission model misallocates penalties attributable to SDOH and social risk factor effects to hospitals with the largest share of high-risk patients.