Accurate prediction of the pKa's of protein residues is crucial to many applications in biological simulation and drug discovery. Here, we present the use of free energy perturbation (FEP) calculations for the prediction of single-protein residue pKa values. We begin with an initial set of 191 residues with experimentally determined pKa values. To isolate sampling limitations from force field inaccuracies, we develop an algorithm to classify residues whose environments are significantly affected by crystal packing effects. We then report an approach to identify buried histidines that require significant sampling beyond what is achieved in typical FEP calculations. We therefore define a clean data set not requiring algorithms capable of predicting major conformational changes on which other pKa prediction methods can be tested. On this data set, we report an RMSE of 0.76 pKa units for 35 ASP residues, 0.51 pKa units for 44 GLU residues, and 0.67 pKa units for 76 HIS residues.