Affinity modulation of antibodies and antibody fragments of therapeutic value is often required in order to improve their clinical efficacies. Virtual affinity maturation has the potential to quickly focus on the critical hotspot residues without the combinatorial explosion problem of conventional display and library approaches. However, this requires a binding affinity scoring function that is capable of ranking single-point mutations of a starting antibody. We focus here on assessing the solvated interaction energy (SIE) function that was originally developed for and is widely applied to scoring of protein-ligand binding affinities. To this end, we assembled a structure-function data set called Single-Point Mutant Antibody Binding (SiPMAB) comprising several antibody-antigen systems suitable for this assessment, i.e., based on high-resolution crystal structures for the parent antibodies and coupled with high-quality binding affinity measurements for sets of single-point antibody mutants in each system. Using this data set, we tested the SIE function with several mutation protocols based on the popular methods SCWRL, Rosetta, and FoldX. We found that the SIE function coupled with a protocol limited to sampling only the mutated side chain can reasonably predict relative binding affinities with a Spearman rank-order correlation coefficient of about 0.6, outperforming more aggressive sampling protocols. Importantly, this performance is maintained for each of the seven system-specific component subsets as well as for other relevant subsets including non-alanine and charge-altering mutations. The transferability and enrichment in affinity-improving mutants can be further enhanced using consensus ranking over multiple methods, including the SIE, Talaris, and FOLDEF energy functions. The knowledge gained from this study can lead to successful prospective applications of virtual affinity maturation.