In this study, we generated a matched molecular pair dataset of halogen/deshalogen compounds with reliable binding affinity data and structural binding mode information from public databases. The workflow includes automated system preparation and setup of free energy perturbation relative binding free energy calculations. We demonstrate the suitability of these datasets to investigate the performance of molecular mechanics force fields and molecular simulation algorithms for the purpose of in silico affinity predictions in lead optimization. Our datasets of a total of 115 matched molecular pairs show highly accurate binding free energy predictions with an average error of <1 kcal/mol despite the semi-automated calculation scheme. We quantify the accuracy of the optimized potential for liquid simulations (OPLS) force field to predict the effect of halogen addition to compounds, a commonly employed chemical modification in the design of drug-like molecules.