The computational prediction of relative binding free energies is a crucial goal for drug discovery, and G protein-coupled receptors (GPCRs) are arguably the most important drug target class. However, they present increased complexity to model compared to soluble globular proteins. Despite breakthroughs, experimental X-ray crystal and cryo-EM structures are challenging to attain, meaning computational models of the receptor and ligand binding mode are sometimes necessary. This leads to uncertainty in understanding ligand-protein binding induced changes such as, water positioning and displacement, side chain positioning, hydrogen bond networks, and the overall structure of the hydration shell around the ligand and protein. In other words, the very elements that define structure activity relationships (SARs) and are crucial for accurate binding free energy calculations are typically more uncertain for GPCRs. In this work we use free energy perturbation (FEP) to predict the relative binding free energies for ligands of two different GPCRs. We pinpoint the key aspects for success such as the important role of key water molecules, amino acid ionization states, and the benefit of equilibration with specific ligands. Initial calculations following typical FEP setup and execution protocols delivered no correlation with experiment, but we show how results are improved in a logical and systematic way. This approach gave, in the best cases, a coefficient of determination (R2) compared with experiment in the range of 0.6-0.9 and mean unsigned errors compared to experiment of 0.6-0.7 kcal/mol. We anticipate that our findings will be applicable to other difficult-to-model protein ligand data sets and be of wide interest for the community to continue improving FE binding energy predictions.