Cholesterol is a central building block in biomembranes, where it induces orientational order, slows diffusion, renders the membrane stiffer, and drives domain formation. Molecular dynamics (MD) simulations have played a crucial role in resolving these effects at the molecular level; yet, it has recently become evident that different MD force fields predict quantitatively different behavior. Although easily neglected, identifying such limitations is increasingly important as the field rapidly progresses toward simulations of complex membranes mimicking the in vivo conditions: pertinent multicomponent simulations must capture accurately the interactions between their fundamental building blocks, such as phospholipids and cholesterol. Here, we define quantitative quality measures for simulations of binary lipid mixtures in membranes against the C-H bond order parameters and lateral diffusion coefficients from NMR spectroscopy as well as the form factors from X-ray scattering. Based on these measures, we perform a systematic evaluation of the ability of commonly used force fields to describe the structure and dynamics of binary mixtures of palmitoyloleoylphosphatidylcholine (POPC) and cholesterol. None of the tested force fields clearly outperforms the others across the tested properties and conditions. Still, the Slipids parameters provide the best overall performance in our tests, especially when dynamic properties are included in the evaluation. The quality evaluation metrics introduced in this work will, particularly, foster future force field development and refinement for multicomponent membranes using automated approaches.