Owing to the increase of available computational capabilities and the potential for providing a more accurate description, polarizable molecular dynamics force fields are gaining popularity in modeling biomolecular systems. It is, however, crucial to evaluate how much precision is truly gained with increasing cost and complexity of the simulation. Here, we leverage the NMRlipids open collaboration and Databank to assess the performance of available polarizable lipid models─the CHARMM-Drude and the AMOEBA-based parameters─against high-fidelity experimental data and compare them to the top-performing nonpolarizable models. While some improvement in the description of ion binding to membranes is observed in the most recent CHARMM-Drude parameters, and the conformational dynamics of AMOEBA-based parameters are excellent, the best nonpolarizable models tend to outperform their polarizable counterparts for each property we explored. The identified shortcomings range from inaccuracies in describing the conformational space of lipids to excessively slow conformational dynamics. Our results provide valuable insights for the further refinement of polarizable lipid force fields and for selecting the best simulation parameters for specific applications.