Probabilistic seismic hazard assessments (PSHAs) provide the scientific basis for building codes to reduce damage from earthquakes. Despite their substantial impact, little is known about how well PSHA predicts actual shaking. Recent PSHA for California, Japan, Italy, Nepal, and France appear to consistently overpredict historically observed earthquake shaking intensities. Numerical simulations show that observed shaking is equally likely to be above or below predictions. This result from independently developed models and datasets in different countries and tectonic settings indicates possible systematic bias in the hazard models, the observations, or both. Analysis of possible causes shows that much of the discrepancy is due to a subtle and rarely considered issue: the conversion equations used in comparing the models-which forecast shaking as peak ground acceleration or velocity-and observations-parameterizations of qualitative shaking reports. Historical shaking reports fill a crucial data gap, but more research is warranted on how qualitative observations relate to instrumental shaking measures for earthquakes.