Many of the measurements used in sport psychology research are arbitrary metrics, and researchers often cannot make the jump from scores on paper-and-pencil tests to what those scores actually mean in terms of real-world behaviors. Effect sizes for behavioral data are often interpretable, but the meaning of a small, medium, or large effect for an arbitrary metric is elusive. We reviewed all the issues in the 2005 volumes of the Journal of Sport & Exercise Psychology, The Sport Psychologist, and the Journal of Applied Sport Psychology to determine whether the arbitrary metrics used in sport psychology research were interpreted, or calibrated, against real-world variables. Of the 54 studies that used quantitative methods, 25 reported only paper-and-pencil arbitrary metrics with no connections to behavior or other real-world variables. Also, 44 of the 54 studies reported effect sizes, but only 7 studies, using both arbitrary and behavioral metrics, had calculated effect indicators and interpreted them in terms of real-world meaning.