Proper pain therapy requires adequate pain assessment. This study evaluated the reliability and validity of the Unesp-Botucatu horse acute pain scale (UHAPS), the Orthopedic Composite Pain Scale (CPS) and unidimensional scales in horses admitted for orthopedic and soft tissue surgery. Forty-two horses were assessed and videotaped before surgery, up to 4 hours postoperatively, up to 3 hours after analgesic treatment, and 24 hours postoperatively (168 video clips). After six evaluators viewing each edited video clip twice in random order at a 20-day interval, they chose whether analgesia would be indicated and applied the Simple Descriptive, Numeric and Visual Analog scales, CPS, and UHAPS. For all evaluators, intra-observer reliability of UHAPS and CPS ranged from 0.70 to 0.97. Reproducibility was variable among the evaluators and ranged from poor to very good for all scales. Principal component analysis showed a weak association among 50% and 62% of the UHAPS and CPS items, respectively. Criterion validity based on Spearman correlation among all scales was above 0.67. Internal consistency was minimally acceptable (0.51-0.64). Item-total correlation was acceptable (0.3-0.7) for 50% and 38% of UHAPS and CPS items, respectively. UHAPS and CPS were specific (90% and 79% respectively), but both were not sensitive (43 and 38%, respectively). Construct validity (responsiveness) was confirmed for all scales because pain scores increased after surgery. The cut-off point for rescue analgesia was ≥ 5 and ≥ 7 for the UHAPS and CPS, respectively. All scales presented adequate repeatability, criterion validity, and partial responsiveness. Both composite scales showed poor association among items, minimally acceptable internal consistency, and weak sensitivity, indicating that they are suboptimal instruments for assessing postoperative pain. Both composite scales require further refinement with the exclusion of redundant or needless items and reduction of their maximum score applied to each item or should be replaced by other tools.