Purpose To overcome methodology limitations for studying auditory development in young children, we have recently developed an observer-based procedure that uses a conditioned, play-based, motor response (see Bonino & Leibold, 2017). The purpose of this article was to examine interrater reliability for the method. Method Video recordings of test sessions of 2- to 4-year-old children (n = 17) were examined. Detection of a 1000-Hz warble tone was measured with the Play Observer-Based, Two-Interval (PlayO2I) method in each of two conditions: for a fixed intensity level (30 dB SPL) or for a variable intensity level signal (0-30 dB SPL). All test sessions were scored independently by three observers (one real-time, two offline). Observer consensus was evaluated with Fleiss' kappa statistic. To determine if summary data were similar across the observers of each test session, the proportion of correct trials (fixed-level condition) or threshold (variable-level condition) were computed. Results The strength of observer consensus was classified as "almost perfect" and "substantial" for the fixed-level and variable-level conditions, respectively. Follow-up analysis of the variable-level data indicated that differences in observer consensus were seen based on the signal level, the type of response behavior provided by the child, and the confidence level of the real-time observer. Resulting summary data were similar across the three observers of each test session: no significant differences for estimates of the proportion of correct trials or threshold. Conclusions Results from this study confirm strong interrater reliability for the method. The PlayO2I method is a powerful tool for measuring detection and discrimination abilities in young children. Supplemental Material https://doi.org/10.23641/asha.12978197.