Development and Initial Psychometrics of a Generic Treatment Integrity Measure Designed to Assess Practice Elements Targeting Social, Emotional, and Behavioral Outcomes in Early Childhood Settings

Bryce D McLeod; Kevin S Sutherland; Michael Broda; Kristen L Granger; Ruben G Martinez; Maureen A Conroy; Patricia A Snyder; Michael A Southam-Gerow

doi:10.1007/s11121-021-01304-w

Development and Initial Psychometrics of a Generic Treatment Integrity Measure Designed to Assess Practice Elements Targeting Social, Emotional, and Behavioral Outcomes in Early Childhood Settings

Prev Sci. 2022 May;23(4):488-501. doi: 10.1007/s11121-021-01304-w. Epub 2021 Oct 29.

Authors

Bryce D McLeod¹, Kevin S Sutherland², Michael Broda², Kristen L Granger², Ruben G Martinez³, Maureen A Conroy⁴, Patricia A Snyder⁴, Michael A Southam-Gerow³

Affiliations

¹ Department of Psychology, Virginia Commonwealth University, 806 West Franklin Street, PO Box 842018, Richmond, VA, 23284-2018, USA. bmcleod@vcu.edu.
² School of Education, Virginia Commonwealth University, Richmond, VA, USA.
³ Department of Psychology, Virginia Commonwealth University, 806 West Franklin Street, PO Box 842018, Richmond, VA, 23284-2018, USA.
⁴ Anita Zucker Center for Excellence in Early Childhood Studies, University of Florida, Gainesville, FL, USA.

PMID: 34714503
DOI: 10.1007/s11121-021-01304-w

Abstract

Though treatment integrity measurement is important for research intended to promote social and behavioral outcomes of children at risk for emotional and behavioral disorders (EBDs) in early childhood settings, measurement gaps exist in the field. This paper reports on the development and preliminary psychometric assessment of the treatment integrity measure for early childhood settings (TIMECS), an observational measure designed to address existing measurement gaps related to treatment integrity with tier 2 interventions in the early childhood field. To assess the preliminary score reliability (interrater) and validity (construct, discriminant) of the TIMECS, live observations (N = 650) in early childhood classrooms from 54 teachers (92.6% female, 7.4% male; 61.1% White) and 91 children (M age = 4.53 years, SD = .44; 45.1% female, 54.9% male; 45.1% Black) at risk for EBDs were scored by 12 coders using the TIMECS and an observational measure designed to assess teacher-child interactions. Teachers also self-reported on the quality of the teacher-child relationship. Interrater reliability (intraclass correlation coefficients, ICC [2,2]) for the quantity (i.e., adherence) item scores had a mean of .81 (SD = .07; range from .68 to .95), and the quality (i.e., competence) item scores had a mean of .69 (SD = .08; range from .52 to .80). Scores on the TIMECS Quantity and Quality items and scales showed evidence of construct validity, with the magnitude of the correlations suggesting that the quantity and quality items assess distinct components of treatment integrity. A TIMECS quantity scale also showed promise for intervention evaluation research by discriminating between teachers who had and had not been trained in a specific evidence-based intervention targeting social and behavioral skills in early childhood. The findings support the potential of the TIMECS to assess treatment integrity of teacher-delivered practices designed to address child social and behavioral outcomes of children at risk for EBDs in early childhood settings.

Keywords: Early childhood; Practice elements; Teacher implementation; Treatment integrity.

Publication types

Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

Child, Preschool
Emotions*
Female
Humans
Male
Psychometrics
Reproducibility of Results
Self Report