Designing a questionnaire with retrospective pre-post items: Format matters

Abstract

Evaluators are frequently asked to evaluate educational interventions that are one-time events. The Retrospective Pretest (RPT) methodology is well suited for these circumstances and the evaluation literature is replete with discussions about the pros and cons of this approach. In RPT, program participants rate their attitudes or knowledge now, and also rate how they were before participation in the intervention. The difference between now and before ratings constitutes the measure of change. Little published literature exists about whether the layout of RPT items within the evaluation questionnaire yields different results. This study compared six different layouts using a sample of 1941 caregivers who participated in one of 96 training workshops. The layouts were compared on inattentiveness, unexpected decline in perceived knowledge, and the degree of before-now change. Findings show that design matters. The best performing layout is where items are placed in the center with before response options on the left of the page and the now response options on the right. Results point to the need for evaluators to pay attention to RPT layout, and for the field to establish criteria for assessing survey layout quality. The unexpectedly high rate of inattentiveness calls for evaluators to pay more attention identifying and addressing this in their survey data.

Keywords: Assessing questionnaire quality; Carelessness in survey research; Questionnaire design; RPT; Retrospective PreTest.