Background: As community case management of childhood illness expands in low-income countries, there is a need to assess the quality of care provided by community health workers. This study had the following objectives: 1) examine methods of recruitment of sick children for assessment of quality of care, 2) assess the validity of register review (RR) and direct observation only (DO) compared to direct observation with re-examination (DO+RE), and 3) assess the effect of observation on community health worker performance.
Methods: We conducted a survey to assess the quality of care provided by Ethiopian Health Extension Workers (HEWs). The sample of children was obtained through spontaneous consultation, HEW mobilization, or recruitment by the survey team. We assessed patient characteristics by recruitment method. Estimates of indicators of quality of care obtained using RR and DO were compared to gold standard estimates obtained through DO+RE. Sensitivity, specificity, and the area under receiver operator characteristic curve (AUC) were calculated to assess the validity of RR and DO. To assess the Hawthorne effect, we compared estimates from RR for children who were observed by the survey team to estimates from RR for children who were not observed by the survey team.
Results: Participants included 137 HEWs and 257 sick children in 103 health posts, plus 544 children from patient registers. Children mobilized by HEWs had the highest proportion of severe illness (27%). Indicators of quality of care from RR and DO had high sensitivity for most indicators, but specificity was low. The AUC for different indicators from RR ranged from 0.47 to 0.76, with only one indicator above 0.75. The AUC of indicators from DO ranged from 0.54 to 1.0, with three indicators above 0.75. The differences between estimates of correct care for observed versus not observed children were small.
Conclusions: Mobilization by HEWs and recruitment by the survey teams were feasible, but potentially biased, methods of obtaining sick children. Register review and DO underestimated performance errors. Our data suggest that being observed had only a small positive effect on the performance of HEWs.