The Cooking and Pneumonia Study (CAPS) is a pragmatic cluster-level randomized controlled trial of the effect of an advanced cookstove intervention on pneumonia in children under the age of 5 years (under 5s) in Malawi (www.capstudy.org). The primary outcome of the trial is the incidence of pneumonia during a two-year follow-up period, as diagnosed by healthcare providers who are using the World Health Organization (WHO) integrated management of childhood illnesses (IMCI) pneumonia assessment protocol and who are blinded to the trial arms. We evaluated the quality of pneumonia assessment in under 5s in this setting via a cross-sectional study of provider-patient encounters at nine outpatient clinics located within the catchment area of 150 village-level clusters enrolled in the trial across the two study locations of Chikhwawa and Karonga, Malawi, between May and June 2015 using the IMCI guidelines as a benchmark. Data were collected using a key equipment checklist, an IMCI pneumonia knowledge test, and a clinical evaluation checklist. The median number of key equipment items available was 6 (range 4 to 7) out of a possible 7. The median score on the IMCI pneumonia knowledge test among 23 clinicians was 75% (range 60% to 89%). Among a total of 176 consultations performed by 15 clinicians, a median of 9 (range 3 to 13) out of 13 clinical evaluation tasks were performed. Overall, the clinicians were adequately equipped for the assessment of sick children, had good knowledge of the IMCI guidelines, and conducted largely thorough clinical evaluations. We recommend the simple pragmatic approach to quality assurance described herein for similar studies conducted in challenging research settings.