Consumers' autonomy regarding health increasingly requires competences to critically appraise health information. Critical health literacy refers to the concept of evidence-based medicine. Instruments to measure these competences in curriculum evaluation and surveys are lacking. We aimed to develop and validate an instrument to measure critical health competences (CHC test). Development and testing of the questionnaire covered three phases: (1) test construction (and feasibility, (2) first field test of scalability and items revision (3) second field test to validate the instrument. Model fit analyses were performed for both field tests for Rasch-, Mixed Rasch- and Hybrid model. Participants were secondary school and university students with and without prior training in evidence-based medicine (1. field test n = 322; with training n = 87; 2. field test n = 107; with training n = 13). The second field test resulted in Rasch scalability of all items in one person class. Mean values (+/-SD) of person parameters were 716.14 (+/-53.74) for trained students and 470.11 (+/-59.63) for untrained students. Reliability of the instrument was 0.91 (WINMIRA ANOVA). In conclusion the CHC instrument is a feasible, reliable and valid instrument to measure critical health literacy. The generalizability of the instrument is to be explored in ongoing studies in different educational settings.