Increasing resistance by malaria parasites to currently used antimalarials across the developing world warrants timely detection and classification so that appropriate drug combinations can be administered before clinical complications arise. However, this is often challenged by low levels of infection (referred to as parasitemia) and presence of predominantly young parasitic forms in the patients' peripheral blood. Herein, we developed a simple, inexpensive and portable image-based cytometer that detects and numerically counts Plasmodium falciparum infected red blood cells (iRBCs) from Giemsa-stained smears derived from infected blood. Our cytometer is able to classify all parasitic subpopulations by quantifying the area occupied by the parasites within iRBCs, with high specificity, sensitivity and negligible false positives (~ 0.0025%). Moreover, we demonstrate the application of our image-based cytometer in testing anti-malarial efficacy against a commercial flow cytometer and demonstrate comparable results between the two methods. Collectively, these results highlight the possibility to use our image-based cytometer as a cheap, rapid and accurate alternative for antimalarial testing without compromising on efficiency and minimal processing time. With appropriate filters applied into the algorithm, to rule out leukocytes and reticulocytes, our cytometer may also be used for field diagnosis of malaria.