Many pathogenic bacteria, fungi, and protozoa achieve chronic infection through an immune evasion strategy known as antigenic variation. In the human malaria parasite Plasmodium falciparum, this involves transcriptional switching among members of the var gene family, causing parasites with different antigenic and phenotypic characteristics to appear at different times within a population. Here we use a genome-wide approach to explore this process in vitro within a set of cloned parasite populations. Our analyses reveal a non-random, highly structured switch pathway where an initially dominant transcript switches via a set of switch-intermediates either to a new dominant transcript, or back to the original. We show that this specific pathway can arise through an evolutionary conflict in which the pathogen has to optimise between safeguarding its limited antigenic repertoire and remaining capable of establishing infections in non-naïve individuals. Our results thus demonstrate a crucial role for structured switching during the early phases of infections and provide a unifying theory of antigenic variation in P. falciparum malaria as a balanced process of parasite-intrinsic switching and immune-mediated selection.