Multilayered defense responses ensure that plants are hosts to only a few adapted pathogens in the environment. The host range of a plant pathogen depends on its ability to fully overcome plant defense barriers, with failure at any single step sufficient to prevent life cycle completion of the pathogen. Puccinia striiformis, the causal agent of stripe rust (=yellow rust), is an agronomically important obligate biotrophic fungal pathogen of wheat and barley. It is generally unable to complete its life cycle on the non-adapted wild grass species Brachypodium distachyon, but natural variation exists for the degree of hyphal colonization by Puccinia striiformis. Using three B. distachyon mapping populations, we identified genetic loci conferring colonization resistance to wheat-adapted and barley-adapted isolates of P. striiformis. We observed a genetic architecture composed of two major effect QTLs (Yrr1 and Yrr3) restricting the colonization of P. striiformis. Isolate specificity was observed for Yrr1, whereas Yrr3 was effective against all tested P. striiformis isolates. Plant immune receptors of the nucleotide binding, leucine-rich repeat (NB-LRR) encoding gene family are present at the Yrr3 locus, whereas genes of this family were not identified at the Yrr1 locus. While it has been proposed that resistance to adapted and non-adapted pathogens are inherently different, the observation of (1) a simple genetic architecture of colonization resistance, (2) isolate specificity of major and minor effect QTLs, and (3) NB-LRR encoding genes at the Yrr3 locus suggest that factors associated with resistance to adapted pathogens are also critical for non-adapted pathogens.