Community assembly of crop-associated fungi is thought to be strongly influenced by deterministic selection exerted by the plant host, rather than stochastic processes. Here we use a simple, sorghum system with abundant sampling to show that stochastic forces (drift or stochastic dispersal) act on fungal community assembly in leaves and roots early in host development and when sorghum is drought stressed, conditions when mycobiomes are small. Unexpectedly, we find no signal for stochasticity when drought stress is relieved, likely due to renewed selection by the host. In our experimental system, the host compartment exerts the strongest effects on mycobiome assembly, followed by the timing of plant development and lastly by plant genotype. Using a dissimilarity-overlap approach, we find a universality in the forces of community assembly of the mycobiomes of the different sorghum compartments and in functional guilds of fungi.