The occurrence of mycotoxins in small grain cereals and their retention in final products are serious concerns for food safety. Previously, we investigated the fate of deoxynivalenol and nivalenol in a Japanese soft red winter wheat cultivar during milling and we found that deoxynivalenol and/or nivalenol was readily distributed among flours for human consumption. In the present study, we analyzed the ergosterol concentrations in the milling fractions as an index of fungal biomass to elucidate the relationship between deoxynivalenol/nivalenol accumulation and fungal invasion into the grain, after the in-house validation of an analytical method for quantifying ergosterol in the resulting milling fractions (patent flour, low-grade flour, bran, and shorts). Using three samples with different levels of deoxynivalenol and/or nivalenol contamination, the contents of deoxynivalenol/nivalenol and ergosterol in the resulting milling fractions were evaluated. The concentration of ergosterol was always lowest in patent flour and highest in bran or shorts, indicating that most of the fungi is retained in the outer layers of grain (bran and shorts) even in highly contaminated grain. On the other hand, the concentrations of deoxynivalenol and nivalenol were similar in the low-grade and patent flours and only slightly lower than in the medium-level and high-level contaminated grains. Moreover, the percentage distribution of ergosterol was higher in bran than in other fractions in all cases, which differed from that of deoxynivalenol/nivalenol. This result indicates the diffusion of deoxynivalenol/nivalenol inside the grain that is independent of fungal invasion.