Consumption of whole grain foods has been associated with decreased risk of type 2 diabetes. Insulin sensitivity and inflammation are key mechanisms in the development of type 2 diabetes, but the cause of the protective effects of whole grains is not known. In this review, we search for evidence to support the hypothesis of a link between whole grains, large intestinal events, and peripheral inflammation. Due to the unique structure and composition of the grain fiber complex, fermentation in the large intestine is probably an important mediator of the effects of whole grains. Fermentation of grain fiber takes place throughout the large intestine affecting beneficially the composition of gut microbiota, hence decreasing the permeability of gut barrier. Improved gut barrier function reduces leaking of endotoxic bacterial lipopolysaccharides (LPS) into the circulation. Lower concentration of LPS in blood seems to alleviate peripheral inflammation. Fermentation of grain fiber also leads to continuous supply and absorption of metabolites such as short chain fatty acids and ferulic acid derivatives which may have anti-inflammatory effects. These phenomena, mainly based on in vitro and animal studies, are associated with fermentation of grain fiber and improve insulin sensitivity, which over time may decrease the risk of type 2 diabetes. To test these mechanisms, more well-designed human studies are needed.