The gastrointestinal tract represents the largest immune interface with the environment. Exposure to large numbers of dietary and microbial antigens requires complex and highly regulated intestinal immune responses by different immune cell types for the maintenance of oral tolerance. Defective immune homeostasis can cause gut barrier dysfunction and breakdown of tolerance, leading to chronic inflammation and autoimmunity. In this review, we summarize the key immune cell populations involved in oral tolerance. We also describe diet-modifiable aspects of gut immunity that alter the intricate balance between inflammatory and tolerogenic immune responses in the gut and contribute to disease development.