This study investigated the in vitro fermentation of polysaccharides from Morus alba L., the contribution of its carbohydrates to the fermentation, and the effect on the composition of gut microbiota. Over 48 h of fermentation, the pH value in the fecal culture decreased from 7.12 to 6.14, and the total short chain fatty acids (SCFA) and acetic, propionic, and butyric acids all significantly increased. After 48 h of fermentation, 45.36 ± 1.36% of the total carbohydrates in the polysaccharide, including 35.72 ± 1.51% of arabinose, 23.1 ± 1.19% of galactose, 41.43 ± 1.52% of glucose, 26.36 ± 1.93% of rhamnose and 65.57 ± 1.07% of galacturic acid, were consumed. The increase in acetic and butyric acids was primarily due to the fermentation of galactose and galacturonic acid in the polysaccharide, while the increase in propionic acid resulted mainly from the fermentation of arabinose and glucose. In addition, the polysaccharide could modulate the gut microbiota composition by increasing the Bacteroidetes population and decreasing the Firmicutes population. The results may facilitate the development of food products known as prebiotics, aimed at improving gastrointestinal health.