Winter wheat (Triticum aestivum L.) monoculture is conventionally cultivated followed by two to three months of summer fallow in the Loess Plateau. To develop a sustainable cropping system, we conducted a six-year field experiment to investigate the effect of leguminous green manure (LGM) instead of bare fallow on the yield and water use efficiency (WUE) of winter wheat and the soil water balance (SWB) in different precipitation years in a semi-arid region of northwest China. Results confirmed that planting LGM crop consumes soil water in the fallow season can bring varied effects to the subsequent wheat. The effect is positive or neutral when the annual precipitation is adequate, so that there is no significant reduction in the soil water supplied to wheat. If this is not the case, the effect is negative. On average, the LGM crop increased wheat yield and WUE by 13% and 28%, respectively, and had considerable potential for maintaining the SWB (0-200 cm) compared with fallow management. In conclusion, cultivation of the LGM crop is a better option than fallow to improve the productivity and WUE of the next crop and maintain the soil water balance in the normal and wet years in the Loess Plateau.