Type 2 diabetes is a growing health problem worldwide that is particularly severe in India and China. In these areas, bitter gourd (Momordica charantia) is a popular vegetable which is traditionally known to have health beneficial effects not only, but mainly, on diabetes. Bitter gourd could be a cheap possibility to help the poor in these and other countries to control their blood glucose levels. This review describes anti-diabetic effects of bitter gourd reported in the literature and discusses what still needs to be clarified for developing an evidence-based and safe use of the bitter gourd for diabetes. Analyses of bioactive compounds have shown that bitter gourd is rich in nutrients and phytochemicals of which some have anti-diabetic effects. Juices, powders, extracts, and isolated compounds have been tested in vitro and in vivo. Bitter gourd increases insulin secretion of the pancreas, decreases intestinal glucose uptake, and increases uptake and utilization of glucose in peripheral tissues. Although human studies with type 2 diabetics are weak in their design and/or results, some of the studies do indicate anti-diabetic effects in patients and safety for bitter gourd treatment in humans. In the future, well designed studies with rodents will help to understand what kind of bitter gourd variety, dosage, preparation, and duration of administration is optimal. Such results will help to design human studies which are necessary to prove the effectiveness of bitter gourd in patients.