Although several methods have been used in bone regeneration medicine, current methods still have many limitations. The tissue used for autogenous bone graft is limited and allograft has weak osteoinductive activity. Tissue engineering provides a good choice for bone regeneration. However, the growth factors needed have a high price and short half-life. Recently, a number of small molecules have been confirmed to have osteoinductive activity and some have been clinically used. Natural small molecules including decalpenic acid, flavonoids, quinones can be extracted from plants and others can be synthesized according to the structure designed or mimicking the structure of natural small molecules. Small molecules can act as co-activator of BMP2 pathway or activate Wnt pathway; others can be the inhibitors of NF-κB signaling pathway. This review gives an overview on the small molecules with osteoinductive activity and discusses the mechanism of the small molecules.