The remarkable mechanical behavior of bone is attributed to its complex nanocomposite structure that, in addition to mineral and collagen, comprises a variety of non-collagenous matrix proteins or NCPs. Traditionally, NCPs have been studied as signaling molecules in biological processes including bone formation, resorption, and turnover. Limited attention has been given to their role in determining the mechanical properties of bone. Recent studies have highlighted that NCPs can indeed be lost or modified with aging, diseases, and drug therapies. Homozygous and heterozygous mice models of key NCP provide a useful approach to determine the impact of NCPs on bone morphology as well as matrix quality, and to carry out detailed mechanical analysis for elucidating the pathway by which NCPs can affect the mechanical properties of bone. In this article, we present a systematic analysis of a large cohort of NCPs on bone's structural and material hierarchy, and identify three principal pathways by which they determine bone's mechanical properties. These pathways include alterations of bone morphological parameters crucial for bone's structural competency, bone quality changes in key matrix parameters (mineral and collagen), and a direct role as load-bearing structural proteins.