Bone as a hierarchical composite structure plays a myriad of roles in vertebrate skeletons including providing the structural stability of the body. Despite this critical role, the mechanical behaviour at the sub-micron levels of bone's hierarchy remains poorly understood. At this scale, bone is composed of Mineralised Collagen Fibrils (MCF) embedded within an extra-fibrillar matrix that consists of hydroxyapatite minerals and non-collagenous proteins. Recent experimental studies hint at the significance of the extra-fibrillar matrix in providing the bone with the stiffness and ductility needed to serve its structural roles. However, due to limited resolution of experimental tools, it is not clear how the arrangement of minerals, and in particular their relative distribution between the intra- and extra-fibrillar space contribute to bone's remarkable mechanical properties. In this study, a Coarse Grained Molecular Dynamics (CGMD) framework was developed to study the mechanical properties of MCFs embedded within an extra-fibrillar mineral matrix and the precise roles extra- and intra-fibrillar mineralisation on the load-deformation response was investigated. It was found that the presence of extra-fibrillar mineral resulted in the development of substantial residual stress in the system, by limiting MCF shortening that took place during intra-fibrillar mineralisation, resulting in substantial compressive residual stresses in the extra-fibrillar mineral phase. The simulation results also revealed the crucial role of extra-fibrillar mineralisation in determining the elastic response of the Extrafibrillar mineralised MCF (EFM-MCF) system up to the yield point, while the fibrillar collagen affected the post-yield behaviour. When physiological levels of mineralisation were considered, the mechanical response of the EFM-MCF systems was characterised by high ductility and toughness, with micro-cracks being distributed across the extra-fibrillar matrix, and MCFs effectively bridging these cracks leading to an excellent combination of strength and toughness. Together, these results provide novel insight into the deformation mechanisms of an EFM-MCF system and highlight that this universal building block, which forms the basis for lamellar bone, can provide an excellent balance of stiffness, strength and toughness, achieving mechanical properties that are far beyond the capabilities of the individual constituents acting alone.