A recently established materials concept of biomimetic composites based on silica, collagen, and calcium phosphates was adapted for the preparation of porous scaffolds suitable for tissue engineering applications. Mineralization was achieved by directed nucleation of silica on the templating organic phase during a sol-gel process with or without addition of hydroxyapatite. Both mineral phases (25 wt %, individually or combined in equal shares) influenced the scaffold's morphology at the nanoscale. Enhancement of apparent density and compressive strength was similar for silica or hydroxyapatite mineralization; however the stiffening effect of hydroxyapatite was much higher. All scaffold modifications provided proper conditions for adhesion, proliferation, and osteogenic differentiation of human bone marrow stromal cells. The open porosity allowed cells to migrate throughout the scaffolds while maintaining their viability, both confirmed by MTT staining and confocal laser scanning microscopy. Initial cell distributions were graduated due to collagen mineralization, but balanced out over the cultivation time of 28 days. RT-PCR analyses revealed higher gene expression of ALP but lower expression of BSP II and osteocalcin because of collagen mineralization. The results demonstrate that both silica and hydroxyapatite offer comparable possibilities to tailor mechanical properties of collagen-based scaffolds without being detrimental to in vitro biocompatibility.