To improve cartilage formation by bone marrow-derived mesenchymal stem cells (BMSCs), the signaling mechanism governing chondrogenic differentiation requires better understanding. We previously showed that the transforming growth factor-β (TGFβ) receptor ALK5 is crucial for chondrogenesis induced by TGFβ. ALK5 phosphorylates SMAD2 and SMAD3 proteins, which then form complexes with SMAD4 to regulate gene transcription. By modulating the expression of SMAD2, SMAD3 and SMAD4 in human BMSCs, we investigated their role in TGFβ-induced chondrogenesis. Activation of TGFβ signaling, represented by SMAD2 phosphorylation, was decreased by SMAD2 knockdown and highly increased by SMAD2 overexpression. Moreover, TGFβ signaling via the alternative SMAD1/5/9 pathway was strongly decreased by SMAD4 knockdown. TGFβ-induced chondrogenesis of human BMSCs was strongly inhibited by SMAD4 knockdown and only mildly inhibited by SMAD2 knockdown. Remarkably, both knockdown and overexpression of SMAD3 blocked chondrogenic differentiation. Chondrogenesis appears to rely on a delicate balance in the amount of SMAD3 and SMAD4 as it was not enhanced by SMAD4 overexpression and was inhibited by SMAD3 overexpression. Furthermore, this study reveals that TGFβ-activated phosphorylation of SMAD2 and SMAD1/5/9 depends on the abundance of SMAD4. Overall, our findings suggest a more dominant role for SMAD3 and SMAD4 than SMAD2 in TGFβ-induced chondrogenesis of human BMSCs.