Guiding institutional investors to actively participate in corporate governance is a hot issue to improve the internal governance of China's listed companies. This study seeks to provide a comprehensive understanding of the mechanism that underlies the governance effects of the heterogeneity of institutional investors on the cost of capital, and the influence of ownership structure on the relationship between them. Using an unbalanced panel data on A-share listed companies of Shanghai and Shenzhen in China's capital market during the 2014-2019 period, this study reveals how institutional investors with longer holding period and higher shareholding ratio are negatively associated with the cost of capital in China's capital market. Furthermore, this study successfully confirms the moderating effect of ownership structure in the relationship between institutional investors and the cost of capital. China's state-owned enterprises are more likely to introduce improvements at the corporate governance level, and ownership concentration weakens the negative influence of institutional investors on the cost of capital. The research contributes to a deeper understanding of the impacts of institutional investor's heterogeneity and ownership structure on the cost of capital in China. In the process, the study yields useful implications for the theory and practice of corporate governance.