Understanding the growth mechanism of single-walled carbon nanotubes (SWCNTs) and achieving selective growth requires insights into the catalyst structure-function relationship. Using an in situ aberration-corrected environmental transmission electron microscope, we reveal the effects of the state and structure of catalysts on the growth modes of SWCNTs. SWCNTs grown from molten catalysts via a vapor-liquid-solid process generally present similar diameters to those of the catalysts, indicating a size correlation between nanotubes and catalysts. However, SWCNTs grown from solid catalysts via a vapor-solid-solid process always have smaller diameters than the catalysts, namely, an independent relationship between their sizes. The diameter distribution of SWCNTs grown from crystalline Co7W6, which has a unique atomic arrangement, is discrete. In contrast, nanotubes obtained from crystalline Co are randomly dispersed. The different growth modes are linked to the distinct chiral selectivity of SWCNTs grown on intermetallic and monometallic catalysts. These findings will enable rational design of catalysts for chirality-controlled SWCNTs growth.