Hybrid rice breeding offers an important strategy to improve rice production, in which the cultivation of a male sterile line is the key to the success of cross-breeding. CRISPR/Cas9 systems have been widely used in target-site genome editing, whereas their application for crop genetic improvement has been rarely reported. Here, using the CRISPR/Cas9 system, we induced specific mutations in TMS5, which is the most widely applied thermo-sensitive genic male sterility (TGMS) gene in China, and developed new "transgene clean" TGMS lines. We designed 10 target sites in the coding region of TMS5 for targeted mutagenesis using the CRISPR/Cas9 system and assessed the potential rates of on- and off-target effects. Finally, we established the most efficient construct, the TMS5ab construct, for breeding potentially applicable "transgene clean" TGMS lines. We also discussed factors that affect the editing efficiency according to the characteristics of different target sequences. Notably, using the TMS5ab construct, we developed 11 new "transgene clean" TGMS lines with potential applications in hybrid breeding within only one year in both rice subspecies. The application of our system not only significantly accelerates the breeding of sterile lines but also facilitates the exploitation of heterosis.