A real-time voice cloning system with multiple algorithms for speech quality improvement

PLoS One. 2023 Apr 3;18(4):e0283440. doi: 10.1371/journal.pone.0283440. eCollection 2023.

Abstract

With the development of computer technology, speech synthesis techniques are becoming increasingly sophisticated. Speech cloning can be performed as a subtask of speech synthesis technology by using deep learning techniques to extract acoustic information from human voices and combine it with text to output a natural human voice. However, traditional speech cloning technology still has certain limitations; excessively large text inputs cannot be adequately processed, and the synthesized audio may include noise artifacts like breaks and unclear phrases. In this study, we add a text determination module to a synthesizer module to process words the model has not included. The original model uses fuzzy pronunciation for such words, which is not only meaningless but also affects the entire sentence. Thus, we improve the model by splitting the letters and pronouncing them separately. Finally, we also improved the preprocessing and waveform conversion modules of the synthesizer. We replace the pre-net module of the synthesizer and use an upgraded noise reduction algorithm combined with the SV2TTS framework to achieve a system with superior speech synthesis performance. Here, we focus on improving the performance of the synthesizer module to achieve higher-quality speech synthesis audio output.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Algorithms
  • Cloning, Molecular
  • Humans
  • Quality Improvement
  • Speech
  • Speech Perception*
  • Voice Quality
  • Voice*

Grants and funding

This work was supported by the Science and Technology Innovation Program of Hunan Province, the system number is 2016TP1020; Hunan Provincial Natural Science Foundation of China, the system number is 2022JJ50138; This work was also supported by The 14th Five-Year Plan Project of Educational Science in Hunan Province, the system number is XJK21BGD007.