DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting

Abstract

Text-to-speech models built on score matching and flow matching produce natural speech, but only by calling the decoder many times, which makes synthesis slow. Existing ways to cut the number of calls either distill from a multi-step teacher or add a stage on top of a pretrained model, leaving single-step TTS without a pretrained TTS model unexplored. We question whether pretraining a TTS model is necessary, and propose DriftTTS, which adopts generative drifting, an objective that moves iterative sampling from inference into training. Drifting assumes samples of fixed shape, whereas speech has variable length; we address this issue with segment drifting. On LJSpeech, DriftTTS matches the multi-step quality of Matcha-TTS and RapFlow-TTS in a single decoder call, while being faster. To our knowledge, this is the first application of generative drifting to text-to-speech.

Contour plots of target and generated mel distributions at epoch 1, 15, 25, and 100. The generated contours move onto the target contours as training proceeds.

Generated mel distribution drifting onto the target. Density contours of target (blue) and generated (orange) mel-spectrogram segments at training epochs 1, 15, 25, and 100. With generative drifting the single-step generator is pulled toward the target distribution during training, so no iterative sampling is needed at inference.

DriftTTS One-stage Single Step Text-to-Speech via Generative Drifting

Abstract

Model comparison