A 2000s text-to-speech system could pronounce every word correctly and still sound wrong. It might glide through a sentence, then stress the wrong word or flatten a question into a statement. Pronunciation was only part of the problem: a synthesizer also had to decide how long sounds lasted, where pitch rose or fell, and when to pause. Statistical methods let researchers learn some of those choices from recordings instead of writing every rule by hand.
No single statistical method displaced earlier synthesis during the decade. Large-corpus unit selection and statistical parametric synthesis developed side by side, sometimes within the same system. Both relied on recorded speech and linguistic analysis, but used the recordings differently. Unit selection assembled pieces of natural audio; parametric systems learned models that generated acoustic features, then converted those features into sound. The trade-offs between them drove much of the period's research.
What a synthesizer had to decide
Speech synthesis is often described as converting text into audio. Between those endpoints, a front end expanded abbreviations and numbers, found word pronunciations, and assigned labels for phonemes, syllables, stress, word boundaries, and sometimes phrase structure. An acoustic component then had to turn those labels into speech. “Dr.” might be read as “doctor,” while a number could be a quantity, a date, or part of an address. A better-sounding acoustic model could not repair a mistake made before waveform generation.
Timing mattered just as much. An unstressed vowel is often shorter than the same vowel under emphasis, while a syllable at the end of a phrase may lengthen. Pitch patterns depend on sentence structure, speaker, and intended emphasis. A system trained on carefully read sentences might produce convincing individual sounds yet mishandle a new phrase if its labels missed what made that phrase different.
Earlier formant synthesizers used explicit acoustic rules. They were controllable, but often sounded conspicuously artificial. Diphone concatenation joined recorded segments chosen to preserve transitions between neighboring sounds; it softened some mechanical qualities, though a small inventory left few options for prosody or voice quality. Growing storage capacity and speech corpora made it practical to choose among many recorded alternatives. Statistical modeling, meanwhile, offered a compact alternative to storing and searching a large inventory of waveform segments.
Unit selection: using statistics to choose real speech
For unit selection, a speaker recorded a substantial corpus. The system segmented and labeled the audio, building an inventory of candidate phones, diphones, syllables, or other spans. Given new text, it searched for a sequence that suited the requested phonetic and prosodic contexts without making the joins too noticeable.
That search often balanced two costs. A target cost measured how well a recorded unit matched its intended context, including neighboring phones, stress, and phrase position. A concatenation cost estimated how cleanly adjacent candidates would join, using acoustic cues near their boundaries. Choosing units one by one could yield plausible sounds separated by audible jumps. Favoring smooth joins above all else could give the wrong stress or timing. Dynamic programming let the system weigh whole sequences rather than isolated matches.
Unit selection's great advantage was that much of the output remained actual recorded speech. When the corpus contained a suitable stretch, the result could retain details of articulation and voice quality that were hard to model explicitly. Its weaknesses came from that same dependence: coverage was uneven, recording and annotation were costly, and one awkward join could interrupt an otherwise natural phrase. A new speaking style generally called for new recordings. Relabeling neutral speech could not supply cheerful or emphatic performances that had never been recorded.
Here, statistics guided the matching and search; they did not chiefly generate the waveform. The sound came largely from selected recordings, sometimes with pitch or duration modified. That is an important distinction from the model-based systems that gained ground during the decade.
Statistical parametric synthesis: modeling an acoustic trajectory
Statistical parametric synthesis started by analyzing recordings into acoustic features, typically a representation of the spectral envelope, fundamental frequency or pitch, and indicators of voicing. Duration models estimated how long sounds or model states should last. During synthesis, linguistic labels guided predictions across successive frames of speech, and a vocoder turned the resulting feature trajectories into a waveform.
Hidden Markov models, or HMMs, became especially important here. An HMM-based system commonly represented a context-dependent phoneme as several successive states. Each state described a distribution over acoustic features and contributed to timing. Training estimated those distributions from labeled recordings; synthesis chose states suited to the input and generated a sequence of feature vectors. The mathematics was related to HMM-based speech recognition, but the task ran in the opposite direction: given linguistic context, the synthesizer produced plausible acoustics instead of inferring words from acoustics. For the recognition side of that relationship, this blog's account of [how 2000s speech recognizers chose words](/statistical-models-2000s-speech-recognition/) describes the probabilistic choices involved.
Take “a blue car.” The vowel in “blue” changes with its neighboring sounds, whether the word is accented, and the phrase's place in a longer utterance. A context-dependent model tried to account for such differences through labels describing nearby phonemes, syllables, words, and phrases. A duration prediction set how long each state would occupy; spectral and pitch models then supplied frame-by-frame values.
Why decision trees mattered
A separate model for every combination of phonetic and prosodic labels would need more examples than a corpus could supply. Decision-tree clustering let related contexts share data. A tree might ask whether the next sound was a vowel, whether the current syllable was stressed, or whether a sound fell near a phrase boundary. Contexts that followed the same answers could share parameters even when their full labels differed.
That was a practical compromise, not a claim that the contexts sounded identical. It made finite training data usable and helped the synthesizer handle combinations missing from its recordings. But poorly chosen questions or sparse examples could blur differences listeners would notice. Linguistic feature design still mattered.
From predicted values to an audible sentence
Predicting each acoustic frame independently would create abrupt variation. HMM-based systems commonly modeled changes between neighboring frames as well as the values within each frame. A parameter-generation procedure then found trajectories consistent with both sets of predictions, encouraging continuous spectral and pitch contours. Voiced and unvoiced regions needed special care: an unvoiced consonant has no ordinary fundamental-frequency track to interpolate through.
The vocoder imposed its own limit. It reconstructed speech from a compressed acoustic description rather than replaying the speaker's waveform. Even sensible feature predictions could sound buzzy or muffled because of limitations in excitation and spectral representation. Better prosody did not automatically produce a more natural timbre.
The trade-off behind the characteristic sound
A statistical parametric voice could be compact beside a large unit inventory. Its parameters could be adapted or combined, which made it useful for experiments with speaker identity and speaking style. It also avoided the conspicuous bad join of concatenated audio. Its characteristic weakness was over-smoothing: predicted spectral trajectories tended toward average values and lost some of the sharp, varied detail of real speech.
Training sought parameters that worked across many examples, but a trajectory that was highly probable under the model was not necessarily lifelike. Averaging several valid ways to say a sound could produce a version unlike any of them. Researchers explored global-variance constraints to restore some variation, while facing the harder question of whether that variation suited the words and their acoustic context.
So the comparison was never simply “natural” versus “statistical.” Unit selection used statistical scoring and could sound exceptionally natural until its inventory forced a poor choice. Parametric synthesis could remain consistent and intelligible across unfamiliar sentences, yet sound less like unmodified human speech. The outcome depended on the corpus, text front end, recording conditions, model design, and test sentences.
Data, languages, and controllability
Statistical modeling did not eliminate the need for good recordings. A training corpus needed consistent audio, usable transcriptions, and reliable alignment between spoken material and linguistic labels. Automatic alignment saved manual work, but incorrect pronunciations and misplaced boundaries could carry through training. If a speaker moved relative to the microphone or changed delivery markedly between sessions, the system might learn that variation for the wrong reasons.
For languages with little recorded speech, parametric modeling offered a way to pool observations across related contexts. A compact voice might be trained without the coverage required by a large unit-selection inventory. Low-resource synthesis was still difficult. Systems needed a pronunciation lexicon or a method for deriving pronunciation from spelling, while stress, tone, phrasing, and number reading could demand language-specific work. In a tonal language, pitch distinguishes words as well as conveying intonation; flattening it is more than a stylistic flaw.
Researchers also explored speaker adaptation: starting with an existing model and adjusting it toward a new voice using less data than full training would require. Model parameters gave them ways to alter pitch range, speaking rate, and aspects of voice identity. Such controls appealed to applications needing multiple voices, but an adjustable parameter did not guarantee a convincing result. Quality depended on how closely the available training speech matched the intended speaker and style.
How the advances were judged
Listening tests remained essential because no single number captured intelligibility, voice quality, and appropriate expression. Listeners might give mean opinion scores for naturalness, compare paired samples, transcribe unfamiliar sentences, or judge resemblance to a target speaker. Each test asked something different. Someone could understand every word and still dislike a voice's rhythm or timbre.
The choice of test sentences could change the apparent winner. Text resembling a unit-selection corpus might favor its recorded fragments; unusual names, rare sound contexts, or unfamiliar prosodic patterns might expose gaps. Parametric systems avoided some inventory failures but could reveal over-smoothed pitch and spectral detail. A fair comparison had to say which text front end was used, how much speech each voice had available, and whether the test concerned naturalness, intelligibility, or control.
One revealing sentence puts a familiar word in an unfamiliar prosodic role: “I said blue car, not green car.” The contrast calls for extra stress on “blue,” changing its duration and pitch in relation to the surrounding words. A system that pronounces every word correctly but reads “blue” as it would in a neutral phrase has solved the sound sequence, not the whole utterance.