Say “cat” twice and the vowel may occupy a different number of acoustic frames each time, while the opening consonant stays brief. A recognizer comparing both recordings with the same fixed-length template, frame by frame, would have trouble lining them up. Hidden Markov models (HMMs) gave statistical speech recognizers a way to let a sequence of speech units occupy different amounts of time while testing how well the recorded sounds matched a proposed word.
That flexibility helped make HMMs central to speech recognition from the 1980s through the 2000s. No single HMM “understood” speech. Instead, the framework brought together acoustic measurements, possible sound sequences, pronunciation dictionaries, and language constraints. Each addressed a different uncertainty; decoding meant finding the word sequence that fit them best.
From acoustic signal to candidate words
Speech arrives as a waveform, but an HMM-based recognizer usually worked with short-time feature vectors. A front end divided the recording into overlapping frames, often a few tens of milliseconds long, and described each frame using measurements such as mel-frequency cepstral coefficients, energy, and changes over time. Feature sets varied by system and period. The recognizer received compact acoustic descriptions, not written phonemes.
Consider an utterance that might be “recognize speech.” The recognizer must work out where one sound gives way to another, which pronunciation the speaker used, and which words were intended. The signal offers no neatly marked boundaries. Neighboring sounds influence one another, speakers differ, and noise can hide useful cues. An HMM treats the underlying speech states as hidden: it observes acoustic features and infers a plausible path through states that could have produced them.
Three kinds of information shaped a conventional decoding decision:
- Acoustic evidence: how well each observed feature vector fits its assigned speech state.
- Pronunciation information: which sequences of phones or other units can represent a word.
- Language constraints: which word sequences are likely or allowed in the task.
An acoustic model alone could struggle to distinguish similar-sounding words. A likely sentence, meanwhile, should not win when its pronunciation poorly matches the recording. The HMM framework was useful in part because a decoder could score and search combinations of these constraints.
What the hidden states represented
A basic HMM has states, probabilities of moving between them, and a way to assign a likelihood to an observation in each state. In speech recognition, states typically represented portions of a phone or another subword unit, rather than whole sentences. A common left-to-right phone model used several sequential states: one might account for the beginning of a sound, another for its middle, and a third for its end. A state could repeat for several frames before advancing, so a phone did not need a fixed duration.
These states were statistical devices, not a claim that speakers deliberately produce three separate stages of every sound. They gave the model local structure for changes within a phone. Models for individual units could be connected according to a pronunciation dictionary to form word paths; a grammar or language model then connected those paths into a larger search space for an utterance.
The “Markov” part is a simplifying assumption: the probability of the next state depends on the current state, not on every earlier state in the utterance. A standard HMM also treats an observation as dependent on its current hidden state. Real speech is less tidy. A vowel can be influenced by sounds more than one state away, and adjacent acoustic frames are strongly correlated. These assumptions made training and decoding manageable, while context-dependent units and richer features addressed some of the mismatch.
Why a self-loop mattered
A self-loop lets a state remain active for another frame. If one speaker holds a vowel longer than another, both recordings can follow much the same state sequence while spending different numbers of frames in its states. This did not capture every aspect of speaking rate: HMM state-duration distributions are implicit and often simple. It did, however, replace rigid frame-by-frame alignment with a probabilistic choice among alignments.
How the models were trained and searched
Training speech HMMs required recordings paired, where possible, with transcriptions. A transcription identified the intended words but usually not the exact frame where each phone began or ended. A pronunciation dictionary supplied candidate phone sequences for the words. Training then estimated how those sequences aligned with the observed frames and adjusted model parameters to improve their likelihood.
One established method was Baum–Welch training, an expectation-maximization procedure. It weighs possible state paths under the current model and uses their probabilities to update parameters. Systems also used alignments obtained by Viterbi decoding, which selects the single highest-scoring path under the model. The two methods handle uncertain state boundaries differently; neither makes a word transcription into a perfectly known phone-level timeline.
During recognition, the Viterbi algorithm was widely used to find a high-scoring path through the network of candidate states. Rather than score every complete path independently, it keeps the best score reaching each state at a given time. Practical decoders also pruned paths that scored far below promising alternatives, saving computation and memory. Some preserved multiple hypotheses in a lattice for later rescoring.
Informally, the goal was to choose the word sequence with the best combination of acoustic and language-model scores. In deployed systems, those scores often needed scaling and insertion penalties: their raw numerical balance did not necessarily produce the best word output. Decoding was an engineering problem as well as a probability calculation.
From whole words to context-dependent phones
When vocabularies were small and each word had enough recorded examples, early recognizers could model whole words. As vocabularies grew, that approach became impractical. Every word needed acoustic coverage, and an unseen word could not simply inherit a model. Subword HMMs allowed phones to be shared across many words, each represented by a sequence of those units.
A phone pronounced in isolation does not sound exactly the same when surrounded by other sounds. Coarticulation changes its realization. By the 1990s, many successful systems used context-dependent phone models, particularly triphones, which identify a central phone by its immediate left and right neighbors. That creates many potential units. Decision-tree state tying kept the approach practical by letting acoustically related states share parameters, including in contexts with little training data.
Acoustic distributions changed too. Earlier systems often used discrete observation symbols produced by vector quantization. Continuous-density HMMs became prominent, commonly using Gaussian mixture models (GMMs) to score feature vectors in each state. Mixtures could represent varied observations within a state without giving every speaker or pronunciation variant a separate state path. This architecture was commonly called a GMM-HMM system.
These changes did not remove the need for a lexicon. A dictionary could list alternatives for a word with multiple common pronunciations, allowing the decoder to compare their paths. But if a name was absent from the vocabulary, even a good acoustic match might not yield the correct spelling. Recognition quality depended on the whole pipeline.
What HMMs enabled in practice
The same broad approach served quite different tasks. A digit recognizer had few permitted words and could use tight sequence constraints. Large-vocabulary dictation faced many acoustically similar candidates and needed stronger language modeling. Broadcast speech added variation in speakers, recording conditions, and vocabulary. These were not simply larger versions of one easy problem.
HMM systems also supported a modular way of working. Researchers could inspect acoustic likelihoods, pronunciation entries, word error rates, and decoder output separately. They could retrain with more speech, adapt parameters to a speaker or recording condition, or change a language model without rebuilding everything else. That made methods easier for research groups to compare and helped recognizers run on the computing resources available at the time.
For languages with little recorded and transcribed speech, modularity helped but did not solve the data problem. Shared subword models reduced the need to collect examples of every word, yet a recognizer still needed suitable pronunciations, acoustic data, and representative text. A model trained on one speaking style or dialect would not automatically work on another. Tests using genuinely separate speakers and conditions were needed to show what it could recognize.
Where the framework fell short
Standard HMMs represented duration indirectly through transitions and made strong independence assumptions about the frames they scored. GMM observation models depended heavily on engineered features and could struggle with complex acoustic variation. Context-dependent phones improved local modeling, but added parameters and forced choices about which states should share data. Noise, overlapping speakers, spontaneous speech, and rare words remained difficult.
Researchers worked around these limits rather than treating a basic HMM as a finished solution. They developed speaker adaptation, discriminative training objectives, better feature transforms, pronunciation variants, and improved decoding and language models. As a result, two systems both called “HMM recognizers” could perform quite differently. The name identifies a core sequencing framework, not a fixed recipe or a level of accuracy.
Later neural-network acoustic models changed how observations were scored. In hybrid systems, a neural network estimated information used to score HMM states, while the HMM still provided state sequencing and alignment. Accounts that describe deep learning as an immediate replacement for the entire earlier pipeline miss this intermediate stage. Other approaches eventually handled alignments and outputs differently, but hybrids show which part of the older architecture remained useful.
Reading an early recognition result carefully
When an older paper reports an HMM-based result, the model name alone says little about the test's difficulty. Check the vocabulary size, whether training and test speakers were distinct, the recording environment, whether speech was read or conversational, and what language constraints the decoder had. Word error rate counts substitutions, deletions, and insertions against a reference transcript; it does not explain why an error occurred.
Suppose a decoder returns “recognize speech” when the reference says “recognition speech.” The difference could come from the acoustic score, a missing pronunciation, a stronger language-model preference, or pruning that removed a candidate path too early. Inspecting the competing word paths and the lexicon entry would be more informative than blaming “the HMM.” Its states organized timing and scores, but the final words came from the complete recognition system.