Automatic Speech Recognition: From Hidden Markov Models to Speech Foundation Models

Automatic speech recognition (ASR) converts a speech waveform into text. A production transcription system also has to decide when speech begins and ends, identify the language, place timestamps, separate speakers, restore punctuation, adapt to domain vocabulary, and deliver results within a latency and compute budget.

No model dominates every operating point. An offline encoder-decoder can use an entire recording for context and maximize throughput through batching, while a streaming transducer can emit partial text after each short audio chunk. A multilingual foundation model can handle unfamiliar content without task-specific training, while a compact language-specific model can be faster and more accurate on its target domain.

The field’s history is therefore a sequence of changing abstractions:

Scope of a Transcription System #

ASR is one stage in a larger audio pipeline:

• Voice activity detection (VAD) finds regions containing speech and prevents silence from consuming model time.
• Endpointing decides when a live utterance is complete. Aggressive endpointing improves responsiveness but can cut off a speaker after a pause.
• Inverse text normalization (ITN) converts forms such as “twenty twenty-six” into 2026, subject to locale and application rules.
• Forced alignment maps a known transcript back to time intervals in the audio.
• Speaker diarization determines who spoke when. Speaker identification additionally maps an anonymous speaker cluster to an enrolled identity.
• Speech translation changes both language and modality, while audio-language models can answer questions about non-speech events. Neither task is identical to literal transcription.

Pipeline boundaries matter because a low word error rate does not imply accurate speaker labels or timestamps. Overlapping speech can also require speech separation before either recognition or diarization works reliably.

Historical Timeline #

The dates identify influential publications and releases rather than clean era boundaries. Hybrid HMM systems remain deployed, CTC remains important for alignment and efficient decoding, and current systems often combine several generations of ideas.

The Modular Statistical Recognizer #

A classical recognizer searches for the word sequence \(W\) that best explains acoustic observations \(X\):

The acoustic model scores how likely the signal is for a candidate word sequence, while the language model scores the sequence itself. A pronunciation lexicon maps words to phones, HMMs represent progress through phone states, and a decoder searches the composed graph. Gaussian mixture models (GMMs) historically scored acoustic feature frames such as mel-frequency cepstral coefficients.

The design has important strengths:

• lexicons can add names or technical terms without retraining every component;
• grammar constraints can make command recognition deterministic;
• WFST decoding integrates acoustic, pronunciation, and n-gram costs efficiently; and
• HMM state progression supports bounded-memory streaming.

Complexity is the corresponding cost. Training needs alignments, context-dependent phone states, pronunciation resources, several optimization stages, and decoder-graph construction. New languages require linguistic resources, while a pipeline assembled from separately optimized components can accumulate interface errors.

Kaldi remains the defining open toolkit for the architecture. Custom grammars, forced alignment, legacy models, and research on explicit decoders still justify Kaldi, although starting a general-purpose recognizer from scratch is more work than adapting a pretrained end-to-end model.

Neural Acoustic Models and End-to-End Objectives #

DNN-HMM Hybrids #

DNN-HMM recognizers retain HMM temporal structure and WFST decoding but replace GMM scores with neural-network predictions over HMM states. Hinton and collaborators documented substantial gains because deep networks model correlated acoustic features and speaker variation more effectively than mixtures of local Gaussian densities.

Hybrid systems offered a practical migration path: established lexicons and language models remained useful, while acoustic quality improved. However, frame alignments and the modular training stack remained necessary.

Connectionist Temporal Classification #

CTC adds a blank label and sums over every monotonic alignment that collapses to the target transcript. Training therefore requires paired audio and text but not the exact time of each character or subword.

Advantages follow from the monotonic formulation:

• encoder frames can be computed in parallel;
• greedy decoding is inexpensive;
• causal or chunked encoders can stream;
• frame-to-token paths support timestamps; and
• CTC works well as an auxiliary alignment loss in larger systems.

CTC assumes that output labels are conditionally independent given the encoder representation. An external language model, prefix beam search, or second-pass decoder is often added because the acoustic encoder alone does not model rich output dependencies.

Attention Encoder-Decoders #

An attention encoder-decoder maps acoustic frames to a latent sequence and generates text autoregressively. LAS established the approach for speech, and Whisper later scaled a Transformer version with multilingual and multitask training.

The decoder conditions each new token on previous tokens and selected encoder states. Rich context improves punctuation, spelling, and phrase-level disambiguation without an explicit pronunciation lexicon. Task and language tokens can also select transcription, translation, or language identification.

Autoregressive decoding is sequential, and unrestricted attention can depend on future audio. Long recordings therefore need windowing, chunking, or specialized attention. A powerful output-language model can also emit plausible words when the acoustic evidence is silence, noise, or music. Whisper’s model card warns about hallucinated and repetitive text, uneven performance across languages and accents, and the absence of robust native diarization.

Transducers #

RNN-T adds a prediction network over prior output labels to a streaming acoustic encoder. A joint network decides whether to emit a token or a blank that advances audio time. Recognition can proceed from an audio prefix because alignment remains monotonic.

RNN-T provides a strong balance for voice interfaces:

• token history gives more linguistic context than plain CTC;
• cached encoder state bounds repeated work;
• partial text can arrive before the utterance ends; and
• no separate pronunciation lexicon is required.

Training and decoding are more complex than CTC, and blank/token alignment can create repeated-token errors. TDT predicts token durations so the decoder can skip several frames at once, while FastEmit encourages earlier emissions.

Transformers, Conformers, and Cached Streaming #

Transformers replaced recurrence with self-attention, improving training parallelism and long-range context. Full attention over \(n\) audio positions has \(O(n^2)\) pairwise cost, so unrestricted Transformers become expensive for long recordings.

Conformer blocks combine:

1. feed-forward transformations;
2. multi-head self-attention for broad context; and
3. convolution for local phonetic patterns.

The combination is effective because speech contains both short acoustic events and dependencies spanning words or sentences. FastConformer reduces the sequence length and attention cost, while cache-aware variants retain keys, values, and convolution state from earlier chunks.

Streaming models limit future context to a lookahead window. More right context can improve recognition because later audio resolves ambiguous phonemes and punctuation, but final output arrives later. NVIDIA’s Nemotron 3.5 ASR model card illustrates the tradeoff with configurable chunks from 80 ms to 1.12 seconds. Its reported average WER across 19 transcription-ready locales falls from 10.38% at 80 ms to 8.84% at 1.12 seconds, under the card’s evaluation conditions. The figures form a latency-quality curve, not a universal benchmark against unrelated systems.

Self-Supervision and Foundation Models #

Large transcribed corpora are expensive because accurate labels require human review. Self-supervised learning instead constructs prediction targets from unlabeled audio.

• wav2vec 2.0 masks latent speech spans and contrasts the correct quantized representation against distractors. Fine-tuning then adds a CTC recognizer.
• HuBERT clusters acoustic features into hidden units and predicts masked cluster labels.
• WavLM adds denoising and overlapping-speech objectives, producing representations useful for ASR, speaker verification, separation, and diarization.
• XLS-R scales cross-lingual pretraining to 436,000 hours in 128 languages.

Self-supervision reduces the amount of labeled speech needed for adaptation, but a pretrained encoder is not a complete transcription product. Tokenization, decoding, fine-tuning, endpointing, text normalization, and language-specific evaluation remain necessary.

Weak supervision takes a different path. Whisper trains directly on large quantities of audio paired with Internet-derived transcripts, accepting noisy labels in exchange for scale and diversity. Robustness improved across domains, while label noise and autoregressive generation introduced recognizable failure modes.

Multilingual scale also expanded:

• MMS uses language adapters with wav2vec-style encoders and lists ASR checkpoints for more than 1,100 languages.
• SeamlessM4T unifies ASR with speech-to-text, text-to-text, text-to-speech, and speech-to-speech translation.
• speech-language models such as Qwen2-Audio accept audio plus text instructions for transcription and audio understanding.
• Canary-Qwen joins a FastConformer speech encoder to a language model, allowing recognition and transcript post-processing in one model family.

Nominal language count is not a quality guarantee. Training hours, writing systems, accents, domain coverage, and tokenization differ by language. MMS also uses CC BY-NC 4.0 weights, while Whisper’s code and weights use MIT; deployment decisions must therefore examine model licenses separately from repository code licenses.

Measuring Quality, Speed, and Latency #

Recognition Quality #

Word error rate (WER) is:

where \(S\), \(D\), and \(I\) are substitutions, deletions, and insertions, and \(N\) is the number of reference words. Character error rate (CER) is more informative where word segmentation is not explicit or where character-level correctness matters.

WER comparisons are valid only when evaluation conditions match. Text normalization can decide whether 2026, twenty twenty-six, and twenty twenty six are equivalent. Punctuation, casing, dataset domain, language, decoding method, VAD, and speaker overlap also change the result.

Additional metrics cover different requirements:

• diarization error rate (DER) measures speaker segmentation and assignment;
• time to first token measures initial responsiveness;
• stable-partial latency measures when displayed words stop changing;
• endpointing delay measures the wait after the speaker finishes;
• final latency includes post-processing and transport;
• real-time factor and RTFx measure offline processing rate; and
• concurrent streams measure serving capacity under a latency objective.

For processing time \(T_p\) and audio duration \(T_a\):

An RTFx of 20 means that one worker processes 20 seconds of audio per second under the measured configuration. RTFx does not reveal first-token latency because batching can increase throughput while delaying each individual result.

Reported Speed-Quality Examples #

Benchmark tables should always record hardware, precision, batch size, audio length, model revision, language, VAD, decoding parameters, and pipeline scope. Omitting those fields turns a measurement into an ambiguous claim.

Offline and Streaming Designs #

Offline Transcription #

Offline systems can examine future audio, batch recordings, use bidirectional encoders, run beam search, and apply second-pass language models or alignment. Archives, subtitles, legal review, and media indexing benefit from throughput and final accuracy more than immediate partial results.

Batching improves accelerator utilization because many encoder windows execute together. Quantization reduces memory bandwidth and can improve CPU or GPU throughput. VAD avoids encoding long silent regions. However, a batch-oriented service can consume significant memory and delay short files behind long jobs, so schedulers need duration-aware queues and bounded batches.

Streaming Transcription #

Streaming systems process fixed or adaptive chunks and preserve state between calls. A complete live path includes:

capture -> resample -> VAD -> encoder cache -> decoder -> partial text
        -> endpointing -> finalization -> application response

The system must manage more than model inference:

• input buffers need backpressure because audio arrives at a fixed wall-clock rate;
• partial hypotheses need stability rules because revisions disrupt captions and downstream agents;
• reconnects need explicit state recovery;
• long sessions need bounded caches; and
• endpointing must distinguish a pause from the end of a turn.

Whisper can be adapted to streaming by repeatedly transcribing overlapping windows and confirming stable prefixes. WhisperStreaming reports about 3.3 seconds of latency in its demonstrated configuration, which is far above the chunk latency of a native streaming transducer. Reprocessing also increases compute. Native RNN-T, TDT, or cache-aware Conformer models are therefore better starting points when subsecond interaction is a primary requirement.

Open-Source Software and Models #

Open source spans complete recognizers, optimized runtimes, research frameworks, and pipeline components. Repository licenses do not automatically cover every downloadable checkpoint.

Practical Open-Source Compositions #

Different requirements imply different compositions:

1. Private desktop transcription: whisper.cpp provides a native local runtime, while faster-whisper favors Python applications and accelerator throughput.
2. Meetings and interviews: WhisperX combines transcription, VAD, forced alignment, and pyannote diarization. Every component must be evaluated on overlap and microphone conditions.
3. Embedded commands: Vosk or a small sherpa-onnx model can provide bounded-memory streaming; a constrained grammar may outperform a general foundation model for a fixed command set.
4. Multilingual archives: Whisper provides broad general coverage, MMS targets long-tail languages, and Seamless adds translation. License and per-language evaluation determine feasibility.
5. High-throughput GPU service: NeMo models, faster-whisper, or FunASR expose batching and optimized inference paths.
6. Speech research and adaptation: ESPnet, SpeechBrain, Kaldi, WeNet, and NeMo provide training recipes rather than only fixed inference APIs.

Bleeding-Edge Directions #

Controllable-Latency Streaming #

Cache-aware models now expose the latency-quality operating point rather than fixing it at training time. Nemotron 3.5 ASR can select chunk sizes from 80 ms to 1.12 seconds, and Moonshine v2 uses sliding-window attention and streaming caches for edge devices. Dynamic systems can choose a mode according to device load, network conditions, or conversational state.

Model-integrated turn detection is another direction. Deepgram Flux couples transcription with end-of-turn prediction and exposes thresholds that trade early responses against premature turn completion. Integrated endpointing can use lexical and acoustic context, while separate VAD remains simpler to inspect and replace.

Promptable and Context-Aware Recognition #

Speech APIs increasingly accept key terms, phrase lists, spelling context, or natural-language prompts. Prompting can improve rare names and domain terminology because the decoder receives application context. Prompt injection through spoken content, over-biasing toward supplied terms, and unpredictable normalization require tests before prompts control downstream automation.

AssemblyAI’s Universal-3 Pro describes prompts for domain vocabulary, disfluencies, audio tags, speaker roles, and code-switching. Google Chirp 3, Azure Speech, Deepgram Nova-3, and current OpenAI transcription APIs expose different forms of adaptation or context. Interfaces and language support change faster than open model architectures, so application code should isolate provider-specific options.

Speech Encoders Joined to Language Models #

Canary-Qwen and Qwen2-Audio represent a broader pattern: a speech encoder maps audio into representations consumed by a language model. One system can then transcribe, normalize, summarize, translate, or answer questions about audio.

Unified models reduce pipeline handoffs and can use broad textual knowledge. They also make literalness harder to guarantee because an instruction-following decoder may paraphrase or infer instead of preserving every spoken word. Regulated transcripts, captions, and evidence processing should retain the audio, raw transcript, timestamps, model revision, and transformation history.

Full-Duplex Speech Interaction #

Voice agents are moving from a serial pipeline—ASR, language model, then text-to-speech—to models that listen and speak concurrently. Full-duplex interaction can handle interruptions and backchannels more naturally, but transcription becomes one internal signal rather than the sole interface. Evaluation must add interruption handling, turn-taking, semantic latency, and recovery from false starts to WER.

Managed Services #

Managed APIs reduce model-serving work and can provide diarization, redaction, phrase adaptation, and regional deployment:

• OpenAI transcription includes Whisper and newer transcription or diarization models with model-dependent streaming and response formats.
• Google Cloud Chirp 3 supports batch, synchronous, and streaming recognition, with language-dependent adaptation and diarization features.
• Azure Speech to Text offers real-time, fast synchronous, batch, and LLM-based transcription paths.
• Deepgram separates general-purpose Nova models from Flux for conversational streaming and endpointing.
• AssemblyAI Universal-3 Pro emphasizes promptable transcription and code-switching.

Cloud selection must include retention policy, training-data policy, encryption, region availability, model-version pinning, outage behavior, and the cost of sending long silent or multi-channel recordings. Local VAD can reduce bandwidth, but aggressive filtering can permanently discard quiet speech.

Selecting an Architecture #

A defensible evaluation process is:

1. Collect representative audio with target microphones, accents, languages, noise, overlap, pauses, and domain terms.
2. Define quality slices instead of one aggregate WER. Names, numbers, commands, and low-resource languages may have different failure costs.
3. Measure end-to-end latency, memory, power, concurrency, and cost on deployment hardware.
4. Test silence, music, truncated utterances, code-switching, and prompt-bias failure modes.
5. Evaluate diarization and timestamps independently from text accuracy.
6. Record exact model revisions, normalization rules, licenses, and provider data policies.

ASR progress has shifted the engineering problem from constructing every acoustic component to selecting and adapting pretrained systems. The central constraint has not disappeared: more context and larger models can improve recognition, while low latency, privacy, language breadth, predictable output, and deployment cost pull the system toward different operating points. Architecture selection should therefore follow measured application requirements rather than a single leaderboard score.