The Hard Parts of Live Transcription Aren't the Model

"Add transcription to calls" is the kind of feature request that sounds like a weekend. Pipe the audio to a speech-to-text model, render the text, done. The model is the easy 20%. The other 80%, the part that decides whether people trust the transcript enough to stop taking their own notes, is everything that happens around the model in a live, multi-speaker, real-world-audio setting.

Over the past several weeks I built the transcription pipeline behind Flowstate's calls: live captions that stream as people talk, and a clean, readable record after the call ends. This is a writeup of the problems that actually consumed the time, none of which were "call the model", and how I solved them. I'm staying out of our internal architecture on purpose and focusing on the engineering problems, because they generalize to anyone putting streaming speech-to-text in front of real users.

The words it gets wrong are the words that matter

Open speech-to-text models, I'm running a Whisper-family model in streaming mode, are remarkable at conversational English and reliably terrible at exactly the tokens that carry the most meaning: product names, internal jargon, acronyms, and people's names. "Whisper" comes back as "WESPA." A feature name becomes three plausible English words. The model is most confident precisely where it's most wrong, because it's resolving an unfamiliar sound to the nearest familiar phrase.

Generic accuracy doesn't save you here. A transcript that's 97% correct but mangles every proper noun reads as untrustworthy, because the proper nouns are what people scan for. So I built accuracy biasing in two layers.

Live biasing. Streaming STT models accept an initial prompt that nudges decoding toward expected vocabulary. Before each chunk of audio, I assemble that prompt from a canonical list of product and domain terms plus lightweight context about who is actually in the room. On top of that, a small, cheap LLM worker watches the running transcript, buffers roughly 90 seconds of it, and every 30 seconds emits a compact "hints" line, the terms it expects to keep hearing, which folds into the bias for the next chunks. The cadence is tuned to the model's per-chunk latency so the hints stay one step ahead of the speech.

Retroactive correction. The bigger win happens after the call. When the last participant leaves, a stronger model walks the entire transcript with full context (the domain vocabulary, the room, a short brief on each speaker) and rewrites every segment. Live biasing reduces the work this pass has to do; the retroactive pass is what makes the saved record actually correct. Against a fixed recitation script, the post-call pass fixed every product noun and normalized the dates and numbers, in a couple of seconds of compute for a multi-minute call.

Don't train your corrector on the model's own mistakes

The live biasing layer had a subtle, self-inflicted failure that's worth dwelling on, because it's an easy trap. The little worker that produced live hints was originally asked to pull out "salient terms" from the running transcript. But the running transcript is the model's output, mistakes included. So when the STT model heard "Whisper" and wrote "WESPA001," the hint worker faithfully flagged "WESPA001" as a salient term, which biased the next chunk toward "WESPA001," which got flagged again. I had built a loop that reinforced the exact errors it was supposed to fix.

The fix was two changes in stance. First, anchor the hint worker on the canonical vocabulary rather than on whatever showed up in the transcript: when it sees a mangled token sitting where a known term should be, it emits the correct spelling, not the heard one. Second, give it its own previous output and explicitly tell it not to echo tokens that look like mishearings of words it already surfaced, which breaks the WESPA-WESPA-WESPA cycle. The general lesson: any system that corrects a model and then feeds on that model's output needs an external source of truth, or it becomes a mistake amplifier.

Silence and a dropped word look identical

When a streaming STT chunk comes back empty, that empty result is ambiguous. It might mean the audio was genuinely silent, the correct answer. Or it might mean there was clear speech and the model produced nothing anyway: a cold-start glitch, momentary GPU pressure, an unlucky interaction between voice-activity detection and decoding. In the second case you've just lost real speech, and the worst part is that you can't see that you lost it. An empty looks like an empty.

I didn't want to bolt on a second audio-analysis pass just to disambiguate, so I reused a signal I already had. The pipeline already runs voice-activity detection to decide where utterances start and stop, and that detector emits a per-frame probability that the frame contains speech. So I aggregate two cheap numbers per utterance on the client, the fraction of frames that were voiced, and the peak speech probability, and ship them alongside the audio.

Now an empty result is interpretable. Empty with low voice confidence is silence; leave it alone. Empty with high voice confidence is a probable dropped word, so the service does a single second-chance retry against the audio buffer it already retains, after a short delay. If that still comes back empty, the utterance is marked as a "gap," a distinct state, rendered differently from an ordinary empty. That last part matters as much as the retry: instead of silently dropping speech, the system now tells me when it thinks it lost something, which turns an invisible failure into a measurable rate I can tune against. And because the retry is gated on the high-confidence case, the common case (actual silence) never pays the latency cost.

Low latency duplicates your words

To keep captions feeling live, you can't wait for a speaker to finish a sentence. You transcribe overlapping windows of audio as they accumulate. The price of that overlap is duplication: the tail of one segment reappears at the head of the next. Sometimes it's verbatim, "...the transcription pipeline." followed by "transcription pipeline. That's...". Worse, sometimes the next window re-transcribes the overlap with more context and slightly different casing or punctuation, so the repeat isn't even an exact string match.

A naive "drop the duplicated prefix" check fails on that second case. What works is fuzzy token alignment at the seam between two segments. I normalize both sides (lowercase, strip punctuation), then scan the tail of the previous segment for a match anywhere in the next segment, head or middle, allowing a mismatch tolerance that scales with the length of the overlap, so casing drift across a ten-word window still counts as a match. The match resolves into one of two repairs: a pure slice-overlap (drop the duplicated head, keep the earlier text) or a re-transcription with a richer prefix (drop the earlier tail, keep the later, fuller version). Two guard rails keep it honest: a minimum window of two tokens, so coincidental repeats like "the the" don't get eaten, and a maximum of twenty, so a phrase someone legitimately repeats survives. Every merge logs how much it dropped, so over-trimming surfaces instead of hiding.

A wall of fragments is not a transcript

Even with clean, deduplicated segments, the raw stream is hard to read. Each model flush is a short clause, so a 70-second monologue lands as roughly 25 stuttering rows and the reader has to mentally stitch them back into thoughts. The transcript was accurate and nearly unusable.

The fix is to group segments into what I called blobs: same-speaker, near-continuous runs of speech rendered as one paragraph. A blob breaks, sealing the current one and opening a new one, on a speaker change, a pause longer than five seconds, or a one-minute soft cap so a long monologue still chunks into readable pieces. I did the grouping once, on the server, so every consumer (the live caption view, the post-call record, text exports, and anything downstream that reads transcripts) shares the same structure instead of each reimplementing it. The grouped view streams as an additive channel alongside the raw segments, so consumers that wanted raw segments kept working untouched and the readable view was purely additive.

Treat enhancements as derived state, never the source of truth

One principle tied all of this together and saved me repeatedly: the corrected text and the grouped blobs never mutate the original recording. The raw segments are the audit trail and the single source of truth; everything else (corrections, grouping) is derived state computed on top, and it can always be thrown away and recomputed.

That decision pays off in three places. Reversibility: a bug in the fancy correction layer degrades to "you see the raw transcript," not "your data is corrupted." Independence: each layer ships and deploys on its own, and every reader falls back gracefully to raw segments when a derived layer is absent or expired. Honesty: there is always an unmodified record of what the model actually heard, which is exactly what you want the moment someone asks "is this transcript right?" Derived layers are allowed to be clever and occasionally wrong precisely because the thing underneath them is dumb and always correct.

The pattern underneath

None of the hard problems here were the model. They were the consequences of putting a probabilistic, latency-bound component into a real-time system with real users and messy audio: biasing it toward the vocabulary that matters, refusing to let it learn from its own errors, distinguishing "nothing happened" from "I missed it," paying for low latency without paying in duplicates, and turning a stream of fragments into something a person actually wants to read. The model is a component. The product is everything you build around its failure modes.