● recastr.io  /  benchmark field notes

Anatomy of a transcription benchmark

How eight speech-to-text models — five vendors, two philosophies — get run through the same eight fixtures, scored on the same four axes, and averaged over ten full passes before a single chart gets drawn.

8 models 5 vendors 8 fixtures 10× repeated runs 4 scoring axes

One fixture's trip through the pipeline

01 / 04

Every audio file in the manifest is sent to all eight providers, scored the same way, and the whole pass repeats ten times before anything is averaged into a chart.

manifest.jsonl · fixture two-speaker-dialogue.mp3 fixture_type: multi_speaker fan out · diarize=True whisper-1 gpt-4o-transcribe gpt-4o-mini-transcribe gpt-4o-transcribe-diarize gpt-audio gemini-3.7-flash grok-stt azure-fast-transcription EvalTranscript{text, words, cost, duration} SCORING · scripts/transcription_alignment.py jiwer.wer(reference, hypothesis) → word error rate keyterm_miss_rate(keyterms, text) → fraction of named terms missed align_words() → |Δ start time| → median / p95 timestamp drift diarization_accuracy() · best permutation → speaker-label accuracy, or null fixture_report{} appended · one per provider run-N.json all 8 fixtures × 8 models, one pass repeat × 10 runs aggregate_benchmark_runs.py mean per (provider, fixture) · failed calls skipped, not zeroed plot_multimodal_transcription_benchmark.py
One fixture, fanned out to all eight providers, scored on four axes, then folded into a per-run report. The whole left-hand loop — every fixture against every model — runs ten times before aggregate_benchmark_runs.py averages matching (provider, fixture) pairs across runs and hands the result to the plotting script.

Two philosophies, eight models

02 / 04

Dedicated ASR models are purpose-built for transcription. General multimodal chat models are asked to transcribe via a prompt. The split matters most in what each side can actually output — not every model returns a word timestamp to hang a subtitle cue on.

Dedicated ASR built to transcribe Multimodal chat models prompted: "transcribe this" whisper-1 OpenAI ● word timestamps no diarization gpt-4o-transcribe OpenAI ○ no timestamps no diarization gpt-4o-mini-transcribe OpenAI ○ no timestamps no diarization gpt-4o-transcribe-diarize OpenAI ◐ segment-level only native diarization grok-stt xAI ● word timestamps native diarization azure-fast-transcription Microsoft ● word timestamps native, up to 35 speakers gpt-audio OpenAI · gpt-audio-2025-08-28 ○ no timestamps ◐ promptable diarization gemini-3.7-flash Google ○ no timestamps ◐ promptable diarization (2 of 8 — the rest of the fleet is dedicated ASR) only 3 of 8 models return word-level timestamps: whisper-1 · grok-stt · azure-fast-transcription — the rest can't reliably time an SRT cue
Vendor color marks the provider; the right-hand annotation is the capability the WER number alone hides. Word-level timestamps gate whether a model can produce a properly timed subtitle file at all — five of the eight can't, regardless of how low their WER scores.

Two kinds of hard, on purpose

03 / 04

Six fixtures are synthesized with exact-by-construction ground truth. Two are real archival recordings, chosen from different eras specifically to separate "real-world audio is hard" from "old audio is hard."

manifest.jsonl — 8 fixtures Synthetic tts-1 · exact ground truth clean-narration · numbers-and-names · technical-jargon noisy-narration — white noise overlay two- and three-speaker dialogue — scripted single_speaker × 3 · noisy × 1 · multi_speaker × 2 Real-world officially-transcribed archives Fountain Hughes interview Library of Congress AFC · 1949 317 words · 2 speakers NASA "Houston We Have a Podcast" episode 414 · 2026, studio-recorded 876 words · 2 speakers 1949 × 2026 — two eras, two recording qualities if only the 1949 clip were hard, it'd be an "old audio" problem. Both clips degrade WER 1.6×–4× — it isn't. a third clip — a 1962 White House Dictabelt recording — was tried and dropped: one speaker was barely audible even to a human listener
Word and speaker counts are the actual reference_words ground truth used to score every model. The two real-world clips are never averaged together in the per-model bar charts — spanning 77 years of recording technology, blending them would hide exactly the variation the comparison exists to surface.

How a diarized transcript gets graded

04 / 05

Provider speaker labels are arbitrary — xAI returns integers, prompted models return "Speaker 1" strings — and have no reason to line up with the reference A/B labels. The scorer brute-forces the label mapping that scores best, then reports that number.

reference_words — ground truth budget A marketing A ten B percent B provider words — arbitrary label IDs budget 1 marketing 1 ten 0 percent 0 align_words() — greedy in-order text match, speaker-blind diarization_accuracy() — try every label mapping, keep the best {0→A, 1→B} 2 of 4 words match → 0.50 {0→B, 1→A} 4 of 4 words match → 1.00 ✓ kept brute-force over label permutations — fine at 2–3 speakers; would need the Hungarian algorithm at larger counts. Word-count accuracy, not time-weighted DER. diarization_accuracy: 1.00
The same mechanism scores WER (via jiwer) and timestamp drift alongside diarization — align_words() matches text greedily in reference order, blind to speaker; only the diarization step needs the extra permutation search, since it's the one axis where the provider's labels don't share a vocabulary with the reference's.

What ten runs actually show

05 / 06

Every number below is a mean across the 10 repeated runs described in chapter one — not a single noisy pass. Sorted by synthetic-audio WER, best first.

Lowest synthetic WER

5.1% · gpt-audio

Cheapest per minute

$0.0017grok-stt

Most consistent diarizer

Δ0.00azure, 1949 vs. 2026

Can produce a timed SRT

3of 8 models

Word error rate, latency, cost

Per-model WER, latency, and cost averaged over 10 benchmark runs
Model Synthetic WER 1949 Interview WER 2026 Interview WER Latency $ / min
gpt-audio 5.1% 29.8% 10.7% 3.4s $0.0214
azure-fast-transcription 7.1% 33.1% 13.1% 3.3s $0.0060
gemini-3.7-flash 7.2% 33.9% 20.3% 11.6s $0.0021
gpt-4o-transcribe 7.2% 33.7% 22.0% 3.2s $0.0035
gpt-4o-mini-transcribe 7.9% 33.0% 11.3% 1.9s $0.0018
whisper-1 8.5% 30.9% 12.2% 4.0s $0.0060
grok-stt 8.9% 35.6% 23.3% 0.9s $0.0017
gpt-4o-transcribe-diarize 15.4% 33.9% 15.7% 22.1s $0.0192
Each WER column's bars scale to that column's own worst model (synthetic max 15.4%, 1949 max 35.6%, 2026 max 23.3%) — compare bar length within a column, read the percentage across columns. The 1949 interview is the harder clip for every single model, often by a wide margin over the cleanly-recorded 2026 podcast.

Diarization accuracy

Per-model speaker-diarization accuracy averaged over 10 benchmark runs
Model Synthetic (2–3 speakers) 1949 Interview 2026 Interview
gpt-audio 100% 83.3% 88.5%
azure-fast-transcription 96.1% 92.3% 92.3%
gemini-3.7-flash 100% 85.6% 91.9%
gpt-4o-transcribe — no diarization — —
gpt-4o-mini-transcribe — no diarization — —
whisper-1 — no diarization — —
grok-stt 84.0% 73.7% 92.3%
gpt-4o-transcribe-diarize 99.8% 85.2% 84.6%
Bars scale from 70% to 100% accuracy (not 0%) so the differences that matter are visible; grok-stt's 1949 bar is the one flagged red — a 22-point drop from its 2026 score, the widest swing of any model here, against azure-fast-transcription's flat ~92–96% across all three fixture types. Three models never emit a speaker label at all, regardless of audio type.

Picking a model depends what you're optimizing for

06 / 06

There's no single "best" model in this fleet — the winner changes with the use case. Eight categories, called strictly from the numbers above.

Cost

grok-stt

$0.0017/min — cheapest by a wide margin over the next-cheapest model.

Accuracy (WER, ignoring diarization)

gpt-audio

Lowest WER on every fixture group — 5.1% synthetic, 29.8% on the 1949 clip, 10.7% on the 2026 clip. Wins outright, not just on average.

SRT / timed captions

azure-fast-transcription

One of only 3 models with word-level timestamps at all, and the only one of those 3 with native diarization too — plus the flattest diarization accuracy across eras (92.3% on both real-world clips, no swing).

Noisy audio

gemini-3.7-flash

Tied for a perfect 0% WER on the white-noise clip with 4 other models — cheapest of that five-way tie, so it wins on tiebreak.

Multi-speaker diarization (clean audio)

gpt-audio tied with gemini-3.7-flash

Both hit 100% diarization accuracy on the clean synthetic 2–3 speaker dialogues.

Proper nouns & jargon

gemini-3.7-flash tied with gpt-4o-transcribe

Lowest keyterm miss rate (10.8%) across the numbers/names and technical-jargon fixtures.

Latency-sensitive / real-time

grok-stt

0.93s mean latency — 2× faster than the next-fastest model, ~24× faster than gpt-4o-transcribe-diarize.

Best value (accuracy per dollar)

gpt-4o-mini-transcribe

$0.0018/min — nearly as cheap as grok-stt, but with far better real-world WER (11.3% vs. grok's 23.3% on the 2026 clip).

Cheap and fast isn't the same as cheap and accurate. grok-stt wins Cost and Latency here, but it also has the worst WER of any model on both real-world clips (35.6% on the 1949 interview, 23.3% on the 2026 podcast) — the category it doesn't win is the one most use cases actually care about most.