Whisper large-v3 streaming on Apple Silicon (MLX), ~1 s per word

Live transcription with Whisper large-v3 on the Mac GPU. On real talks (mTEDx, Portuguese and French), words appear 1.08–1.15 s after they are said (median), 90 % within 1.6 s, and are never revised once shown. The transcript is as accurate as the normal Whisper large-v3 transcribing the whole talk offline, and as accurate as NVIDIA Parakeet TDT 0.6B v3 (WER pt 7.7 vs 7.4 %, fr 6.3 vs 6.3 %), whose final words arrive 1–1.7 s later (median).

This repository holds the code, the measurements and the 8-bit MLX weights (1.7 GB). Made for ucast.me (live subtitles), as an experiment.

Use

pip install -r requirements.txt   # mlx, mlx-whisper, numpy, huggingface_hub
from whisper_streaming_mlx import StreamingWhisper

stream = StreamingWhisper(model="marcosremar2/whisper-large-v3-streaming-mlx", language="pt")  # 8-bit weights from this repo
for chunk in microphone_chunks():                 # float32, 16 kHz, mono; ~0.5 s each
    stream.insert_audio(chunk)
    for word in stream.process():                 # words that are now safe to show
        print(word.text, end=" ", flush=True)
for word in stream.finish():                      # end of the stream
    print(word.text, end=" ")

Call process() every chunk_s (0.5 s) of new audio, or as soon as the previous call returns if it took longer. model= takes any mlx-whisper model (e.g. mlx-community/whisper-large-v3-mlx-8bit) or a Transformers Whisper checkpoint, which is converted to MLX once into ~/.cache/whisper-streaming-mlx (WHISPER_STREAMING_MLX_CACHE). StreamingConfig holds the knobs: chunk_s 0.5, frame_threshold 15, audio_max_s 8, audio_min_s 1.0, max_context_tokens 40, smart_hold, drop_hallucinations (on), and two options measured below that are off by default: beams (beam search) and vad (Silero VAD, needs onnxruntime; the model is downloaded once). WHISPER_STREAMING_MLX_DEBUG=1 prints one line per pass (stop reason and text).

Simulate live streaming from a file and measure it:

python simulate.py speech.wav --language pt --out words.jsonl
python eval/ref_times.py mlx-community/whisper-large-v3-mlx pt speech.wav ref.jsonl   # reference word times
python eval/latency.py ref.jsonl words.jsonl
python eval/wer.py reference.txt hypothesis.txt
python convert_hf_to_mlx.py MahmoudAshraf/acft-whisper-large-v3 out/acft-mlx-8bit --bits 8   # by hand

How the error stays low

Live Whisper usually loses accuracy because it hears a few seconds at a time, not the 30 s windows it was trained on. Three things keep the error close to transcribing the whole talk:

  1. A model trained for short audio. The encoder runs on 8 s or less. On real talks the stock large-v3 fell into repetition loops this way: 6,157 words for a talk of 754, and three other talks stalled. ACFT is the same model, with the same size and speed, after a short extra training on short, variable audio contexts. It never looped in these runs. On an RTX 3060 in fp16, ACFT gave pt 8.2 % against 17.7 % and fr 7.3 % against 10.6 %, and invented far fewer words (pt 35 against 127, fr 52 against 91).
  2. A word is shown only when the audio behind it has arrived. AlignAtt watches where the model is reading and stops 0.3 s before the end of the audio, so words are not guessed from cut speech. The last word is held back, and no words come out before 1 s of audio.
  3. Guards for real speech. Repeated blocks are cut, each pass has a word limit, and known Whisper hallucinations are dropped.

Both parts matter. Whole files tie (fr 5.39 % each, pt 6.25 % against 5.78 %), so ACFT is not a better model in general. Without AlignAtt, on chunks of up to 5 s in faster-whisper int8, the two also tie (pt 9.45 % against 9.99 %, fr 8.39 % against 7.98 %). The cause of the loops and why the extra training removes them were not tested directly.

How it works

Every chunk_s of new audio:

  1. No words before the buffer holds audio_min_s (1 s) of audio (as SimulStreaming's audio_min_len). Words are never revised, and the first ones decided from 0.5 s of a talk that starts mid-sentence came out "Com de chance a chance é saber a lei" for "Quando eu tinha sete anos, eu não sabia ler".
  2. The buffer grows by the new audio and keeps at most audio_max_s (8 s). Older audio is dropped 1 s at a time; its words go back to the decoder as the prompt (max_context_tokens), except words aligned within 1 s of the cut (the aligned times are approximate, and a word whose audio is still in the buffer would be written twice).
  3. The encoder runs on the real audio only (rounded up to whole seconds) instead of padding it to 30 s, like whisper.cpp's audio_ctx. This is the biggest saving: 30 s cost 882 ms on an M4, 8 s about 200 ms. It is also why the ACFT model is the default: Whisper was trained on 30 s windows only.
  4. The decoder continues from the words already shown (forced prefix), greedily.
  5. AlignAtt: before each token, the cross-attention of Whisper's alignment heads (from openai/whisper) tells where in the audio the model is reading. Within frame_threshold frames (15 × 20 ms = 0.3 s) of the end of the audio, decoding stops and waits for more audio.
  6. The last word is held back (it may be cut by the end of the buffer), unless an end of text came with at least 0.5 s of silence after it — an end of text right at the edge is not trusted: Whisper also ends the text when a word is cut.
  7. Words are never revised, so nothing waits for two passes to agree (LocalAgreement, as in whisper_streaming, Macháček et al., 2023).

Guards that real speech needed: a block of tokens repeated three times (or one token four times) is cut and the pass stops; a pass writes at most 10 + 25 tokens per second of new audio (speech is ~5); the text never exceeds the decoder's 448-token context. Also: every buffer length is run once at load (the first passes at a new length were 4–5 s late without it), the attention is read only over frames with real audio, and finish() adds 0.5 s of silence before the last pass so the last word is not dropped. Known Whisper hallucinations ("Sous-titrage ST' 501", "Obrigado por assistir", Amara.org credits…) are dropped when they make up a whole sentence.

Options (off by default): beams runs a beam search, stopping on the best hypothesis' attention as SimulStreaming; vad runs Silero VAD on the incoming audio — passes with no new speech are skipped, a 0.5 s pause releases the held-back word, and 2 s of silence close the buffer.

Benchmarks

Apple M4 (24 GB), macOS, mlx 0.32.4, mlx-whisper 0.4.3, other work keeping the machine at load 3–5 (7–18 during the audio-minimum runs, which moves latency, not WER). Default configuration (chunk 0.5 s, threshold 15 frames, buffer 8 s, audio minimum 1 s, context 40 tokens) unless said. The simulation is computation-aware: audio arrives in real time and the clock advances by the real compute time, so a slow pass delays everything after it. Latency of a word = when it was shown − when it was said (end of the word in offline mlx-whisper large-v3 word timestamps, DTW). One run per configuration on real speech.

Real talks (mTEDx)

Four 5-minute continuous excerpts of the mTEDx test set (openslr.org/100) with human transcripts — Portuguese 2EpIZVU-N3Y, dfHIJD90Q5s (1,281 words) and French 0u7tTptBo9I, 16MWHwqz0mQ (1,466 words). WER after lower-casing, spelling numbers out, removing punctuation, hesitations (uh, euh, hum…) and notes like "(Applause)", both talks of a language pooled.

Accuracy:

system live? WER pt WER fr
Whisper large-v3, fp16, whole talk offline (mlx-whisper transcribe defaults) no 16.3 % ¹ 6.8 %
Whisper large-v3, 8-bit, whole talk offline no 8.8 % 6.4 %
ACFT large-v3, 8-bit, whole talk offline no 12.2 % 7.8 %
Parakeet TDT 0.6B v3 (parakeet-ultra) in the ucast.me app pipeline ² yes 7.4 % 6.3 %
streaming, Whisper large-v3 8-bit, no audio minimum yes 13.0 % 10.5 %
streaming, ACFT, no audio minimum (2 runs) yes 8.4 / 8.0 % 6.4 / 6.7 %
streaming, ACFT, audio minimum 1 s (default) yes 7.7 % 6.3 %
streaming, ACFT, audio minimum 1.5 s yes 7.6 % 7.1 %
streaming, ACFT, beam 3 yes 8.4 % 7.6 %
streaming, ACFT, VAD yes 9.5 % 7.3 %
streaming, ACFT, VAD + beam 3 yes 7.5 % 8.3 %

¹ On dfHIJD90Q5s the offline large-v3 invented whole passages ("se a gente vive o que quiser" twice, "pushpa nu detect…"): 19.8 % there, 12.9 % on the other talk — Whisper's known hallucinations on long audio. Offline transcription took 71–167 s per 5-minute talk.

² CPU, 4 threads, translating every line into two languages at the same time; it transcribes whole stretches of speech (cut at pauses, at most 5 s) and shows drafts in between.

Two identical streaming runs differed by ~0.4 point, so differences under ~1 point are noise. Most errors are the same for Parakeet and Whisper (16–29 per talk); the rest are style ("pra" vs "para", "tá" vs "está"), French singular/plural (Parakeet), and words split at the buffer edge ("da quilo", "só oro": streaming).

Latency per word (range over the four talks):

system median p90 max words > 2 s
streaming, ACFT, audio minimum 1 s (default) 1.08–1.15 s ≤ 1.59 s 2.20–2.68 s 0.9–2.5 %
streaming, ACFT, no audio minimum 1.05–1.11 s ≤ 1.45 s 2.31–2.65 s 1.2–2.4 %
streaming, ACFT, VAD 1.05–1.18 s ≤ 1.57 s 2.32–2.99 s 0.3–1.1 %
streaming, Whisper large-v3 8-bit (stock) 1.07–1.18 s ≤ 1.57 s 2.29–7.35 s 0.6–5.8 %
Parakeet, first draft showing the word (may still change) 1.04–1.10 s ≤ 1.97 s 4.1–8.7 s 3.8–9.9 %
Parakeet, final version of the line 2.01–2.79 s ≤ 7.78 s 10.6–15.1 s 51–65 %

A pass of the default takes 360–470 ms on the M4.

Before the repetition guard, the stock large-v3 wrote "rentrer en relation" 6,157 words long on a 754-word French talk and crashed on the three others (decoder context exceeded). ACFT never looped. Every run is in results/benchmark_real_mtedx.json (metrics only: the talks are CC-BY-NC-ND-4.0).

Synthetic talks

Two short talks with known text — Portuguese (50 s, 122 words) and French (38 s, 126 words) — generated with Kyutai Pocket TTS: short replies, proper names, sentences glued together. ~120 words, so one wrong word is 0.8 % of WER.

system WER pt WER fr median max
ACFT 8-bit, with the real-speech guards, no audio minimum 4.1 % 0.8 % 1.15–1.23 s 2.03–2.14 s
ACFT 8-bit, before the real-speech guards (2 runs) 2.5 / 4.1 % 0.8 / 2.5 % 0.96–1.06 s 1.65–2.07 s
Whisper large-v3 8-bit, same config (2 runs) 0.0 / 0.0 % 3.3 / 3.3 % 0.99–1.17 s 1.56–1.97 s

results/benchmark_synthetic_acft.json, results/benchmark_m_series.json (the earlier large-v3 default).

NVIDIA GPU (RTX 3060, PyTorch fp16)

ACFT against the stock large-v3 with no MLX and no 8-bit conversion: one RTX 3060 (12 GB), Linux, Transformers, fp16, same streaming logic and defaults, the same four talks and two synthetic ones. Streaming runs in real time; "whole file" decodes in 30 s windows. WER summed per group (errors / reference words), one run each:

group streaming ACFT streaming large-v3 whole file ACFT whole file large-v3
mTEDx fr (1466 words) 7.30 % 10.64 % 5.39 % 5.39 %
mTEDx pt (1281 words) 8.20 % 17.72 % 6.25 % 5.78 %
synthetic (241 words) 2.49 % 2.90 % 0.83 % 0.83 %
  • Streaming: ACFT invents far fewer words (pt 35 vs 127, fr 52 vs 91).
  • Whole file: the two are level. The gap ACFT shows on the Mac above (pt 12.2 vs 8.8 %) comes from the 8-bit MLX conversion, not from the model: in fp16 it is 6 words out of 1281.
  • Speed: the 3060 keeps up with both: a pass takes 244–294 ms (ACFT) and 208–298 ms (large-v3) at the median, and the transcript ends at most 0.88 s behind the talk. A whole 5-minute talk takes 53–79 s.
  • Not measured here: the latency of each word. faster-whisper was not tried: both models run in Transformers, so this compares the models, not a production server. That port is not in this repository, which stays MLX.

Every run is in results/benchmark_rtx3060.json (metrics only).

How we got here

Same synthetic audio, same machine, Whisper large-v3 (median / max latency):

step median max
LocalAgreement-2 (as whisper_streaming), CPU, faster-whisper int8, beam 5 79–99 s 102–117 s
LocalAgreement-2, Mac GPU (mlx-whisper), 15 s buffer 11–12 s 19–21 s
AlignAtt, encoder padded to 30 s, 1 s chunks 2.8–2.9 s 4.6–5.2 s
+ encoder on the real audio, 10 s buffer, 0.5 s chunks 1.5 s 3.0 s
+ warm-up, attention over real audio only, hold back only at the attention stop, 8-bit 1.3 s 2.0–2.4 s
+ threshold 0.3 s, 8 s buffer, safe buffer cut 1.0 s 1.6–2.0 s
+ ACFT and the real-speech guards 1.0–1.2 s 2.0–2.1 s (synthetic), 2.5–3.2 s (real)
+ 1 s audio minimum (this version; real talks) 1.08–1.15 s 2.2–2.7 s

For scale: the same large-v3 one sentence at a time (not streaming) takes 1.6–3.0 s per sentence on this GPU and 2.9–4.7 s on the CPU (ONNX int8, 4 threads).

Tried and not kept

  • large-v3-turbo (4 decoder layers): repetition loops with this policy ("Claro. Claro. Claro…", WER > 100 %); its alignment heads do not hold the AlignAtt stop.
  • Normalized attention (z-score over tokens + median filter, as SimulStreaming): with the hold-back rule above it stalled the French talk for 10 s at the start.
  • 6 s buffer with the stock large-v3: WER 6–8 % (ACFT tolerates it: 0.8–2.5 %, max 1.8–2.2 s). 0.3 s chunks: no lower maximum. 8-bit vs fp16: same speed (~500 ms per pass), kept for memory.
  • Quality setting (10 s buffer, 0.5 s threshold): fewer errors on synthetic speech (ACFT fr 0.8 %), but 2.2–2.3 s maximum.
  • Beam search (3 beams): no gain on real talks (pt 8.4 %, fr 7.6 %), ~60 ms more per pass. Kept as an option.
  • Silero VAD: half as many words over 2 s (0.3–1.1 %), but more errors in Portuguese (9.5 %). A first version that restarted the buffer at every 0.5 s pause made Whisper invent sentences from the first fraction of a second of each new stretch; the kept version only restarts after 2 s of silence. Kept as an option — useful with a live microphone and long silences, which these talks do not have.
  • 1.5 s audio minimum: the start of the talk nearly right ("quando tinha sete anos"), but no better overall (pt 7.6 %, fr 7.1 %) and a longer tail (max 3.1 s).

Limits

  • One M4 and one RTX 3060, two languages, four real talks and two synthetic ones, one run each on real speech. Noisy rooms, distant microphones, accents, conversation and sessions longer than 5 minutes are not measured.
  • On real talks 1–2.5 % of the words take more than 2 s (at most 2.7 s with the default).
  • The first second of each stretch of speech waits for the audio minimum.
  • Word end times from the attention are approximate; they only decide what leaves the buffer.
  • The latency is transcription only (no translation, no display).
  • Runs vary: the weights published here gave pt 7.7 % and fr 6.6 % (2026-11-05). One Portuguese talk gave 15.4 % in a run on a heavily loaded Mac and 10.8 % when repeated.
  • These 8-bit weights are for streaming. Transcribing a whole talk offline with them was worse than the fp16 model (see the benchmarks).

Ideas not tried yet

  • A block-causal encoder with a key/value cache, so each pass encodes only the new audio (WhisperKit, CarelessWhisper, U2-Whisper) — needs fine-tuning.
  • Simul-Whisper's integrate-and-fire truncation detector to decide whether the last word is complete — no checkpoint exists for large-v3.

References

Licence

MIT for the code in this repository (LICENSE). The weights are a converted and 8-bit quantized copy of the ACFT checkpoint by Mahmoud Ashraf, under Apache-2.0 (LICENSE-weights). Whisper large-v3 and the ACFT checkpoint are Apache-2.0 (their model cards); AlignAtt as implemented in SimulStreaming (MIT) and the alignment heads from openai/whisper (MIT) were the starting point.

Downloads last month
116
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for marcosremar2/whisper-large-v3-streaming-mlx

Finetuned
(2)
this model

Papers for marcosremar2/whisper-large-v3-streaming-mlx