Instructions to use marcosremar2/whisper-large-v3-streaming-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use marcosremar2/whisper-large-v3-streaming-mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download marcosremar2/whisper-large-v3-streaming-mlx --local-dir whisper-large-v3-streaming-mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Whisper large-v3 streaming on Apple Silicon (MLX), ~1 s per word
Live transcription with Whisper large-v3 on the Mac GPU. On real talks (mTEDx, Portuguese and French), words appear 1.08–1.15 s after they are said (median), 90 % within 1.6 s, and are never revised once shown. The transcript is as accurate as the normal Whisper large-v3 transcribing the whole talk offline, and as accurate as NVIDIA Parakeet TDT 0.6B v3 (WER pt 7.7 vs 7.4 %, fr 6.3 vs 6.3 %), whose final words arrive 1–1.7 s later (median).
- Policy: AlignAtt (Papi et al., 2023), as in Simul-Whisper and ufal SimulStreaming, re-implemented on mlx-whisper and tuned for latency on Apple Silicon.
- Model: Whisper large-v3 fine-tuned for a short audio context with
FUTO's ACFT recipe
(
MahmoudAshraf/acft-whisper-large-v3), converted to 8-bit MLX and included here. With the stock large-v3 this method falls into repetition loops on real speech (see below).
This repository holds the code, the measurements and the 8-bit MLX weights (1.7 GB). Made for ucast.me (live subtitles), as an experiment.
Use
pip install -r requirements.txt # mlx, mlx-whisper, numpy, huggingface_hub
from whisper_streaming_mlx import StreamingWhisper
stream = StreamingWhisper(model="marcosremar2/whisper-large-v3-streaming-mlx", language="pt") # 8-bit weights from this repo
for chunk in microphone_chunks(): # float32, 16 kHz, mono; ~0.5 s each
stream.insert_audio(chunk)
for word in stream.process(): # words that are now safe to show
print(word.text, end=" ", flush=True)
for word in stream.finish(): # end of the stream
print(word.text, end=" ")
Call process() every chunk_s (0.5 s) of new audio, or as soon as the previous call
returns if it took longer. model= takes any mlx-whisper model (e.g.
mlx-community/whisper-large-v3-mlx-8bit) or a Transformers Whisper checkpoint, which is
converted to MLX once into ~/.cache/whisper-streaming-mlx (WHISPER_STREAMING_MLX_CACHE).
StreamingConfig holds the knobs: chunk_s 0.5, frame_threshold 15, audio_max_s 8,
audio_min_s 1.0, max_context_tokens 40, smart_hold, drop_hallucinations (on), and two
options measured below that are off by default: beams (beam search) and vad (Silero VAD,
needs onnxruntime; the model is downloaded once). WHISPER_STREAMING_MLX_DEBUG=1 prints one
line per pass (stop reason and text).
Simulate live streaming from a file and measure it:
python simulate.py speech.wav --language pt --out words.jsonl
python eval/ref_times.py mlx-community/whisper-large-v3-mlx pt speech.wav ref.jsonl # reference word times
python eval/latency.py ref.jsonl words.jsonl
python eval/wer.py reference.txt hypothesis.txt
python convert_hf_to_mlx.py MahmoudAshraf/acft-whisper-large-v3 out/acft-mlx-8bit --bits 8 # by hand
How the error stays low
Live Whisper usually loses accuracy because it hears a few seconds at a time, not the 30 s windows it was trained on. Three things keep the error close to transcribing the whole talk:
- A model trained for short audio. The encoder runs on 8 s or less. On real talks the stock large-v3 fell into repetition loops this way: 6,157 words for a talk of 754, and three other talks stalled. ACFT is the same model, with the same size and speed, after a short extra training on short, variable audio contexts. It never looped in these runs. On an RTX 3060 in fp16, ACFT gave pt 8.2 % against 17.7 % and fr 7.3 % against 10.6 %, and invented far fewer words (pt 35 against 127, fr 52 against 91).
- A word is shown only when the audio behind it has arrived. AlignAtt watches where the model is reading and stops 0.3 s before the end of the audio, so words are not guessed from cut speech. The last word is held back, and no words come out before 1 s of audio.
- Guards for real speech. Repeated blocks are cut, each pass has a word limit, and known Whisper hallucinations are dropped.
Both parts matter. Whole files tie (fr 5.39 % each, pt 6.25 % against 5.78 %), so ACFT is not a better model in general. Without AlignAtt, on chunks of up to 5 s in faster-whisper int8, the two also tie (pt 9.45 % against 9.99 %, fr 8.39 % against 7.98 %). The cause of the loops and why the extra training removes them were not tested directly.
How it works
Every chunk_s of new audio:
- No words before the buffer holds
audio_min_s(1 s) of audio (as SimulStreaming'saudio_min_len). Words are never revised, and the first ones decided from 0.5 s of a talk that starts mid-sentence came out "Com de chance a chance é saber a lei" for "Quando eu tinha sete anos, eu não sabia ler". - The buffer grows by the new audio and keeps at most
audio_max_s(8 s). Older audio is dropped 1 s at a time; its words go back to the decoder as the prompt (max_context_tokens), except words aligned within 1 s of the cut (the aligned times are approximate, and a word whose audio is still in the buffer would be written twice). - The encoder runs on the real audio only (rounded up to whole seconds) instead of padding
it to 30 s, like whisper.cpp's
audio_ctx. This is the biggest saving: 30 s cost 882 ms on an M4, 8 s about 200 ms. It is also why the ACFT model is the default: Whisper was trained on 30 s windows only. - The decoder continues from the words already shown (forced prefix), greedily.
- AlignAtt: before each token, the cross-attention of Whisper's alignment heads
(from openai/whisper) tells
where in the audio the model is reading. Within
frame_thresholdframes (15 × 20 ms = 0.3 s) of the end of the audio, decoding stops and waits for more audio. - The last word is held back (it may be cut by the end of the buffer), unless an end of text came with at least 0.5 s of silence after it — an end of text right at the edge is not trusted: Whisper also ends the text when a word is cut.
- Words are never revised, so nothing waits for two passes to agree (LocalAgreement, as in whisper_streaming, Macháček et al., 2023).
Guards that real speech needed: a block of tokens repeated three times (or one token four times)
is cut and the pass stops; a pass writes at most 10 + 25 tokens per second of new audio (speech
is ~5); the text never exceeds the decoder's 448-token context. Also: every buffer length is run
once at load (the first passes at a new length were 4–5 s late without it), the attention is
read only over frames with real audio, and finish() adds 0.5 s of silence before the last
pass so the last word is not dropped. Known Whisper hallucinations ("Sous-titrage ST' 501",
"Obrigado por assistir", Amara.org credits…) are dropped when they make up a whole sentence.
Options (off by default): beams runs a beam search, stopping on the best hypothesis' attention
as SimulStreaming; vad runs Silero VAD on the incoming audio — passes with no new speech are
skipped, a 0.5 s pause releases the held-back word, and 2 s of silence close the buffer.
Benchmarks
Apple M4 (24 GB), macOS, mlx 0.32.4, mlx-whisper 0.4.3, other work keeping the machine at load 3–5 (7–18 during the audio-minimum runs, which moves latency, not WER). Default configuration (chunk 0.5 s, threshold 15 frames, buffer 8 s, audio minimum 1 s, context 40 tokens) unless said. The simulation is computation-aware: audio arrives in real time and the clock advances by the real compute time, so a slow pass delays everything after it. Latency of a word = when it was shown − when it was said (end of the word in offline mlx-whisper large-v3 word timestamps, DTW). One run per configuration on real speech.
Real talks (mTEDx)
Four 5-minute continuous excerpts of the mTEDx test set
(openslr.org/100) with human transcripts — Portuguese
2EpIZVU-N3Y, dfHIJD90Q5s (1,281 words) and French 0u7tTptBo9I, 16MWHwqz0mQ (1,466 words).
WER after lower-casing, spelling numbers out, removing punctuation, hesitations (uh, euh, hum…)
and notes like "(Applause)", both talks of a language pooled.
Accuracy:
| system | live? | WER pt | WER fr |
|---|---|---|---|
Whisper large-v3, fp16, whole talk offline (mlx-whisper transcribe defaults) |
no | 16.3 % ¹ | 6.8 % |
| Whisper large-v3, 8-bit, whole talk offline | no | 8.8 % | 6.4 % |
| ACFT large-v3, 8-bit, whole talk offline | no | 12.2 % | 7.8 % |
| Parakeet TDT 0.6B v3 (parakeet-ultra) in the ucast.me app pipeline ² | yes | 7.4 % | 6.3 % |
| streaming, Whisper large-v3 8-bit, no audio minimum | yes | 13.0 % | 10.5 % |
| streaming, ACFT, no audio minimum (2 runs) | yes | 8.4 / 8.0 % | 6.4 / 6.7 % |
| streaming, ACFT, audio minimum 1 s (default) | yes | 7.7 % | 6.3 % |
| streaming, ACFT, audio minimum 1.5 s | yes | 7.6 % | 7.1 % |
| streaming, ACFT, beam 3 | yes | 8.4 % | 7.6 % |
| streaming, ACFT, VAD | yes | 9.5 % | 7.3 % |
| streaming, ACFT, VAD + beam 3 | yes | 7.5 % | 8.3 % |
¹ On dfHIJD90Q5s the offline large-v3 invented whole passages ("se a gente vive o que quiser"
twice, "pushpa nu detect…"): 19.8 % there, 12.9 % on the other talk — Whisper's known
hallucinations on long audio. Offline transcription took 71–167 s per 5-minute talk.
² CPU, 4 threads, translating every line into two languages at the same time; it transcribes whole stretches of speech (cut at pauses, at most 5 s) and shows drafts in between.
Two identical streaming runs differed by ~0.4 point, so differences under ~1 point are noise. Most errors are the same for Parakeet and Whisper (16–29 per talk); the rest are style ("pra" vs "para", "tá" vs "está"), French singular/plural (Parakeet), and words split at the buffer edge ("da quilo", "só oro": streaming).
Latency per word (range over the four talks):
| system | median | p90 | max | words > 2 s |
|---|---|---|---|---|
| streaming, ACFT, audio minimum 1 s (default) | 1.08–1.15 s | ≤ 1.59 s | 2.20–2.68 s | 0.9–2.5 % |
| streaming, ACFT, no audio minimum | 1.05–1.11 s | ≤ 1.45 s | 2.31–2.65 s | 1.2–2.4 % |
| streaming, ACFT, VAD | 1.05–1.18 s | ≤ 1.57 s | 2.32–2.99 s | 0.3–1.1 % |
| streaming, Whisper large-v3 8-bit (stock) | 1.07–1.18 s | ≤ 1.57 s | 2.29–7.35 s | 0.6–5.8 % |
| Parakeet, first draft showing the word (may still change) | 1.04–1.10 s | ≤ 1.97 s | 4.1–8.7 s | 3.8–9.9 % |
| Parakeet, final version of the line | 2.01–2.79 s | ≤ 7.78 s | 10.6–15.1 s | 51–65 % |
A pass of the default takes 360–470 ms on the M4.
Before the repetition guard, the stock large-v3 wrote "rentrer en relation" 6,157 words long
on a 754-word French talk and crashed on the three others (decoder context exceeded). ACFT
never looped. Every run is in results/benchmark_real_mtedx.json
(metrics only: the talks are CC-BY-NC-ND-4.0).
Synthetic talks
Two short talks with known text — Portuguese (50 s, 122 words) and French (38 s, 126 words) — generated with Kyutai Pocket TTS: short replies, proper names, sentences glued together. ~120 words, so one wrong word is 0.8 % of WER.
| system | WER pt | WER fr | median | max |
|---|---|---|---|---|
| ACFT 8-bit, with the real-speech guards, no audio minimum | 4.1 % | 0.8 % | 1.15–1.23 s | 2.03–2.14 s |
| ACFT 8-bit, before the real-speech guards (2 runs) | 2.5 / 4.1 % | 0.8 / 2.5 % | 0.96–1.06 s | 1.65–2.07 s |
| Whisper large-v3 8-bit, same config (2 runs) | 0.0 / 0.0 % | 3.3 / 3.3 % | 0.99–1.17 s | 1.56–1.97 s |
results/benchmark_synthetic_acft.json,
results/benchmark_m_series.json (the earlier large-v3 default).
NVIDIA GPU (RTX 3060, PyTorch fp16)
ACFT against the stock large-v3 with no MLX and no 8-bit conversion: one RTX 3060 (12 GB), Linux, Transformers, fp16, same streaming logic and defaults, the same four talks and two synthetic ones. Streaming runs in real time; "whole file" decodes in 30 s windows. WER summed per group (errors / reference words), one run each:
| group | streaming ACFT | streaming large-v3 | whole file ACFT | whole file large-v3 |
|---|---|---|---|---|
| mTEDx fr (1466 words) | 7.30 % | 10.64 % | 5.39 % | 5.39 % |
| mTEDx pt (1281 words) | 8.20 % | 17.72 % | 6.25 % | 5.78 % |
| synthetic (241 words) | 2.49 % | 2.90 % | 0.83 % | 0.83 % |
- Streaming: ACFT invents far fewer words (pt 35 vs 127, fr 52 vs 91).
- Whole file: the two are level. The gap ACFT shows on the Mac above (pt 12.2 vs 8.8 %) comes from the 8-bit MLX conversion, not from the model: in fp16 it is 6 words out of 1281.
- Speed: the 3060 keeps up with both: a pass takes 244–294 ms (ACFT) and 208–298 ms (large-v3) at the median, and the transcript ends at most 0.88 s behind the talk. A whole 5-minute talk takes 53–79 s.
- Not measured here: the latency of each word. faster-whisper was not tried: both models run in Transformers, so this compares the models, not a production server. That port is not in this repository, which stays MLX.
Every run is in results/benchmark_rtx3060.json (metrics only).
How we got here
Same synthetic audio, same machine, Whisper large-v3 (median / max latency):
| step | median | max |
|---|---|---|
| LocalAgreement-2 (as whisper_streaming), CPU, faster-whisper int8, beam 5 | 79–99 s | 102–117 s |
| LocalAgreement-2, Mac GPU (mlx-whisper), 15 s buffer | 11–12 s | 19–21 s |
| AlignAtt, encoder padded to 30 s, 1 s chunks | 2.8–2.9 s | 4.6–5.2 s |
| + encoder on the real audio, 10 s buffer, 0.5 s chunks | 1.5 s | 3.0 s |
| + warm-up, attention over real audio only, hold back only at the attention stop, 8-bit | 1.3 s | 2.0–2.4 s |
| + threshold 0.3 s, 8 s buffer, safe buffer cut | 1.0 s | 1.6–2.0 s |
| + ACFT and the real-speech guards | 1.0–1.2 s | 2.0–2.1 s (synthetic), 2.5–3.2 s (real) |
| + 1 s audio minimum (this version; real talks) | 1.08–1.15 s | 2.2–2.7 s |
For scale: the same large-v3 one sentence at a time (not streaming) takes 1.6–3.0 s per sentence on this GPU and 2.9–4.7 s on the CPU (ONNX int8, 4 threads).
Tried and not kept
- large-v3-turbo (4 decoder layers): repetition loops with this policy ("Claro. Claro. Claro…", WER > 100 %); its alignment heads do not hold the AlignAtt stop.
- Normalized attention (z-score over tokens + median filter, as SimulStreaming): with the hold-back rule above it stalled the French talk for 10 s at the start.
- 6 s buffer with the stock large-v3: WER 6–8 % (ACFT tolerates it: 0.8–2.5 %, max 1.8–2.2 s). 0.3 s chunks: no lower maximum. 8-bit vs fp16: same speed (~500 ms per pass), kept for memory.
- Quality setting (10 s buffer, 0.5 s threshold): fewer errors on synthetic speech (ACFT fr 0.8 %), but 2.2–2.3 s maximum.
- Beam search (3 beams): no gain on real talks (pt 8.4 %, fr 7.6 %), ~60 ms more per pass. Kept as an option.
- Silero VAD: half as many words over 2 s (0.3–1.1 %), but more errors in Portuguese (9.5 %). A first version that restarted the buffer at every 0.5 s pause made Whisper invent sentences from the first fraction of a second of each new stretch; the kept version only restarts after 2 s of silence. Kept as an option — useful with a live microphone and long silences, which these talks do not have.
- 1.5 s audio minimum: the start of the talk nearly right ("quando tinha sete anos"), but no better overall (pt 7.6 %, fr 7.1 %) and a longer tail (max 3.1 s).
Limits
- One M4 and one RTX 3060, two languages, four real talks and two synthetic ones, one run each on real speech. Noisy rooms, distant microphones, accents, conversation and sessions longer than 5 minutes are not measured.
- On real talks 1–2.5 % of the words take more than 2 s (at most 2.7 s with the default).
- The first second of each stretch of speech waits for the audio minimum.
- Word end times from the attention are approximate; they only decide what leaves the buffer.
- The latency is transcription only (no translation, no display).
- Runs vary: the weights published here gave pt 7.7 % and fr 6.6 % (2026-11-05). One Portuguese talk gave 15.4 % in a run on a heavily loaded Mac and 10.8 % when repeated.
- These 8-bit weights are for streaming. Transcribing a whole talk offline with them was worse than the fp16 model (see the benchmarks).
Ideas not tried yet
- A block-causal encoder with a key/value cache, so each pass encodes only the new audio (WhisperKit, CarelessWhisper, U2-Whisper) — needs fine-tuning.
- Simul-Whisper's integrate-and-fire truncation detector to decide whether the last word is complete — no checkpoint exists for large-v3.
References
- A. Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022.
- S. Papi, M. Negri, M. Turchi, AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation, Interspeech 2023.
- H. Wang et al., Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection, Interspeech 2024.
- D. Macháček, R. Dabre, O. Bojar, Turning Whisper into Real-Time Transcription System (whisper_streaming, LocalAgreement), 2023.
- D. Macháček et al., Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025 (SimulStreaming), 2025; code: ufal/SimulStreaming.
- FUTO, whisper-acft (audio-context fine-tuning); large-v3 checkpoint by Mahmoud Ashraf: acft-whisper-large-v3.
- WhisperKit: On-device Real-time ASR, Argmax, 2025.
- CarelessWhisper, 2025; U2-Whisper, 2025.
- whisper.cpp, partial encoder context (
audio_ctx). - E. Salesky et al., The Multilingual TEDx Corpus for Speech Recognition and Translation, Interspeech 2021.
- MLX and mlx-whisper (Apple); mlx-community conversions.
- NVIDIA Parakeet TDT 0.6B v3; the int8 ONNX export used by ucast.me: marcosremar2/parakeet-ultra-onnx.
- Silero VAD (MIT).
- Kyutai Pocket TTS (synthetic test speech).
Licence
MIT for the code in this repository (LICENSE). The weights are a converted and 8-bit quantized
copy of the ACFT checkpoint by Mahmoud Ashraf, under Apache-2.0 (LICENSE-weights). Whisper large-v3 and the ACFT checkpoint are Apache-2.0
(their model cards); AlignAtt as implemented in SimulStreaming (MIT) and the alignment heads
from openai/whisper (MIT) were the starting point.
- Downloads last month
- 116
Quantized
Model tree for marcosremar2/whisper-large-v3-streaming-mlx
Base model
openai/whisper-large-v3