ACFT Whisper large-v3 for whisper.cpp (ggml)
MahmoudAshraf/acft-whisper-large-v3,
Whisper large-v3 fine-tuned for a short audio context with
FUTO's ACFT recipe, converted to the ggml format of
whisper.cpp. Made for ucast.me (live subtitles).
| file | size | |
|---|---|---|
ggml-acft-large-v3-q8_0.bin |
1.7 GB | 8-bit, recommended: same accuracy as f16 in our test |
ggml-acft-large-v3.bin |
3.1 GB | f16 |
Use
Best result: cut the audio at pauses into pieces of 12 to 20 s and decode each one greedily. chunk.py (in this
repo, needs numpy) takes a 16 kHz mono 16-bit WAV and ends each piece at the quietest 200 ms of that range.
python chunk.py speech.wav pieces 12 20
whisper-cli -m ggml-acft-large-v3-q8_0.bin -l pt -mc 0 -bs 1 -bo 1 -nt -otxt -f pieces/*.wav
cat pieces/*.txt
For a whole file with no cutting, pass at least -mc 0. With the default text context, whole talks lose long
stretches (table below).
whisper-cli -m ggml-acft-large-v3-q8_0.bin -f speech.wav -l pt -mc 0
How this was achieved
- The model. ACFT is Whisper large-v3 after a short extra training on short, variable audio contexts, by FUTO's recipe. Size, architecture and speed do not change. It helps when the encoder runs on less than the usual 30 s. On whole files it ties with the stock model, so do not expect a gain there.
-mc 0. It stops whisper.cpp from passing the previous text to the decoder. In our test this alone took the Portuguese WER from 18.1 % to 8.2 %, because the long gaps in the transcript went away. We saw the effect; we did not isolate why the previous text causes it.- Pieces cut at pauses. Taken from the streaming setup of whisper-large-v3-streaming-mlx, which never feeds the model a long stretch at once. Cutting the talks at pauses took the WER from 8.2 % to 5.7 % in both languages. Pieces of 12 to 20 s were the best pair; 5 to 8 s was worse (6.5 %).
q8_0. Quantizing to 8 bits kept the accuracy here (pt 8.2 % in both, fr 8.3 % against 7.9 %) at about half the size.
Word error rate
Four mTEDx talks of about 5 minutes, CPU, q8_0 unless noted, one run each. Lower case, numbers spelled out,
punctuation removed. Portuguese 1,281 words, French 1,466.
whisper.cpp, -mc 0 unless noted |
pt | fr |
|---|---|---|
| pieces of 12β20 s, greedy | 5.7 % | 5.7 % |
| pieces of 12β20 s, default beam | 5.8 % | 5.8 % |
| pieces of 20β29 s, default beam | 6.2 % | 5.5 % |
| pieces of 20β29 s, greedy | 6.7 % | 5.3 % |
| pieces of 8β12 s, default beam | 6.7 % | 6.0 % |
| pieces of 5β8 s, default beam | 6.6 % | 6.6 % |
pieces of 5β8 s, greedy, -ac 512 |
7.0 % | 6.6 % |
| whole file, greedy | 7.6 % | 7.6 % |
| whole file, default beam | 8.2 % | 8.3 % |
| whole file, f16 | 8.2 % | 7.9 % |
| whole file, default text context | 18.1 % | 10.0 % |
| whole file, f16, default text context | 17.5 % | 10.6 % |
With the default text context, the worst Portuguese talk lost 142 of its 638 words (WER 31 %); with -mc 0, 24 (WER 11 %).
The piece sizes were chosen on these same four talks, so the best rows are a little optimistic. Differences under about 0.5 point can be run-to-run noise: the rows from 5.3 % to 6.2 % are not clearly apart.
Limits
- A piece's text exists only after the piece ends, so 12β20 s pieces arrive that late: good for files and recordings, not for live subtitles.
- This is not the ~1 s word-by-word streaming of whisper-large-v3-streaming-mlx: that uses the AlignAtt policy, which is not in whisper.cpp. The streaming numbers there do not apply here.
- Measured on CPU (
-ng). In our whisper.cpp build (commit4da27c4, Apple M-series, Command Line Tools only) Metal aborted at start, also with the officialggml-tiny.bin, so GPU results are unknown. - No speed numbers: the machine was busy with other work during these runs.
- Two languages, clean talks, one run. Noise, distance and conversation are not measured.
How it was made
models/convert-h5-to-ggml.py and whisper-quantize ... q8_0 from whisper.cpp 4da27c4, on the checkpoint
as published by its author.
Licence
Apache-2.0, like the ACFT checkpoint by Mahmoud Ashraf (LICENSE). These files are a format conversion of
that checkpoint, and q8_0 is also quantized.
Model tree for marcosremar2/acft-whisper-large-v3-ggml
Base model
openai/whisper-large-v3