ACFT Whisper large-v3 for whisper.cpp (ggml)

MahmoudAshraf/acft-whisper-large-v3, Whisper large-v3 fine-tuned for a short audio context with FUTO's ACFT recipe, converted to the ggml format of whisper.cpp. Made for ucast.me (live subtitles).

file size
ggml-acft-large-v3-q8_0.bin 1.7 GB 8-bit, recommended: same accuracy as f16 in our test
ggml-acft-large-v3.bin 3.1 GB f16

Use

Best result: cut the audio at pauses into pieces of 12 to 20 s and decode each one greedily. chunk.py (in this repo, needs numpy) takes a 16 kHz mono 16-bit WAV and ends each piece at the quietest 200 ms of that range.

python chunk.py speech.wav pieces 12 20
whisper-cli -m ggml-acft-large-v3-q8_0.bin -l pt -mc 0 -bs 1 -bo 1 -nt -otxt -f pieces/*.wav
cat pieces/*.txt

For a whole file with no cutting, pass at least -mc 0. With the default text context, whole talks lose long stretches (table below).

whisper-cli -m ggml-acft-large-v3-q8_0.bin -f speech.wav -l pt -mc 0

How this was achieved

  • The model. ACFT is Whisper large-v3 after a short extra training on short, variable audio contexts, by FUTO's recipe. Size, architecture and speed do not change. It helps when the encoder runs on less than the usual 30 s. On whole files it ties with the stock model, so do not expect a gain there.
  • -mc 0. It stops whisper.cpp from passing the previous text to the decoder. In our test this alone took the Portuguese WER from 18.1 % to 8.2 %, because the long gaps in the transcript went away. We saw the effect; we did not isolate why the previous text causes it.
  • Pieces cut at pauses. Taken from the streaming setup of whisper-large-v3-streaming-mlx, which never feeds the model a long stretch at once. Cutting the talks at pauses took the WER from 8.2 % to 5.7 % in both languages. Pieces of 12 to 20 s were the best pair; 5 to 8 s was worse (6.5 %).
  • q8_0. Quantizing to 8 bits kept the accuracy here (pt 8.2 % in both, fr 8.3 % against 7.9 %) at about half the size.

Word error rate

Four mTEDx talks of about 5 minutes, CPU, q8_0 unless noted, one run each. Lower case, numbers spelled out, punctuation removed. Portuguese 1,281 words, French 1,466.

whisper.cpp, -mc 0 unless noted pt fr
pieces of 12–20 s, greedy 5.7 % 5.7 %
pieces of 12–20 s, default beam 5.8 % 5.8 %
pieces of 20–29 s, default beam 6.2 % 5.5 %
pieces of 20–29 s, greedy 6.7 % 5.3 %
pieces of 8–12 s, default beam 6.7 % 6.0 %
pieces of 5–8 s, default beam 6.6 % 6.6 %
pieces of 5–8 s, greedy, -ac 512 7.0 % 6.6 %
whole file, greedy 7.6 % 7.6 %
whole file, default beam 8.2 % 8.3 %
whole file, f16 8.2 % 7.9 %
whole file, default text context 18.1 % 10.0 %
whole file, f16, default text context 17.5 % 10.6 %

With the default text context, the worst Portuguese talk lost 142 of its 638 words (WER 31 %); with -mc 0, 24 (WER 11 %).

The piece sizes were chosen on these same four talks, so the best rows are a little optimistic. Differences under about 0.5 point can be run-to-run noise: the rows from 5.3 % to 6.2 % are not clearly apart.

Limits

  • A piece's text exists only after the piece ends, so 12–20 s pieces arrive that late: good for files and recordings, not for live subtitles.
  • This is not the ~1 s word-by-word streaming of whisper-large-v3-streaming-mlx: that uses the AlignAtt policy, which is not in whisper.cpp. The streaming numbers there do not apply here.
  • Measured on CPU (-ng). In our whisper.cpp build (commit 4da27c4, Apple M-series, Command Line Tools only) Metal aborted at start, also with the official ggml-tiny.bin, so GPU results are unknown.
  • No speed numbers: the machine was busy with other work during these runs.
  • Two languages, clean talks, one run. Noise, distance and conversation are not measured.

How it was made

models/convert-h5-to-ggml.py and whisper-quantize ... q8_0 from whisper.cpp 4da27c4, on the checkpoint as published by its author.

Licence

Apache-2.0, like the ACFT checkpoint by Mahmoud Ashraf (LICENSE). These files are a format conversion of that checkpoint, and q8_0 is also quantized.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for marcosremar2/acft-whisper-large-v3-ggml

Finetuned
(2)
this model