kokoro-inno-clone-tuner
pip install inno-kokoro
Zero-shot voice tuner for Kokoro-82M.
Outputs base Kokoro compatible voice packs @ [510, 1, 256]. (write-up)
Samples
Same passage for every voice, enrolled from the references. LibriTTS-R speakers are dev-clean held out from training.
| voice | tuned | reference |
|---|---|---|
| am_libritts1272 | ||
| af_libritts5338 | ||
| am_libritts6241 | ||
| af_ameliaearhart | ||
| bf_janegoodall | ||
| am_vincentprice |
Usage
Code available on Github: remsky/inno-kokoro.
Integrated into Kokoro-FastAPI (v0.9.0+)
PyTorch, with kokoro
pip install "inno-kokoro[torch]"
from inno_kokoro.enroll import Tuner, enroll, read
import torch
from kokoro import KPipeline
tuner = Tuner(backend="torch") # caches model from HF on first use
pack, _ = enroll(*read("my_ref.wav"), tuner, return_pt=True) # [510, 1, 256]
torch.save(pack, "am_me.pt")
pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio
ONNX (no torch), with kokoro-onnx
pip install inno-kokoro kokoro-onnx
from inno_kokoro.enroll import Tuner, enroll, read
import numpy as np
from kokoro_onnx import Kokoro
tuner = Tuner(backend="onnx") # downloads model.safetensors + model.onnx on first use
pack, _ = enroll(*read("my_ref.wav"), tuner) # [510, 1, 256] float32 NumPy
np.save("am_me.npy", pack)
kokoro = Kokoro("kokoro-v1.0.onnx", "voices-v1.0.bin") # from the kokoro-onnx releases
wav, sr = kokoro.create("Hello from a tuned voice.", voice=pack, lang="en-us")
Notes:
- prefix names with stock accent+gender codes for use in other libraries: e.g
af_,am_,bf_,bm_ - pitch ceiling is set automatically
enroll(..., fmax=180): can override when needed (archival sources, etc)
ONNX
model.onnx: the speaker encoder and style head for onnxruntime (fp32, 31.5 MB). Used by Tuner(backend="onnx") from inno-kokoro 0.4.0. Same packs as torch, about 2x faster on CPU.
Graph details
- input
fbank[1, frames, 80] float32: kaldi log-mel of the first 30 s at 16 kHz (inno_kokoro/fbank.py) - outputs
embedding[1, 512] (unit, UniSpeech-SAT-sv space) andstyle[1, 256] - opset 20, standard ops only, no DFT or custom ops
- matches torch within 5.4e-7 on 540 clips (parity check)
- exported with
export_onnx.py(inno-kokorotools/export_onnx.pyat b442727)
| reference | onnxruntime | torch |
|---|---|---|
| 8 s | 100 ms | 165 ms |
| 30 s | 410 ms | 890 ms |
CPU (best of 5 runs)
Architecture
A 5 to 30 s reference clip in, a stock-shaped Kokoro voice pack out. Everything loads from model.safetensors, no extra downloads.
Architecture details
Enrollment embeds an input audio sample via the baked speaker encoder and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half.
- speaker encoder: a 6.6M-param WeSpeaker ResNet34 distilled from UniSpeech-SAT-sv embeddings
- matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
- style head: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
- prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
- Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
- Packs more than 4 st from the reference's pitch are excluded.
- Higher graded packs are weighted as preferred during selection.
- prosody head (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
- fitted to deltas optimized through the frozen predictor.
enroll(..., head=False)skips it.
- fitted to deltas optimized through the frozen predictor.
Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.
Total model size @ fp16: ~24mb
Limits
- English trained (LibriTTS-R) currently
- Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
- Rythym is a best-effort match to the speaker.
- Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
- Reference clip requirements:
- 3 second minimum, up to a max of 30s
- Reasonably free of audio artifacts
- Single speaker
Training and Metrics
All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.
Trained via HF Jobs a10g-small; under $20 in GPU time:
v0.1.0 training notes
- Stage 1: mapped speaker embedding to timbre:
- Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
- [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
- Stage 2: the embedding alone missed texture and pitch; adjustments:
- Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
- [11k steps: dev cosine 0.934, DNSMOS 3.30]
- Stage 3: tuned the encoder side to match the blended predictors:
- Predictor half fixed to the blended rows, only the decoder half trained.
- [3k steps: dev cosine 0.932, DNSMOS 3.48]
- Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
- WeSpeaker ResNet34 plus a projection, trained on
1 - cos(student, teacher)over LibriTTS-R crops with synthetic channel damage; the style head is untouched. - [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
- WeSpeaker ResNet34 plus a projection, trained on
- Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
- Mixed real public-domain archival broadcast audio into the distillation crops.
- [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]
v0.2.0 training notes
- Stage 1: the blend under-predicted pitch range and sentence-final falls:
- Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
- Targeted F0 mean, spread, IQR, pause fraction and speech time.
- Ridge head from reference F0 mean + spread to the fitted delta.
- [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
- Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
- Added VoxPopuli and Emilia speakers to the distillation crops.
- [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
- Stage 3: the timbre half had only seen LibriTTS speakers:
- Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
- [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]
Zero-shot identity benchmark
Inno gets about a third of the normalized similarity range, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).
Full benchmark results and methodology
LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers. Reference in, new sentence out, scored against the speaker's real recording.
- normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
- RTF: RTX 4060 Ti
| system | SIM-o | normalized | ECAPA | UTMOS | RTF |
|---|---|---|---|---|---|
| ground truth (second recording) | 0.695 | 1.00 | 0.656 | 4.10 | |
| F5-TTS v1 base | 0.650 | 0.94 | 0.713 | 3.86 | 0.48 |
| StyleTTS2 (LibriTTS zero-shot) | 0.386 | 0.46 | 0.415 | 4.40 | 0.06 |
| Inno v0.2 | 0.288 | 0.32 | 0.288 | 4.45 | 0.07 |
| OpenVoice v2 | 0.227 | 0.23 | 0.316 | 3.80 | 0.12 |
| Kokoro, nearest stock pack | 0.167 | 0.15 | 0.215 | 4.25 | 0.06 |
StyleTTS2 uses a related architecture with an unfrozen decoder, a possible ceiling for this style space.
- StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
- OpenVoice v2: MeloTTS EN-US, tau 0.3.
- F5-TTS: 32 NFE, CFG 2.
v0.1 vs v0.2 shipped-weight comparison
Measured on the shipped weights through the stock KPipeline: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.
| metric | v0.1 blend only | v0.2 with head |
|---|---|---|
| speaker cosine (WavLM-sv), samples / dev-clean | 0.926 / 0.956 | 0.935 / 0.959 |
| F0 mean, render minus reference | 0.65 st mean abs | 0.8 st mean abs |
| F0 spread, render minus reference | 0.65 st mean abs | 0.5 st mean abs |
| DNSMOS | 3.50 | 3.50 |
The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.
Licenses and credits
Adapter weights and code: Apache-2.0, except the baked speaker encoder (enc.* in model.safetensors), which is CC BY-SA 3.0 (see below)
- Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture)
- The speaker encoder is a WeSpeaker ResNet34 (code Apache-2.0)
- initialised from the VoxCeleb-trained ResNet34-LM weights (CC BY 4.0)
- distilled from microsoft/unispeech-sat-base-plus-sv
- (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub); encoder weights under CC BY-SA 3.0
- Training data (no audio redistributed beyond the samples):
- LibriTTS-R (CC BY 4.0), also the LibriTTS sample references
- VoxPopuli (CC0)
- Emilia-YODAS EN (CC BY 4.0)
- Sample references:
- Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022
- Jane Goodall from BBC Woman's Hour excerpt, BBC, CC BY 3.0
- Vincent Price public domain from "The House On Haunted Hill"
Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.
- Downloads last month
- 1,629,543
