You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Sori-1B is built from NVIDIA's frozen Audio Flamingo Next audio encoder (NVIDIA OneWay Noncommercial License, academic use only) and the SmolLM2-360M language model (Apache-2.0), fully fine-tuned here. The complete terms are in LICENSE. Access is granted on these conditions: you use the model for academic research only and never for any commercial purpose; you do not copy, mirror or redistribute the weights or any part of this repository; you obtain the author's written agreement before releasing or deploying any model trained, fine-tuned, distilled or merged from it or from its outputs; and you cite this work in any publication whose results depend on it.
Log in or Sign Up to review the conditions and access this model content.
Sori: An Audio Language Model Agent
Sori-1B is its core model, addressed as a Python interpreter.
print shows —
speech, music and environmental sound, in one grammar that is ordinary Python.
Sori-1B — sori (소리) is Korean for sound — is an audio language model that you address as a Python interpreter. A sound clip is a value in the session,
a task is a function call on that value, and the answer is what print shows. It listens to speech, music and
environmental sound and answers about what is heard — transcription, description, questions, speaker-attributed
transcripts, timed events, counts — inside one grammar that is ordinary Python.
Philosophy
Understanding is answering from the sound. An answer should be caused by what is audible. Sori-1B is judged, at every evaluation, not only by whether the answer is right but by how much it depends on the sound.
Borrow the language, invent nothing. A language model already knows how an interpreter session reads. Sori-1B adds no special tokens, no chat template and no task tokens; everything it has to learn is about sound.
Time is structure, not annotation. The clip is displayed as a list of seconds. Duration is len(audio), a window
is audio[3:7], and a question about when is a question about an index.
Every ability has a type. A transcript is a string, a speaker-attributed transcript is a list of (start, end, speaker, text) tuples, a count is an int, a verdict is True or False. Answers parse with Python's own parser.
Interactive by construction. The model's turn ends when it hands the prompt back, so a session continues: more questions about the same clip, a second clip, an exception when it cannot tell.
A tool, not a chatbot. Every ability is a typed function, so an agent can call Sori-1B the way it calls any
other tool: one call is one >>> line, the result parses into a Python value, and a Traceback is an honest refusal.
The function table below is also the model's MCP tool surface — Session.tools() emits it as tool schemas and
sorilm.mcp_server serves it — so a planner can transcribe a span, count speakers, verify a claim, and compose the
results, without prompt engineering and without parsing prose.
Sori is the agent; Sori-1B is its core. Sori takes an ordinary question in natural language and answers it with the interpreter: from the outside it looks like any audio-language model, one call in and one answer out, and what it did to get there can be read back as interpreter lines. The core model can also be used directly, as the interpreter below — the caller decides which function to call. The usage section opens with the agent and continues with the interpreter.
Setup
pip install "transformers>=5.13" "torch>=2.6" soundfile scipy numpy
huggingface-cli login # the repository is gated: accept the terms on this page first
Everything the model needs travels with the checkpoint (trust_remote_code=True loads the processor, the grammar and the
session code from this repository). The processor decodes audio with soundfile (paths or http(s) URLs; wav, flac, ogg, and mp3 with libsndfile 1.1+),
resamples with scipy, and computes log-mel features with the Hugging Face extractor; an optional Rust module (sori_audio, not distributed) only makes batch decoding faster and
produces identical features. Any sample rate and any length are accepted: clips are zero-padded to whole seconds and
never cut. Memory grows with length; a 24 GB GPU in bf16 handles clips of roughly a quarter of an hour, and CPU works.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
proc = AutoProcessor.from_pretrained("snkii/Sori-1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("snkii/Sori-1B", trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()
Ask: the agent
One call, a question in natural language, one answer; explain=True returns the interpreter lines behind it. The agent
runtime is released together with the weights; until then the interpreter below is the working surface.
sori = model.agent(proc)
sori.ask("clip.wav", "How many dogs are barking?") # '2'
sori.ask("clip.wav", "What is the speaker's emotion?", ["angry", "calm", "happy", "sad"]) # 'angry'
sori.ask("clip.wav", "Which comes first, the siren or the horn?", ["the siren", "the horn"])
sori.ask(["a.wav", "b.wav"], "Which clip is louder?", ["the first", "the second"]) # two clips
sori.ask("clip.wav", "Is there a siren?", ["yes", "no"], explain=True) # the answer with the interpreter lines behind it
sori.ask_many([{"audio": "a.wav", "question": "Describe the scene."}, {"audio": "b.wav", "question": "How many speakers?"}])
sori.tools() # MCP schemas: `ask` plus every function below
Verdicts on answers can be recorded locally (sori.feedback(id, correct=False, answer="the horn")), opt-in and never
uploaded.
The interpreter
The model itself is addressed as a Python interpreter. A Session keeps the interpreter state (the clip is encoded once,
the transcript grows), run types one line and returns what it prints, and call is the typed form of the same line: it
builds the call from the function signature and parses the printed value into a Python object.
s = model.session(proc) # a fresh interpreter
s.load_audio("clip.wav") # -> `audio` (a path, an http(s) URL, or a waveform; any format, rate or length)
Speech
s.call("transcribe") # 'the quick brown fox jumped over the lazy dog'
s.run('print(transcribe(audio[3:7]))') # only what is said between 3 s and 7 s
s.call("transcribe", speaker=2) # only the second speaker (numbered by first appearance)
s.call("segments") # [(0.4, 3.2, 1, "that's why we cry"), (0.5, 6.1, 2, 'but i would not')]
s.call("count", "speakers") # 2
s.call("count", "turns") # 5
s.call("count", "words") # 41
Description and questions
s.call("describe") # 'A woman speaks while water runs from a faucet.'
s.call("answer", "What is the man doing?") # 'He is washing dishes.'
s.call("answer", "What is heard right after the door?", ["drilling", "chainsaw", "jackhammer", "vacuum cleaner"])
# 'drilling' (one of the choices, verbatim)
s.call("verify", "Water is running.") # True
s.call("present", "dog barking") # False
Events and time
s.call("events") # [(0.5, 1.3, 'breathing'), (1.8, 6.8, 'chirping birds')]
s.call("events", "dog barking") # [(0.0, 2.5, 'dog barking'), (7.3, 9.9, 'dog barking')] or []
s.call("count", "dog barking") # 2
s.run('print(len(audio))') # 12 -> the clip is 12 s long (a list of seconds)
Attributes, tags and measurements
s.call("label", "emotion") # 'angry'
s.call("label", "gender") # 'female'
s.call("label", "genre") # 'jazz'
s.call("tags", "instruments") # ['piano', 'drums', 'bass']
s.call("tags", "sounds") # ['Fire engine, fire truck (siren)', 'Truck']
s.call("measure", "tempo") # 120.5
s.call("measure", "pitch") # 440.0
s.call("measure", "words per minute") # 149
Two clips
s.load_audio("other.wav") # -> `audio2`
s.call("same_speaker", "audio2") # False
s.call("compare", "audio2", "loudness") # 'audio2' (which clip has more of it: 'audio' or 'audio2')
s.call("compare", "audio2", "speakers") # 'audio'
Several questions about one clip. The session is an interpreter, so it simply continues; every line sees everything above it, and the transcript is exactly what the model reads:
s = model.session(proc); s.load_audio("meeting.wav")
s.call("count", "speakers") # 3
s.call("segments") # [(0.0, 4.1, 1, 'okay so'), (4.3, 9.0, 2, 'right'), ...]
s.call("transcribe", speaker=3) # 'i think we should start'
print(s.transcript)
# >>> audio = load_audio("sori.wav")
# >>> audio
# [[<25 frames>], [<25 frames>], ...]
# >>> print(count(audio, "speakers"))
# 3
# >>> print(segments(audio))
# [(0.0, 4.1, 1, 'okay so'), (4.3, 9.0, 2, 'right'), ...]
# >>> print(transcribe(audio, speaker=3))
# i think we should start
# >>>
Batches without a session. The processor renders the same transcript for a list of clips (padding on the left, so
every prompt ends where generation starts) and generate stops at the >>> that hands control back:
batch = proc(["a.wav", "b.wav", "c.wav"], mode="caption") # modes: asr, caption, qa, mcq, count, events, ...
out = model.generate(**{k: v.cuda() for k, v in batch.items() if k != "audio_seconds"}, max_new_tokens=64)
for row in out: print(proc.decode_output(row[batch["input_ids"].shape[1]:].tolist()))
batch = proc("a.wav", mode="mcq", question="Which instrument leads?", choices=["piano", "violin", "trumpet", "flute"])
Raw interpreter lines. Anything the registry knows can be typed directly, including slices, keyword arguments and
compound statements; the model's reply is what print would show, and a Traceback is how it declines:
s.run('print(transcribe(audio[10:20]))')
s.run('print(events(audio, "siren"))')
s.run('for t in segments(audio):\n print(t)', block=True)
For people. A terminal conversation about a clip, where plain requests become interpreter lines and are shown:
model.chat(proc, "clip.wav") # you> transcribe 3-7 -> >>> print(transcribe(audio[3:7]))
Terminal demo. sori_tui.py in this repository is a one-file terminal application with an example of every
function. --demo is a guided tour on audio the script synthesises itself (beeps, a tone, a click track, a melody, a
chord, two synthetic voices), where the constructed answers grade the model; the shell has one colon command per
function, a spectrogram, multiple choice with a likelihood ranking, and the interpreter transcript:
pip install rich # optional: colour; `espeak` on the PATH adds the speech scenes
python sori_tui.py --demo # every function, graded where the answer is constructed
python sori_tui.py --selftest # install check: return types on a synthetic clip
python sori_tui.py # shell: :load clip.wav :describe :count speakers :events dog :mcq :py <line>
python sori_tui.py --audio clip.wav --fn count --what speakers
For agents. The function table is the tool surface: s.tools() emits one JSON schema per function, and a Model
Context Protocol server is a thin loop over s.call:
from mcp.server import Server # pip install mcp
import mcp.types as types
server = Server("sori"); s = model.session(proc)
@server.list_tools()
async def list_tools():
return [types.Tool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]) for t in s.tools()]
@server.call_tool()
async def call_tool(name, arguments):
if name == "load_audio": res = s.load_audio(arguments["path"])
elif name == "run": res = s.run(arguments["code"])
else: res = s.call(name, **arguments)
return [types.TextContent(type="text", text=str(res))]
| call | returns |
|---|---|
transcribe(audio) · transcribe(audio[a:b]) · transcribe(audio, speaker=k) |
str |
describe(audio) |
str |
answer(audio, question) · answer(audio, question, choices) |
str (with choices: one of them verbatim) |
segments(audio) |
[(start, end, speaker, text), ...] |
events(audio) · events(audio, what) |
[(start, end, label), ...] |
count(audio, what) |
int |
label(audio, what) · tags(audio, what) · measure(audio, what) |
str · list · number |
present(audio, what) · verify(audio, claim) |
bool |
same_speaker(audio, audio2) · compare(audio, audio2, what) |
bool · 'audio' or 'audio2' |
Times are seconds with one decimal, speakers are numbered by first appearance, lists are sorted by start time, and every
structured value round-trips through ast.literal_eval. See AGENTS.md for the instructions written for automated users.
Evaluation
The checkpoint currently published is the final step of the first training run. On our own held-out sets it reaches 0.47 on compositional audio-reasoning multiple choice and 0.43 exact match on free-form generation, and its text perplexity is below the base language model's, so nothing was forgotten. Better checkpoints are published as they exist.
License
Sori-1B is for non-commercial, academic use only, and access is gated. Commercial use, copying or redistribution of the weights or repository, and releasing models derived from these weights or their outputs are not permitted without the author's written agreement. The full terms — including the NVIDIA OneWay Noncommercial License of the Audio Flamingo Next encoder (redistributed unchanged and frozen) and the Apache License 2.0 of SmolLM2-360M (initialisation of the language model) — are in LICENSE.
Citation
@software{kim_sori_1b_2026,
author = {Kim, Seonuk},
title = {{Sori-1B: An Audio Language Model Addressed as a Python Interpreter}},
year = {2026},
month = aug,
date = {2026-08-28},
version = {1.0},
url = {https://huggingface.co/snkii/Sori-1B}
}
CITATION.cff in this repository carries the same metadata in CFF 1.2.0 form.
- Downloads last month
- 243