ConceptualMachines/magda-command-model

MAGDA command model

Turns a typed request into a MAGDA DAW command, offline and on-device. "put @serum on a new track" becomes track(name="", new=true).fx.add(name="<serum>"), which the DAW executes as one undoable transaction.

Fine-tuned from microsoft/deberta-v3-base. English only.

Scope: edits, not gestures

The command set covers editing operations — creating and naming tracks, adding devices, building racks, clip and note operations, grouping, selection by predicate. Deliberately excluded are real-time transport and mixer states (play/stop, track mute, solo): those have keyboard shortcuts, and a model with latency is strictly worse than the button. Queries ("what's the tempo") are excluded too — they are answers, not edits.

Those requests are not merely unsupported; they are trained as an explicit abstain class, so the model returns nothing rather than executing whichever command happened to be nearest. A closed label set cannot decline unless declining is itself a label.

Note that clip enable/disable is covered: it is a persistent property of the project, unlike a mixer mute, and picking one clip out by name is exactly the kind of thing typing beats clicking at.

What it actually does

It never writes the command. It does perception only, in a single forward pass, and emits two things:

1. which of 39 commands this is (one of which is "none of these")
2. a BIO tag per word saying what role that word plays

"can you   drop  @1176   onto the  vocals"
  O   O     O   PLUGIN    O    O  TRACK_NAME     intent = add_plugin

Deterministic code then assembles the DSL from those labels. The model cannot emit malformed output, because it does not emit output — a property a generative model would have to buy back with constrained decoding.

Plugin references never reach the model as identities: the host rewrites every @mention to an opaque <alias> token before tokenization and substitutes the real one back by position. So an unseen plugin generalizes for free, and the vocabulary does not grow with the plugin registry.

Results

Scored as exact-match on the fully rendered command, not on tags.

	in-distribution	held-out phrasing
previous model (51k-param conv net, from scratch)	98.2%	46.5%
this model	100%	~90%

The two columns matter more than the numbers. The in-distribution set is authored from the same case list the training templates come from, so a high score there means the model recalls its templates. The held-out set is 68 cases hand-written from the command list alone, in the register people actually type: lowercase, filler, contractions, typos. That is the number that predicts behaviour on input nobody has seen.

The previous model had no language prior — trained from scratch on synthetic English, it named tracks after whichever word sat in the name position during training:

"can you drop @1176 onto the vocals" -> track(name="Can You Drop @1176")...
"push Keys up to 2 dB"               -> track(name="Push").track.set(volume_db=2)
"the bass needs @pro_q_3"            -> track(name="Bass Needs").track.set(colour="None")

Politeness wrappers scored 10%. With a pretrained encoder they score 100%.

Files

file	size	what
command_model.onnx	442 MB	encoder + intent head + slot head, one graph, opset 17
tokenizer.json	8 MB	HF fast tokenizer, including the <alias> special tokens
maps.json	1 KB	intent and BIO tag id maps

Inputs are input_ids and attention_mask, both int64 [batch, seq] with dynamic sequence length. Outputs are intent_logits [batch, 39] and slot_logits [batch, seq, 20].

Quantization warning

The embedding is int8; the attention and FFN deliberately are not.

build	size	accuracy
fp32	736 MB	96.5%
int8 embedding only (this)	442 MB	96.1%
int8 everything	538 MB	0.9%

DeBERTa-v3's disentangled attention is quantization-hostile. Running quantize_dynamic over the whole graph produces a model that loads, runs at full speed, and emits garbage. If you re-quantize this, re-measure it.

Usage

Two details will cost you accuracy if you get them wrong.

Words, not subwords. The tags are per word, and the deterministic reconstructor consumes per-word tags. Label the first subword of each word and read the tag back at that same index. Reading tags positionally shifts every tag after the first multi-piece word — it still produces plausible-looking output, so it fails quietly.

The alias tokens must stay atomic. <alias> and <alias.param> are registered as additional special tokens. If your tokenizer splits them into <, alias, >, every plugin slot breaks.

from transformers import AutoTokenizer
import onnxruntime as ort, numpy as np

tok = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base", from_slow=True)
tok.add_special_tokens({"additional_special_tokens": ["<alias>", "<alias.param>"]})
sess = ort.InferenceSession("command_model.onnx")

words = ["put", "<alias>", "on", "a", "new", "track"]
enc = tok(words, is_split_into_words=True, return_tensors="np")
intent_logits, slot_logits = sess.run(None, {
    "input_ids": enc["input_ids"].astype(np.int64),
    "attention_mask": enc["attention_mask"].astype(np.int64),
})

A reference implementation in C++ (ONNX Runtime plus a hand-written SentencePiece Unigram tokenizer) is in magda/agents/command_model_onnx.cpp and unigram_tokenizer.cpp in the MAGDA repo.

Training

24,000 template-generated English examples (no LLM paraphrasing in this build), 4 epochs, lr 3e-5 for the encoder and 1e-3 for the two heads. Checkpoints are selected on a hand-authored held-out set, never on the reported one — the val split is drawn from the same templates as training and saturates at 100% after one epoch, which makes it useless for ranking checkpoints.

Limitations

• English only. The word splitter in front of the model is ASCII-only, so non-English input reaches it as an empty sequence. This is a limitation of the surrounding pipeline, not the encoder, which is multilingual-capable.
• Plugin references need the @ sigil. track with @serum works; track with serum does not, and degrades into an unrelated command rather than failing cleanly.
• Closed command set. 38 commands plus an explicit abstain class. Anything outside them returns nothing rather than being mapped onto the nearest command — but abstain recall is not perfect, so an unusual out-of-scope request can still land on a command.
• Sentences where the verb trails its object ("i don't need the perc track anymore, delete it") are a known weak spot.

License

MIT, following microsoft/deberta-v3-base.