musicar-L

A 464M-parameter autoregressive transformer that writes multi-instrument MIDI one note at a time, pretrained on 8.3B notes of public MIDI data.

Parameters	Context	Instruments	Training notes	License
464.24M	2,048 notes	129 (128 General MIDI programs + drums)	8.3B (GigaMIDI, Aria-MIDI, Discover MIDI)	CC BY-NC-SA 4.0

Model summary

musicar-L is the project's pretrained multi-instrument model. Each sequence position is one whole note, and five cascade output heads choose its time shift, instrument, pitch, duration and velocity one after another. It is a pretrained model only: it has no text captions, and besides a MIDI prompt its only controls are two optional window conditions (the set of instruments and the notes per second). It is the base of the project's captioned model, musicar-L-caption.

Smaller bases of the same family: musicar-S-piano and musicar-S.

Model details

Developed by	Dmytro Hilei
Model type	Decoder-only transformer, autoregressive over notes (symbolic music, MIDI)
Parameters	464.24M: trunk 445.98M, cascade heads 16.28M, embeddings 1.99M
Architecture	28 pre-norm blocks, width 1152: RMSNorm → causal self-attention (18 heads × 64, full attention; Q, K, V from one Linear 1152 → 3456; QK-norm = RMSNorm over each head's queries and keys; RoPE; output Linear 1152 → 1152) → residual add; RMSNorm → SwiGLU feed-forward (gate and up 1152 → 3072, down 3072 → 1152) → residual add; final RMSNorm. Five cascade output heads: Δt → instrument → pitch → duration → velocity
Cascade heads	Head i reads a running state z₀ = h_t (the trunk state), z_i = z_{i−1} + E_i(a_i), where E_i embeds the value chosen by head i (tables of 512, 129, 130 and 512 rows × 1152; the last head needs none); each head is LayerNorm → Linear 1152 → 1152 → GELU → Linear 1152 → 1152 (+ residual) → LayerNorm → Linear 1152 → V. a_i is the true value in training (teacher forcing) and the sampled value in generation
Tokenization	One compound token per note on a 20 ms grid: delta-time = time since the previous note's onset (512 steps of 20 ms), instrument (129), pitch (130: 128 MIDI pitches + BOS + EOS), duration (512 steps of 20 ms), velocity (32 levels); delta-time and duration are clipped at 10.22 s. The notes of all tracks are merged into one sequence in onset order; a drum note's pitch is the drum sound. No bar, beat or tempo tokens
Input embedding	Pitch, velocity, duration and delta-time tables of width 288 (130, 32, 512 and 512 rows), concatenated (1152) → Linear 1152 → 1152; an instrument table (129 × 1152) added per note; the optional window conditions added at every position: the window's instrument set (129-dim multi-hot → Linear 129 → 1152) and its notes per second (a table of 16 rows: 15 levels + "none")
Instruments	129: the 128 General MIDI programs + drums
Context	2,048 notes (about 1–4 minutes of music)
Positions	RoPE (rotary position embedding, base 10,000) in every attention layer; no position table
Special tokens, conditions	BOS / EOS (pitch values 128 / 129) mark the start and end of a piece; window conditions as above, "none" = unconditional
Precision of the weights	bfloat16
License	CC BY-NC-SA 4.0
Code	https://github.com/DmytroHilei/Music_generative_model

How to use

From a clone of the code repository, with its environment installed (README, "Setup"):

hf download Dimitri-206265/musicar-L --local-dir checkpoints/musicar-L
.venv/bin/python generate.py --checkpoint checkpoints/musicar-L/model_bf16.pt --sample-instruments \
    --max-new-tokens 1000 --output samples/musicar-L.mid

One token is one note, so this writes up to 1,000 new notes, starting from the start-of-piece token (BOS); if the model ends the piece earlier, the output stops at its EOS (--min-notes forbids an early end). --sample-instruments lets the instrument head choose each note's instrument; without it every note is piano. --instruments limits the choice (a list of General MIDI programs such as 0,33,25,128, where 128 = drums, or prompt = the prompt's own instruments) and --instrument-temperature sets a separate temperature for it. --prompt file.mid continues the first --prompt-notes notes (default 64) of a MIDI file. The window conditions are set with --cond-instruments (a program list, prompt or none; by default the prompt's instruments when the prompt has them, else none) and --cond-density (notes per second, prompt or none, the default). Sampling: --temperature (default 0.85), --top-k, --top-p. Past 2,048 notes the KV cache is rebuilt from the most recent notes every --slide notes (default 512). generate.py also renders an MP3 unless --no-mp3 is given (needs fluidsynth, ffmpeg and a General MIDI soundfont).

Loading the pickle-free files directly:

import json
from safetensors.torch import load_file
from musicar.model.gpt import GPT, MusicConfig

cfg = json.load(open("checkpoints/musicar-L/config.json"))
model = GPT(MusicConfig(**cfg["model_args"]))
model.load_state_dict(load_file("checkpoints/musicar-L/model.safetensors"))
model.eval()

MusicConfig defaults to training settings (dropout 0.2), so call model.eval() before inference; musicar.model.checkpoint.load_model does this for model_bf16.pt. Sampling with the KV cache is musicar.generation.sampling.generate.

Intended use

• Research on symbolic music generation: multi-instrument modelling, sampling, conditioning.
• Continuing a MIDI prompt (piano or multi-instrument), or generating from the start token.
• A base for fine-tuning: the project's fine-tunes on Ukrainian songs (piano arrangements and multi-track) start from it, and so does the captioned model musicar-L-caption.

Out of scope:

• Commercial use (see License).
• Text-conditioned generation: this checkpoint has no caption input; musicar-L-caption adds one.
• Audio: the model reads and writes MIDI notes, not sound.

Training data

Dataset	Sampling weight	What it is	License note
GigaMIDI (arXiv:2502.17726)	0.4	Large web collection of multi-track MIDI (its own training split)	Non-commercial (the project's notes list CC BY-NC 4.0); check GigaMIDI's own terms (Hugging Face Metacreation/GigaMIDI)
Aria-MIDI, deduplicated (arXiv:2504.15071)	0.3	Piano transcriptions from recordings; its notes enter as piano	CC BY-NC-SA 4.0 (non-commercial)
Discover MIDI (Hugging Face)	0.3	Large MIDI collection, deduplicated against GigaMIDI and within itself	CC BY-NC-SA (non-commercial)

Each training window of 2,048 notes is drawn from one dataset by these weights. BOS / EOS mark where pieces start and end, and pieces shorter than a window share windows. Pitches are transposed at random by up to ±5 semitones (drums are not transposed). The run saw 8.3B notes, about 1.8 passes over the data on average (about 4.5 over Aria-MIDI). None of the project's audio-derived transcriptions of Ukrainian songs were used.

Training procedure

Initialization	From scratch
Steps	126,582
Batch	65,536 notes per step (32 windows × 2,048 notes)
Notes seen	8.3B (≈ 1.8 epochs on average; Aria-MIDI ≈ 4.5)
Data mix	GigaMIDI 0.4, Aria-MIDI 0.3, Discover MIDI 0.3
Objective	Sum of the five cross-entropies (Δt, instrument, pitch, duration, velocity), teacher forcing
Optimizer	Muon (Moonlight scaling, momentum 0.95) for the weight matrices of the transformer blocks, AdamW (β 0.9, 0.95) for the rest; one learning rate for both; weight decay 0.1; gradient clipping 1.0
Learning rate	10⁻³, WSD schedule: 1,000 warmup steps, constant, then 1 − √ decay to 0 over the last 20% of the steps
Regularization	Dropout 0, label smoothing 0; both window conditions replaced by "none" in 15% of the windows, so "none" stays usable
Precision	bf16 autocast with fp8 (torchao float8) matmuls in the transformer blocks, torch.compile; the checkpoint holds bf16 weights
Hardware, time	One rented RTX 5090, about 41 h
Config	config/pretrain/long_450m.py (launched through cloud/run/launch.sh)

Evaluation

Metric: teacher-forced next-note cross-entropy on held-out validation windows of 2,048 notes, in nats per note, summed over the four note attributes (Δt, pitch, duration, velocity); the instrument head's cross-entropy is reported separately. Lower is better. It measures how well the model predicts real music, not how good its samples sound; it is not a listening test. One run.

Validation set: clean GigaMIDI validation, i.e. GigaMIDI's validation split without the files whose notes also occur in its training split. The window conditions are measured on each validation window.

Clean GigaMIDI validation	Cross-entropy (nats / note)
First evaluation (step 2,000, 0.13B notes)	2.807
Step 100,000, just before the cooldown	1.876
End of training (8.3B notes)	1.679 (Δt 0.170, pitch 0.367, duration 0.644, velocity 0.497)
End of training, both window conditions "none"	1.681
Instrument head, end of training	0.117

The diagram (section 6) shows every evaluation of the run, one every 2,000 steps.

Comparison with other open models

The project's evidence sheet (findings.tex in the code repository): the comparison below is its section 1.

Every system wrote MIDI for one frozen prompt set, and every file was read back from the MIDI and scored the same way, per note and per second (never per token), with each model at its authors' sampling settings. Code: eval/compare/head2head.py and eval/compare/h2h_baselines/ in the code repository. One seed per system.

Continuation of real multi-instrument music (32 clean GigaMIDI validation prompts of 128 notes with at least 3 instruments; up to 1,000 new notes each, the first 1,000 scored, a continuation that ends sooner scored as it is). D = weighted gap in six row statistics of density and instrument use (one instrument taking over, top-instrument share, notes per second 0.20 each; out-of-band instruments, instrument count 0.15 each; runaway 0.10) to the real continuations of the same files, each divided by the real rows' spread (with a floor); lower = closer to the real continuations on these statistics. Runaway = share of rows above 40 notes per second. The last column scores the first 200 new notes only:

System	Parameters	Notes / s	Instruments	Runaway	D	D, first 200 notes
Real continuation		24.0	7.7	0.16	0	0
musicar-L (temperature 0.9)	464M	23.5	6.9	0.13	0.121	0.090
musicar-L-caption (no caption)	477M	12.8	7.8	0.06	0.218	0.198
Anticipatory Music Transformer, large (arXiv:2306.08620)	780M	21.2	7.4	0.06	0.289	0.239
SkyTNT midi-model tv2o-medium	234M	22.9	5.3	0.16	0.314	0.184
Music-Llama (SkyTNT architecture)	457M	28.9	5.7	0.25	0.357	0.218
Moonbeam (arXiv:2505.15559), temperature 0.9, top-p 0.95 (its README)	839M	49.5	8.9	0.31	0.703	0.322
Moonbeam, temperature 0.6, top-p 0.9 (its completion example)	839M	85.9	5.3	0.06	1.316	0.525

Moonbeam's context (1,024 notes) is shorter than prompt + continuation; past it the harness restarts its window on the last 768 notes (the head-to-head's scheme, not the authors'). At temperature 0.9 its notes per second climb inside the first window, before any restart: rows above 40 notes per second are 16% over its first 200 new notes and 46% over new notes 447-894 (real continuations 13% and 18%). At 0.6 the density stays flat except in 2 files that stack notes on one onset (most of its mean), and it repeats itself more: 46% of its new pitched notes sit inside exact copies of 16 or more chords (real continuations 5%). Over the first 200 notes every model is closer to the real continuations; SkyTNT, Music-Llama and Moonbeam gain most.

Solo-piano continuation (32 prompts from the project's held-out split of Aria-MIDI, which Aria may have seen in training; up to 1,000 new notes each, the real continuations all 1,000 long). With one instrument, D here measures only how close the note rate is to the real one (and runaway):

System	Parameters	Notes / s	Runaway	D	D, first 200 notes
Real continuation		7.95	0	0	0
musicar-L (temperature 0.9)	464M	8.95	0	0.040	0.019
Aria medium, continuation weights (arXiv:2506.23869)	659M	9.99	0.03	0.096	0.067

On the same prompts the project's 108M piano-cover fine-tune (not released) is closer still in note rate (D 0.018; 0.006 over the first 200 notes); its 450M piano-cover fine-tune of this model (not released), which writes short covers that end, is further (0.177; 0.078).

Limitations and biases

• The data lean toward Western popular and classical music in General MIDI instrumentation; other traditions, and instruments outside General MIDI, are thin or missing.
• GigaMIDI is collected from the web, so its quality varies, and the model learns from all of it.
• Aria-MIDI is transcribed from recordings, so it contains transcription errors (missed, extra or mistimed notes), which the model learns as if they were music.
• Control is limited to the prompt and the two window conditions (instrument set, notes per second): no control of genre, mood or form, and no text input.
• Long pieces lose structure: past 2,048 notes the window slides and earlier material leaves the context, and time is absolute (no bars or beats).
• Sampling can run away into very dense passages: when continuing 12 Ukrainian songs (one seed; temperature 0.9, top-p 0.9, Δt bias 0.1, conditioned on the prompt's instruments), 3 of the 12 continuations reached 45 to 179 notes per second.
• Only notes are modelled: no tempo, pedal or other controller events; velocity has 32 levels and time a 20 ms grid.
• Generated output may reproduce passages from the training data.

Future work

Done: the captioned fine-tune (F1) of this model (a short first stage on a laptop RTX 5060, then about 14 h on one rented RTX 5090), released as musicar-L-caption: captions from a frozen multilingual LaBSE encoder in English and Ukrainian, section captions that change during a piece, and length control. It keeps full attention: grouped-query attention was tested for it and dropped, because merging the heads broke this QK-norm model (held-out cross-entropy 2.01 → 6.57).

Planned, not done:

• F2: a captioned fine-tune on about 5,400 Ukrainian songs transcribed from audio.
• Later: regenerating parts of a piece (infilling), and inference on small devices (int8 / int4 on a Raspberry Pi 5); a grouped-query attention conversion would be its own optional step.
• A listening test against the other open models of the comparison above.

Files

File	What it is
model_bf16.pt	Training checkpoint: bf16 weights + model_args + the training config. This is what the repository's scripts load (generate.py --checkpoint, train.py with init_from='finetune')
model.safetensors + config.json	The same weights without pickle; config.json holds model_args
diagram.png	The architecture diagram above
findings.png	The project's evidence sheet: comparison with other models (section 1), training curves

License

CC BY-NC-SA 4.0. The training data are under non-commercial terms (Aria-MIDI CC BY-NC-SA 4.0, Discover MIDI CC BY-NC-SA, GigaMIDI non-commercial), so the model is released under the same non-commercial, share-alike terms. Check GigaMIDI's own terms before any reuse.

Citation

@misc{hilei2026musicar,
  author       = {Dmytro Hilei},
  title        = {musicar: autoregressive symbolic music models},
  year         = {2026},
  howpublished = {\url{https://github.com/DmytroHilei/Music_generative_model}},
  note         = {musicar-L}
}

Please also cite the training datasets: GigaMIDI (arXiv:2502.17726), Aria-MIDI (arXiv:2504.15071) and Discover MIDI (Hugging Face).