Instructions to use kashif/Qwen3-4B-Gated-LoRA-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use kashif/Qwen3-4B-Gated-LoRA-MTP with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B") model = PeftModel.from_pretrained(base_model, "kashif/Qwen3-4B-Gated-LoRA-MTP") - Notebooks
- Google Colab
- Kaggle
Qwen3-4B Gated-LoRA Multi-Token Prediction
A gated LoRA adapter with an eight-depth conditioned sampler for the frozen Qwen3-4B model. This revision contains checkpoint step 906 from the corrected one-epoch run on one H100.
The adapter contains all LoRA weights, the complete sampler MLP, eight fixed
input-only mask vectors, and their IDs. The sampler is saved through PEFT's
modules_to_save; base embedding and LM-head weights are excluded. Mask IDs
151700 through 151707 identify depths and do not alter the base vocabulary.
The sampler consists of two Linear / SiLU / LayerNorm blocks without a residual
connection. Each draft conditions on the previous predicted token. LoRA operates
only on mask positions; masks attend causally to the clean prefix and earlier
masks in the same draft block.
Draft logits and mask inputs
At inference, the eight saved mask vectors are appended through inputs_embeds;
the reserved mask IDs are not looked up in the unchanged base embedding table.
The clean prefix uses the frozen decoder, while LoRA is enabled only on the mask
positions. The base model produces the first next-token prediction. Each
subsequent depth uses:
refined = sampler(mask_hidden_state, embedding(previous_predicted_token))
draft_logits = frozen_lm_head(refined)
Drafting must use these sampler-refined logits. Directly projecting the raw mask
hidden state skips the trained sampler. The shared base LM head stays frozen;
there is no separate replacement output matrix to load. The sampler MLP and its
persistent mask buffers are included in adapter_model.safetensors through
modules_to_save=["gated_mtp_sampler"].
Training
- Base revision:
1cfa9a7208912126459214e8b04321603b3df60c. - LoRA rank 16, alpha 32; attention q/k/v/o and MLP gate/up/down projections.
- 52,706,304 trainable parameters: 33,030,144 LoRA plus 19,676,160 sampler.
- AdamW, learning rate 1e-4, betas (0.9, 0.999), no weight decay; gradient norm capped at 1.
- BF16 decoder computation, FP32 trainable parameter storage, activation checkpointing. LoRA is exported in BF16; sampler weights and frozen mask vectors are exported in FP32.
- Global batch 16, microbatch 4; one epoch, 907 optimizer steps numbered 0 through 906.
- Dolly rows 0โ14,499 train; rows 14,500โ15,010 validate. Assistant-answer loss masking, Qwen chat template, maximum 512 tokens, shuffled training, seed 1111.
- Objective: base CE plus conditioned-sampler CE on clean and future mask positions, with detached latent consistency matching. CE is normalized per sample over its supervised clean and mask positions; consistency balances matches per teacher position.
The original run's complete YAML was unavailable. The base model, adapter size,
mask count, data split, and epoch count were recovered from its metadata. Batch
size 16 was inferred from its step count; learning rate, sequence length, chat
formatting, and shuffle settings were reconstructed. This is a new run with the
corrected objective and saved sampler. Its exact configuration is in
training_config.yaml, with runtime versions in environment.json and source
files in automodel_source.tar.gz. The source files overlay the base commit
recorded in source_manifest.json.
Validation
| Checkpoint step | Frozen-base CE | Teacher-forced acceptance diagnostic |
|---|---|---|
| 299 | 4.0624 | 1.4695 |
| 599 | 4.0624 | 1.6357 |
| 899 | 4.0624 | 1.7284 |
The last full held-out validation is at step 899. The final step-906 adapter was checked separately for export and generation correctness.
The acceptance diagnostic compares sequential sampler drafts against frozen-base
greedy predictions on held-out teacher-forced prefixes and excludes ignored
anchors. It is not a decoding speed measurement or an autoregressive benchmark.
The final export is checked for finite tensors, exact sampler-state reload via
HF PEFT, unchanged logits with ordinary-token gating, and matching 16-token greedy
outputs on three prompts using draft verification. See adapter_validation.json
on main for those final-export checks.
Generation checks and measured throughput
These tests used the final step-906 weights at immutable revision
cd6f84de00dd1f5b9bc20e4ac585876290860f4f
on one NVIDIA H100 80GB, using greedy decoding with thinking disabled.
Two longer requests each generated 512 new tokens: an exercise guide and an explanation of sky color and sunsets. Both outputs were coherent on manual inspection, with connected sections and no repetitive collapse through the token limit. Both matched the uncached base-model greedy output token-for-token in these tests. Samples: exercise guide and sky explanation.
After a full 512-token warm-up of each path, three measurements were taken per prompt and method, alternating execution order. Values below are median output tokens per second, including prefill and generation time:
| Prompt | Cached base | Current MTP helper | MTP / base throughput |
|---|---|---|---|
| Exercise guide, 512 tokens | 54.65 | 28.84 | 0.53x |
| Sky explanation, 512 tokens | 54.58 | 28.46 | 0.52x |
The baseline is the same PEFT-wrapped model with the adapter disabled, using
Hugging Face generate(use_cache=True). The MTP helper uses eight mask depths and
recomputes the prefix without a KV cache. The current helper is about 1.9x
slower in these warmed measurements; it does not demonstrate an inference
speedup. An earlier cold measurement suggested a speedup and did not hold up
under repeated warmed timing. These are small reference-implementation tests,
not a comparison with an optimized serving engine.
MTP was active: the long runs accepted an average of 0.38 and 0.36 extra MTP draft tokens per verifier round, respectively. This count excludes the ordinary base next token and the verifier's correction/bonus token. Many proposed deeper tokens were rejected. Faster inference needs a KV-cached draft/verify implementation and sufficient accepted drafts to offset drafting and verification overhead.
BF16 numerical behavior
In an eight-prompt short-generation test, seven outputs matched uncached base
greedy generation exactly. One sky-explanation continuation diverged at a close
BF16 logit decision: the ordinary prefix forward scored appears at 39.5 and
is at 39.25; batched verification rounded both to 39.5, changing the argmax
choice. Ordinary-token gating still preserved the base logits exactly when the
forward layout was identical. Cached and uncached base generation also differed
on the longer prompts. Greedy text is therefore not guaranteed to be bitwise
identical across BF16 batching and cache layouts. This limitation should be
considered separately from sampler loading, token gating, and text coherence.
Raw results and benchmark scripts are included in benchmarks/:
short_generation_results.json, long_generation_results.json,
repeated_timing.json, and bf16_diagnosis.json. The decoder is a single-prompt,
uncached greedy correctness reference, with no stochastic speculative sampler.
Usage
Download inference.py and gated_mtp_sampler.py from this revision alongside
the adapter, then use the included loader:
from inference import load_adapter, greedy_generate
model, sampler, tokenizer = load_adapter("kashif/Qwen3-4B-Gated-LoRA-MTP")
text = tokenizer.apply_chat_template(
[{"role": "user", "content": "Name three benefits of regular exercise."}],
tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").input_ids.to("cuda")
outputs = greedy_generate(
model, sampler, inputs, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0, inputs.shape[1]:], skip_special_tokens=True))
To load an intermediate checkpoint, pass revision="step-299", "step-599", or
"step-899" to load_adapter. Ordinary generation must disable the adapter:
with model.disable_adapter():
outputs = model.generate(inputs, max_new_tokens=32, do_sample=False)
Standard PEFT loading requires attaching the sampler module to the base model
first. Plain PeftModel.from_pretrained does not create the sampler architecture,
token gating, or speculative decoding. The included helpers provide a single
unpadded prompt, uncached greedy reference implementation; they do not implement
KV-cache acceleration or stochastic speculative sampling. The examples were
checked with PyTorch 2.11.0, Transformers 5.18.0, and PEFT 0.20.0.
Earlier weights remain available through the archive-2026-11-14-* tags.
License
The adapter uses Apache 2.0. The training dataset is licensed CC BY-SA 3.0.
- Downloads last month
- 71