Qwen3-4B Gated-LoRA Multi-Token Prediction

A gated LoRA adapter with an eight-depth conditioned sampler for the frozen Qwen3-4B model. This revision contains checkpoint step 906 from the corrected one-epoch run on one H100.

The adapter contains all LoRA weights, the complete sampler MLP, eight fixed input-only mask vectors, and their IDs. The sampler is saved through PEFT's modules_to_save; base embedding and LM-head weights are excluded. Mask IDs 151700 through 151707 identify depths and do not alter the base vocabulary. The sampler consists of two Linear / SiLU / LayerNorm blocks without a residual connection. Each draft conditions on the previous predicted token. LoRA operates only on mask positions; masks attend causally to the clean prefix and earlier masks in the same draft block.

Draft logits and mask inputs

At inference, the eight saved mask vectors are appended through inputs_embeds; the reserved mask IDs are not looked up in the unchanged base embedding table. The clean prefix uses the frozen decoder, while LoRA is enabled only on the mask positions. The base model produces the first next-token prediction. Each subsequent depth uses:

refined = sampler(mask_hidden_state, embedding(previous_predicted_token))
draft_logits = frozen_lm_head(refined)

Drafting must use these sampler-refined logits. Directly projecting the raw mask hidden state skips the trained sampler. The shared base LM head stays frozen; there is no separate replacement output matrix to load. The sampler MLP and its persistent mask buffers are included in adapter_model.safetensors through modules_to_save=["gated_mtp_sampler"].

Training

  • Base revision: 1cfa9a7208912126459214e8b04321603b3df60c.
  • LoRA rank 16, alpha 32; attention q/k/v/o and MLP gate/up/down projections.
  • 52,706,304 trainable parameters: 33,030,144 LoRA plus 19,676,160 sampler.
  • AdamW, learning rate 1e-4, betas (0.9, 0.999), no weight decay; gradient norm capped at 1.
  • BF16 decoder computation, FP32 trainable parameter storage, activation checkpointing. LoRA is exported in BF16; sampler weights and frozen mask vectors are exported in FP32.
  • Global batch 16, microbatch 4; one epoch, 907 optimizer steps numbered 0 through 906.
  • Dolly rows 0โ€“14,499 train; rows 14,500โ€“15,010 validate. Assistant-answer loss masking, Qwen chat template, maximum 512 tokens, shuffled training, seed 1111.
  • Objective: base CE plus conditioned-sampler CE on clean and future mask positions, with detached latent consistency matching. CE is normalized per sample over its supervised clean and mask positions; consistency balances matches per teacher position.

The original run's complete YAML was unavailable. The base model, adapter size, mask count, data split, and epoch count were recovered from its metadata. Batch size 16 was inferred from its step count; learning rate, sequence length, chat formatting, and shuffle settings were reconstructed. This is a new run with the corrected objective and saved sampler. Its exact configuration is in training_config.yaml, with runtime versions in environment.json and source files in automodel_source.tar.gz. The source files overlay the base commit recorded in source_manifest.json.

Validation

Checkpoint step Frozen-base CE Teacher-forced acceptance diagnostic
299 4.0624 1.4695
599 4.0624 1.6357
899 4.0624 1.7284

The last full held-out validation is at step 899. The final step-906 adapter was checked separately for export and generation correctness.

The acceptance diagnostic compares sequential sampler drafts against frozen-base greedy predictions on held-out teacher-forced prefixes and excludes ignored anchors. It is not a decoding speed measurement or an autoregressive benchmark. The final export is checked for finite tensors, exact sampler-state reload via HF PEFT, unchanged logits with ordinary-token gating, and matching 16-token greedy outputs on three prompts using draft verification. See adapter_validation.json on main for those final-export checks.

Generation checks and measured throughput

These tests used the final step-906 weights at immutable revision cd6f84de00dd1f5b9bc20e4ac585876290860f4f on one NVIDIA H100 80GB, using greedy decoding with thinking disabled.

Two longer requests each generated 512 new tokens: an exercise guide and an explanation of sky color and sunsets. Both outputs were coherent on manual inspection, with connected sections and no repetitive collapse through the token limit. Both matched the uncached base-model greedy output token-for-token in these tests. Samples: exercise guide and sky explanation.

After a full 512-token warm-up of each path, three measurements were taken per prompt and method, alternating execution order. Values below are median output tokens per second, including prefill and generation time:

Prompt Cached base Current MTP helper MTP / base throughput
Exercise guide, 512 tokens 54.65 28.84 0.53x
Sky explanation, 512 tokens 54.58 28.46 0.52x

The baseline is the same PEFT-wrapped model with the adapter disabled, using Hugging Face generate(use_cache=True). The MTP helper uses eight mask depths and recomputes the prefix without a KV cache. The current helper is about 1.9x slower in these warmed measurements; it does not demonstrate an inference speedup. An earlier cold measurement suggested a speedup and did not hold up under repeated warmed timing. These are small reference-implementation tests, not a comparison with an optimized serving engine.

MTP was active: the long runs accepted an average of 0.38 and 0.36 extra MTP draft tokens per verifier round, respectively. This count excludes the ordinary base next token and the verifier's correction/bonus token. Many proposed deeper tokens were rejected. Faster inference needs a KV-cached draft/verify implementation and sufficient accepted drafts to offset drafting and verification overhead.

BF16 numerical behavior

In an eight-prompt short-generation test, seven outputs matched uncached base greedy generation exactly. One sky-explanation continuation diverged at a close BF16 logit decision: the ordinary prefix forward scored appears at 39.5 and is at 39.25; batched verification rounded both to 39.5, changing the argmax choice. Ordinary-token gating still preserved the base logits exactly when the forward layout was identical. Cached and uncached base generation also differed on the longer prompts. Greedy text is therefore not guaranteed to be bitwise identical across BF16 batching and cache layouts. This limitation should be considered separately from sampler loading, token gating, and text coherence.

Raw results and benchmark scripts are included in benchmarks/: short_generation_results.json, long_generation_results.json, repeated_timing.json, and bf16_diagnosis.json. The decoder is a single-prompt, uncached greedy correctness reference, with no stochastic speculative sampler.

Usage

Download inference.py and gated_mtp_sampler.py from this revision alongside the adapter, then use the included loader:

from inference import load_adapter, greedy_generate

model, sampler, tokenizer = load_adapter("kashif/Qwen3-4B-Gated-LoRA-MTP")
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Name three benefits of regular exercise."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").input_ids.to("cuda")
outputs = greedy_generate(
    model, sampler, inputs, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0, inputs.shape[1]:], skip_special_tokens=True))

To load an intermediate checkpoint, pass revision="step-299", "step-599", or "step-899" to load_adapter. Ordinary generation must disable the adapter:

with model.disable_adapter():
    outputs = model.generate(inputs, max_new_tokens=32, do_sample=False)

Standard PEFT loading requires attaching the sampler module to the base model first. Plain PeftModel.from_pretrained does not create the sampler architecture, token gating, or speculative decoding. The included helpers provide a single unpadded prompt, uncached greedy reference implementation; they do not implement KV-cache acceleration or stochastic speculative sampling. The examples were checked with PyTorch 2.11.0, Transformers 5.18.0, and PEFT 0.20.0.

Earlier weights remain available through the archive-2026-11-14-* tags.

License

The adapter uses Apache 2.0. The training dataset is licensed CC BY-SA 3.0.

Downloads last month
71
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for kashif/Qwen3-4B-Gated-LoRA-MTP

Finetuned
Qwen/Qwen3-4B
Adapter
(1,264)
this model

Dataset used to train kashif/Qwen3-4B-Gated-LoRA-MTP