Instructions to use canada-quant/glm-5.3-w4a16-mtp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use canada-quant/glm-5.3-w4a16-mtp with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="canada-quant/glm-5.3-w4a16-mtp") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("canada-quant/glm-5.3-w4a16-mtp") model = AutoModelForMultimodalLM.from_pretrained("canada-quant/glm-5.3-w4a16-mtp", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use canada-quant/glm-5.3-w4a16-mtp with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "canada-quant/glm-5.3-w4a16-mtp" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "canada-quant/glm-5.3-w4a16-mtp", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/canada-quant/glm-5.3-w4a16-mtp
- SGLang
How to use canada-quant/glm-5.3-w4a16-mtp with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "canada-quant/glm-5.3-w4a16-mtp" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "canada-quant/glm-5.3-w4a16-mtp", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "canada-quant/glm-5.3-w4a16-mtp" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "canada-quant/glm-5.3-w4a16-mtp", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use canada-quant/glm-5.3-w4a16-mtp with Docker Model Runner:
docker model run hf.co/canada-quant/glm-5.3-w4a16-mtp
- GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP
- 🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)
- Why this quant exists
- Hardware
- Benchmarks vs competitors
- vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)
- vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024
- vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2
- Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)
- Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE
- 2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode
- Quality
- Serving
- Docker images
- Quick start
- Details
- Validation
- License
- 🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)
GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP
INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 MTP draft head preserved for speculative decoding. 177.7 GiB (−70%) — serves on 4× H100/H200, 4× RTX PRO 6000, or 2× DGX Spark at full 1M context.
🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)
- −70% weights: 177.7 GiB vs ~599 GiB BF16 → 4 datacenter GPUs instead of 8, or a 2× DGX Spark desktop pair at full 1M context.
- vs NVFP4 (LibertAI, strongest community quant) — 2× DGX Spark: W4A16 serves 1,048,576 tokens there (KV pool 1.30×); the NVFP4 checkpoint OOM'd 9/9 boots at any context on identical nodes (dmesg-verified). Throughput @262K: +53% seq1 / +101% seq6.
- vs NVFP4 — H100 TP=4, matched MTP: wins c1 +4.8% · c8 +13.6% · saturated c32 +14.0% (vs NVFP4's best c32 config); mtp-off fixed-drafter c32 +22.4% — the batch edge is the marlin INT4 MoE GEMM, not the drafter.
- vs EXL3 (MiaAI-Lab 2×-Spark recipe) — independent matched-protocol head-to-head, FINAL: W4A16 leads single-stream +10.4% (33.0 vs 29.9 tok/s) and long-prefill +31–39% (~1.2K vs 864–914 tok/s @300K-class); KV capacity 1.30× vs 1.20×. The vendor README overstated decode ~2.2× and prefill ~1.2× vs independent measurement.
- Quality: AIME 2026 85.0% (102/120, n=120 final, max thinking) — beats the matched EXL3 arm 96/120 (80.0%) by +6 correct (+5.0 pt), FINAL 2026-09-06; AIME 2025 parity with NVFP4 on H100 (0.8833 vs 0.9000, 0.42σ); GSM8K 0.97; GPQA-Diamond within noise.
- SM120 (Blackwell): throughput parity with NVFP4 (±0.7%) with MTP paying at every concurrency — +69.9% c1 / +37.6% c8 / +47.8% c32.
- Long context: single-stream decode is KV-length-insensitive to ≥486K (up to 357 tok/s, bf16 KV); long-prefill ceiling ~6.1–8.5K tok/s aggregate across the full 3-KV × c1–c32 matrix.
- Research-stack bug found and fixed: a 21-leg investigation root-caused a hang-forever admission wedge in the DFlash2 spec-decode stack and validated a fix to 256,388-token prompts (24/24 quality spot-checks pass).
Honest ledger: EXL3 leads mid-concurrency (c2 +67% / c4 +89%; ≈+81% cap-corrected c6); both sides saturate at c32 (EXL3 881–898 agg tok/s). SM120 vs NVFP4 is parity. The matched AIME-2026 quality arm vs EXL3 is FINAL 2026-09-06: W4A16 102/120 (85.0%) vs EXL3 96/120 (80.0%) — W4A16 wins. Results as of 2026-09-06; in-flight arms are labeled.
Why this quant exists
- BF16 GLM-5.3-Flash needs eight 80 GB GPUs per replica; this artifact cuts weights to 177.7 GiB — four GPUs, or a dual DGX Spark desktop at 1M context.
- Only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ); attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head (+40–50% single-stream) stay unquantized; canonical FP32 tensors preserved verbatim.
- Not a new model — all capability comes from the base model.
Hardware
| Config | GPUs | Context | Notes |
|---|---|---|---|
| H100/H200 (SM90), TP=4 | 4× 80 GB | 262K, bf16 KV | bench-validated recipe |
| H200 (SM90), TP=2 | 2× 141 GB | ~64–128K, fp8 KV | fit-estimated (not rig-measured) |
| RTX PRO 6000 (SM120), TP=4 | 4× 96 GB | 262K, fp8 KV | bench-validated |
| DGX Spark GB10 (SM121), TP=2 | 2× 128 GB UMA | 1M, fp8 KV | DFlash2 stack — recipe below |
| B300 (SM103) | 8× | — | quantization node (transformers smoke-tested) |
Benchmarks vs competitors
Identical flags/prompts/hardware per row. Throughput = output tok/s, thinking ON.
vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)
- EXL3 was brought to full serving at 1M context on the same hardware (KV pool 1,200,348 = 1.20×; W4A16 1,360,420 = 1.30×).
- Like-for-like decode (TTFT excluded, 8K prompts, 256-token completions): W4A16 +10.4% c1 (33.0 vs 29.9 agg tok/s).
- EXL3 leads mid-concurrency: c2 +67.4% (59.6 vs 35.6), c4 +89.1% (112.7 vs 59.6), c6 ≈+80.9% after correcting its max-num-seqs=4 scheduler cap; it sustains ~29 tok/s per stream at every concurrency (W4A16 steps down 33.0→11.2 seq1→6 on the 1M serve; the 262K serve holds ~20 at seq4+).
- Saturated c32 (clean refire): EXL3 881–898 agg tok/s (32/32 streams, per-stream 27.4–28.1).
- W4A16 +31–39% long-prefill (~1.2K tok/s vs 864–914 @300K-class prompts).
- Vendor README overstated measured decode ~2.2× (claimed 62.9/51.7/37.1 per-stream c1/2/4 vs measured 26.6/21.2/17.0) and prefill ~1.2×.
- Two mid-bench hardware deaths on the competitor node were quarantined and never banked; the c32 set was re-run clean.
- Matched AIME-2026 quality arm: FINAL 2026-09-06 — W4A16 102/120 (85.0%) beats EXL3 96/120 (80.0%), +6 correct (+5.0 pt). EXL3 epochs 25/25/24/22, zero harness errors; 23/120 EXL3 rows hit the 131,072-token cap (W4A16: 16/18 misses were budget-exhaustion) — scored as written under the identical temp-0.0 / 131,072-token / n=120 config as our banked 85.0% arm.
vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024
| concurrency | W4A16 MTP N=2 (this) | NVFP4 MTP N=2 | NVFP4 MTP N=5 (its best c32 config) |
|---|---|---|---|
| 1 | 183.89 | 180.96 | 175.44 |
| 8 | 249.94 | 251.14 | 220.05 |
| 32 | 1161.13 † | 770.66 | 1018.13 |
| MTP acceptance | 52–55% | ~52% | ~30% |
- Headline: +4.8% c1 / +13.6% c8 / +14.0% saturated c32 vs NVFP4's banked N=5 config.
- † c=32 with
--gpu-memory-utilization 0.92(KV-pool capacity): 1161.13 tok/s (independent tuned run: 1145.05). The c=32 lever is KV-pool size, not a kernel gap. - Fixed-drafter 2×2 re-confirm (both arms on the corrected #54374-port overlay): matched c1/c8 within noise (180.22/254.39 vs 182.13/252.89); saturated c32 win holds (+18.1%, 804.02 vs 680.99).
- MTP-off on both: W4A16 leads +2.7% / +10.5% / +22.4% at c1/c8/c32 — the c32 advantage is the marlin INT4 MoE GEMM at batch, not an MTP artifact.
- MTP adds +41.1%/+14.3%/−4.0% (W4A16) and +46.4%/+25.6%/−0.4% (NVFP4) at c1/c8/c32 on SM90.
vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2
| concurrency | W4A16 (this) | NVFP4 | Δ |
|---|---|---|---|
| 1 | 109.90 | 109.34 | +0.5% |
| 8 | 318.54 | 319.46 | −0.3% |
| 32 | 534.37 | 530.75 | +0.7% |
- Parity within noise (±0.7%); verbatim-methodology re-grid: c1 106.97 (−2.7%, 8-prompt high-variance class), c8 320.51 (+0.6%), c32 539.63 (+1.0%), MTP acceptance 50–53% — no stack drift.
- MTP pays at every concurrency on SM120: +69.9% c1 (106.97 vs 62.96), +37.6% c8 (320.51 vs 232.89), +47.8% c32 (539.63 vs 365.02) — a sign flip vs SM90 (drafter costs ~4% at saturated c32 there). Keep MTP on at every concurrency on SM120.
- KV-dtype at batch: fp8 vs bf16 costs nothing at c1/c8 (±0.05%) and ≈4.7% at c32 (365.02 vs 382.07) — lever ranking MTP ≫ KV dtype. (
autois the working bf16-KV path;bf16/bfloat16variants fail on this stack.)
Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)
Per-stream decode tok/s (full-contention cohort bands — serialized prefills split stages into early/late cohorts, so bands, not blended medians):
| KV length | KV dtype | c1 | c4 | c8 | c32 |
|---|---|---|---|---|---|
| 8K (aggregate grid) | fp8 | 106.97 | — | 40.06 | 16.86 |
| 138.6K | fp8 | 143.94 | 2.92–7.53 | 2.49–3.74 | 2.24–2.72 |
| 138.6K | bf16 (auto) |
152.19 | 3.23–4.21 | 2.53–2.64 | 2.45–2.63 |
| 277K | bf16 (auto) |
177.15 | 2.23–6.15 | 2.28–4.32 | 2.17–2.79 |
| 486K | bf16 (auto) |
357.34* | 2.27–3.32 | 2.31–2.68 | 2.28–4.27† |
| 486K | fp8 | (degenerate‡) | 2.31–3.26 | 2.37–3.62 | 2.03–3.93† |
* n=1 short completion, noisy. † 21/32 rows — 11 bench-client TimeoutErrors disclosed in-repo. ‡ the fp8-KV c1 row recorded an absurd >480K tok/s (bench artifact) and is never cited.
Additional bands: 277K c2 5.03, c6 2.29–3.38; 486K c6 2.34–3.68 (bf16) / 2.34–3.60 (fp8); 138.6K fp8 c2 9.58.
- Single-stream decode is KV-length-insensitive to ≥486K (143.94 → 177.15 → 357.34 tok/s at 138.6K/277K/486K bf16 — rises, if anything).
- At-batch collapse saturates with KV length:
13× per-stream drop at c8 vs the 8K grid, then flat ~2.2–3.7 tok/s/stream at c4–c32 from 138.6K through 486K — mechanism is target-model forward compute at batch × long-KV (4.4M tokens of live KV at c32), not the drafter. - KV dtype is a c1-only ~5% lever (bf16 +5.7% @138.6K); at-batch bands overlap fp8/bf16 — the long-KV-amplifies-fp8-cost hypothesis is refuted.
- MTP acceptance never collapses: c1 ≈80–86% → c8 73.9% (n=1,286) → c32 71.6% (n=4,714) at 138.6K; 60.0% single-stream at 486K (small sample, disclosed). The saturated c32 stage decoded 2.53 tok/s/stream while the drafter accepted 71.6% — conclusively not acceptance-driven.
- At-batch lever ranking FINAL: MTP drafter ≫ everything measured; KV dtype ≈ c1-only ~5%; drafter acceptance is not the bottleneck.
Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE
- Single-stream prefill: ~6,100–7,100 tok/s for 277K–486K prompts (TTFT first request 39.2–79.3 s).
- At batch, long prefills are essentially serialized (stage wall ≈ N × single-stream time).
- Aggregate prefill ceiling ~6.1–8.5K tok/s — KV-length-insensitive AND batch-insensitive:
- 138.6K: 6,735 (c1) → 7,134 (c8) → 7,897 (c16) → 8,486 (c32) tok/s.
- 277K: 5,211 (c1) → 7,177 (c8) → 7,784 (c16) → 7,357 (c32) tok/s.
- 486K: 6,136 (c1) → 6,892 (c8) → 7,483 (c16) → 6,988.6 (c32) tok/s.
- Serialization ratios (wall / N×c1): 138.6K 0.94/0.85/0.79 · 277K 0.73/0.67/0.71 · 486K 0.89/0.82/0.88 (c8/c16/c32) — deeper-KV rows behave closer to strict serialization.
- mnb lever fully mapped: 8192 ✓ (validated production config) → 16384 ✓ (validated fallback, ceiling within ~1% at both 277K and 486K) → 32768 ✗ (CUDA OOM at 486K — 3.98 GiB needed vs 1.70 GiB free at mem-util 0.95). Ceiling is chunk-size-insensitive → attention/forward compute-bound, not scheduling-bound.
- Operational guidance: admission pacing / prefill scheduling is the only lever that moves long-context batch UX (mnb tuning and MTP do not address prefill serialization).
2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode
8K prompts, 256-token completions, aggregate tok/s:
| config | seq1 | seq2 | seq4 | seq6 |
|---|---|---|---|---|
| W4A16+DFlash2 @262K (this) | 33.29 | 42.01 | 80.98 | 121.81 |
| NVFP4-MTP4 reference | 21.8 | 41 | 66 | 60.5 |
| Δ | +53% | +2.4% | +23% | +101% |
| W4A16+DFlash2 @1M serve | 32.99 | 35.61 | 59.61 | 67.05 |
- 256K-token single prompt on the 1M serve: 30.85 tok/s decode (ttft 5.8 s).
- Structural: the NVFP4 checkpoint (90.67 GiB/node weights vs ~30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.
Quality
- AIME 2026 (30 problems, max thinking, 131,072-token budget, DFlash2 stack, 2× Spark): 102/120 = 85.0% — n=120 FINAL. Epochs 28/30, 25/30, 26/30, 23/30; 16/18 misses are budget-exhaustion truncations, 2/18 genuine wrong answers. The LibertAI NVFP4 external AIME-2025 reference (0.9000) remains +5.0 points ahead; the matched AIME-2026 EXL3 arm is FINAL 2026-09-06: 96/120 (80.0%) — W4A16 wins by +6 correct (+5.0 pt).
- AIME 2025 n=120, 4×30, max thinking, 131,072 budget — H100 (SM90): parity — W4A16 0.8833 (106/120) vs NVFP4 0.9000 (0.42σ). An earlier 0.7833 read was a serving-stack drafter-corruption bug (upstream vLLM #54374), since fixed.
- AIME 2025 — RTX PRO 6000 (SM120): raw-arm deficit real (~2σ; 0.8083/0.8000 pooled vs 0.9000), mechanism decomposed and fix VALIDATED. ≈63% of the deficit is a mechanical budget-wall (EMPTY-answer rate 11.7–14.2% vs NVFP4's 3.3%; doubling the budget recovered +5.8 pt); ≈37% is SM120 kernel numerics (near-tie argmax flips vs SM90, logit-quantified).
- Fix stack VALIDATED (zero GPU cost): whole-trace commit hook + thinking-budget force-commit (98,304-token cap) lifts the score to 0.8833 two-way / 0.8917 hook-credited — within noise of NVFP4 0.9000 (0.42σ / 0.21σ).
- Production-overlay pilot passed (committed 0.8917, zero modified non-empty responses across 120 requests); production adoption remains gated on the eval sign-off chain (owner decision pending).
- GSM8K: parity — 0.9704–0.9750 across all three quants on both x86 rigs.
- GPQA-Diamond n=198 @131k: 0.8586 vs 0.8687 NVFP4 (SM90), 0.8586 vs 0.8737 (SM120) — within noise.
- Matched AIME-2026 single-epoch arms on x86 (n=30, max thinking): W4A16 24/30 (RTX PRO 6000) and 22/30 (H100) vs NVFP4 23/30 — within noise at n=30.
Serving
The same checkpoint serves on every NVIDIA architecture from Hopper through Blackwell. Each recipe is the exact config the benchmarks ran.
SM90 — H100 / H200 (recommended: TP=4)
Image vllm/vllm-openai:glm53-flash-x86_64-cu130 (CUDA 13.0). Gotcha: always pass --max-num-seqs ≤512 — the default 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
-v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
vllm serve /models/glm53-flash-w4a16-mtp \
--served-model-name glm53-w4 \
--tensor-parallel-size 4 --enable-expert-parallel \
--max-model-len 262144 --max-num-seqs 512 \
--block-size 64 --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--trust-remote-code --port 8000
- MTP
num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance, best c1/c8); N=5 helps only at c=32. TP=8 works too.
SM120 — RTX PRO 6000 (Blackwell)
Image cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 (SM120-patched build).
docker run --gpus '"device=4,5,6,7"' --ipc=host --network=host --rm \
-v /models:/models cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 \
vllm serve /models/glm53-flash-w4a16-mtp \
--served-model-name glm53-w4 \
--tensor-parallel-size 4 --enable-expert-parallel \
--max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 \
--kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--trust-remote-code --port 8001
- KV fp8 is required at 262K on 96 GB cards;
--max-num-seqs 64is bench-validated (48/36 fallback ladder); MTP N=2 matches NVFP4 to ±0.7%.
SM121 — 2× DGX Spark GB10 (desktop, 1M context)
Full 1,048,576-token context on two Sparks over RoCE (TP=2) with the DFlash2 block-diffusion drafter:
- Image
radixark/vllm-glm53-flash:sm121-v11-dflash2; drafterGLM-5.3-Flash-DFlash2(block 8, selector_rank 256, top_k 16, layers [5,14,24,33,42]). MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416→ KV fp8_e4m3 pool 1,360,420 tokens (1.30× a full 1M request); staging @262K =MAX_MODEL_LEN=262144 KV_CACHE_MEM=3221225472.--max-num-seqs 6 --block-size 2304 --enforce-eager; 7 speculative tokens (any other count boot-wedges).- Worker rank first, 25 s, then head rank; cold boot ≈6–8 min.
- NCCL on the RoCE switch path (97.98 Gbps line-rate; direct-attach measured worse).
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600(cold JIT otherwise kills the boot).- Keep single prompts ≤ ~310K tokens (larger wedged a host twice).
Docker images
Upstream note (2026-09-03): GLM-5.3-Flash support landed in vLLM main (vllm-project/vllm#53906, merged 16:40 UTC). The pinned images below remain the validated serving stacks for this quant; upstream-main validation is a separate, queued leg.
| Architecture | Image |
|---|---|
| SM90 (H100/H200) | vllm/vllm-openai:glm53-flash-x86_64-cu130 |
| SM120 (RTX PRO 6000) | cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 |
| SM121 (DGX Spark) | radixark/vllm-glm53-flash:sm121-v11-dflash2 |
Quick start
# 1. Download the weights (~178 GiB)
huggingface-cli download canada-quant/glm-5.3-w4a16-mtp --local-dir /models/glm53-flash-w4a16-mtp
# 2. Serve (SM90 example — see recipes above)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
-v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
--tensor-parallel-size 4 --enable-expert-parallel --max-num-seqs 512 \
--max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--trust-remote-code
# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "glm53-w4",
"messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'
Details
| Field | Value |
|---|---|
| Base model | zai-org/GLM-5.3-Flash (BF16) |
| Architecture | Glm5NextForConditionalGeneration (glm5_next) — 45 decoder layers (+ MTP layer 45), hidden 4096, 288 routed experts (top-8) + 1 shared, MoE-intermediate 2048, KDA + DSA attention, 24-block vision tower, vocab 154,880 |
| Weight quantization | W4A16, INT4, symmetric, group-size 128 (GPTQ, compressed-tensors pack-quantized), routed experts only |
| Quantized tensors | 36,288 (42 MoE layers × 288 experts × 3 GEMMs) — verified exact |
| Kept in BF16 | attention (KDA + DSA incl. indexer), dense prefix (layers 0–2), shared experts, router/gate, mHC tensors, embeddings, lm_head, norms, vision tower (348 keys), MTP layer 45 (889 keys) |
| Canonical FP32 preserved | A_log, dt_bias, e_score_correction_bias, hc_* — verbatim from source |
| Calibration | 256 samples × 4096 tokens, in-distribution chat/code mix (glm53_mix), sequential per-layer GPTQ |
| Size | 177.7 GiB (from ~599 GiB BF16) |
| License | MIT (inherited from the base model) |
vLLM serving note: the checkpoint's
quantization_config.ignorealready excludes the vision tower andre:model\.language_model\.layers\.45\..*, so the MTP head loads BF16 for speculative decoding out of the box.
Validation
Static gates at build time (2026-08-27), all PASS:
- exactly 36,288 packed tensors; zero compression outside routed experts;
- vision key-set 348/348 identical to source, zero compressed, all dtypes match;
- MTP layer present (889 keys), index rewrite purely additive;
- dtype drift vs source: zero (282 pipeline-drifted tensors restored verbatim);
- expert scale audit: no zero/collapsed scales.
Quantized on 8× NVIDIA B300, 2026-08-27. B300 smoke tests (transformers ≥5.16): loads as Glm5NextForConditionalGeneration (~110 s); text generation coherent; image captioning accurate.
Vision-quality note: the vision tower is BF16-passthrough but was not covered by the text-only calibration set. Image smoke tests pass; a vision eval leg (MMMU-class + OCR) is queued.
Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)
Affects the SM90 research stack only (x86_64, DFlash2 drafter n=7). The standard MTP serve path used by all production configs above is unaffected.
- Symptom: prompts above a block-size-dependent bound are never admitted —
Waiting: 1 / Running: 0, 0% KV, retry-forever until timeout (no crash, no OOM). - Measured bounds: block 2304 → wedges above ~15.5K tokens; block 1024 → ~17.4K; block 1536 → no wedge at all across the tested 16–24K class.
- Mechanism (pinned): spec-decode admission requires a full-sequence reservation against the unified KV pool (1,012 blocks); the binding group is the drafter's SlidingWindowManager (block-16) — a 15,523-token request needs exactly 1,011 blocks (passes, zero margin), a ~15,556-token request needs 1,013 (wedges).
- Root cause (pinned): the drafter SW manager allocates execution-side at full-sequence scale (cdiv(tokens,16)) while its true sliding-window need is only 130 blocks; the built-in skip-free mechanism fires but freed blocks are instantly re-consumed by the next full-sequence allocation.
- Fix VALIDATED: cap the SW manager's execution allocation at
min(cdiv(tokens, bs), cdiv(sliding_window, bs) + 2)(= 130 blocks) — serves 8,528 / 15,787 / 25,344 / 63,734 / 127,952 / 256,388 tokens, all PASS at the shipping block-2304 config (pre-fix wedge at ≥15,537), zero engine errors. - Quality spot-suite PASS: 24 deterministic wedge-class prompts, fixed stack vs spec-OFF control — 24/24 checks both arms, zero tokenization mismatch; no detectable quality impact at spot scope.
- Workarounds without the patch: block 1536 serves the whole tested 16–24K class (its SW cap saturates) and holds the largest KV pool (1.12× of a 262K request vs 0.78× at block 2304).
- Upstream status: the stock upstream DFlash2 image cannot boot this drafter at all, so the defect is provable only on the research overlay; the closest upstream PR (#50169) would not fix it (no execution-side cap). A fully-evidenced issue draft is finalized in-repo (filing = owner decision); production adoption of the patch = owner decision.
- Full leg chain:
findings/df2w*.md(DF2W → DF2W27).
License
MIT, inherited from the base model. Follow the base model's usage terms.
Built, benchmarked, and documented with the Digby.ai coding harness, developed by CQL.ca.
- Downloads last month
- 781
Model tree for canada-quant/glm-5.3-w4a16-mtp
Base model
zai-org/GLM-5.3-Flash