GLM-5.3 quantization accuracy and MTP comparison
Generated: 2026-08-31 06:39:50
Three checkpoints: the official FP8 release, this W4A8 (compressed-tensors), and PhalaCloud's published W4AFP8 (SGLang-native format; numbers taken from that model card, not reproduced locally).
All local numbers were measured with vLLM. The W4A8 checkpoint also loads and runs on SGLang (see the model card, Deploy / SGLang), but evals were not rerun there. The kernels differ; do not copy numbers across engines.
1. Checkpoint basics
| GLM-5.3 (FP8 original) | GLM-5.3-W4A8 (this work) | PhalaCloud W4AFP8 (reported) | |
|---|---|---|---|
| Size | 703.7 GiB | 372.3 GiB | 372.3 GiB |
| Expert weights | FP8 block 128×128 | INT4 group-128 (RTN + MSE clip search) | INT4 group-128 (AWQ calibration) |
| Quantized from | Official release | Dequant then requant of the FP8 release | BF16 parent |
| Non-expert layers | FP8 block | FP8 block (byte-identical) | FP8 block |
| Activations | FP8 dynamic | FP8 per-token dynamic | FP8 dynamic |
| Format | fp8 | compressed-tensors (experts pack-quantized, non-experts float-quantized) | w4afp8 (SGLang) |
| Engines | vLLM / SGLang | vLLM / SGLang | SGLang (vLLM support in PR) |
| Expert reconstruction relative error | 0.0265 | 0.103 | 0.108–0.13 (their card) |
2. GPQA-Diamond
198 questions, 4 samples each, temperature 1.0 / top_p 0.95. Option order is shuffled per question ID; both models see identical prompts.
| Metric | GLM-5.3 (FP8) | GLM-5.3-W4A8 | PhalaCloud W4AFP8 (reported) |
|---|---|---|---|
| pass@1 | 89.65 | 90.28 | 91.92 (182/198) |
| majority@4 | 90.91 | 90.91 | — |
| Answer parse failures | 0 | 0 | — |
| Truncated by length | 122 | 131 | — |
| Mean output tokens | 12159 | 12661 | — |
W4A8 vs FP8 pass@1: +0.63 percentage points. At 198 × 4 samples, 1pp is about 8 samples — treat as sampling noise.
PhalaCloud numbers come from a different engine (SGLang) and protocol (truncation retry). For reference only; do not subtract from the first two columns.
3. Teacher-forced NLL (held-out text, lower is better)
64 windows × 2048 tokens of held-out text. Both models score the exact same token sequences. This is the most quantization-sensitive, low-noise metric.
| Metric | GLM-5.3 (FP8) | GLM-5.3-W4A8 | PhalaCloud W4AFP8 (reported) |
|---|---|---|---|
| NLL (nats/token) | 0.8929 | 0.8940 | +0.282 nats vs BF16 |
| Perplexity | 2.442 | 2.445 | — |
| SE across windows | 0.0740 | 0.0743 | — |
W4A8 vs FP8 dNLL: +0.0010 nats/token. Baselines differ: PhalaCloud's +0.282 is vs the BF16 parent; this table is vs the FP8 release.
4. MTP / speculative decoding
vLLM loads the layer-78 draft head via glm_moe_dsa -> deepseek_mtp -> DeepSeekMTPModel.
Test: 512-token input / 256-token output, MTP on vs off.
| Checkpoint | Concurrency | no MTP (out tok/s) | MTP on (out tok/s) | Speedup |
|---|---|---|---|---|
| GLM-5.3 (FP8) | 1 | 86.2 | 148.7 | 1.72× |
| GLM-5.3 (FP8) | 4 | 264.7 | 349.6 | 1.32× |
| GLM-5.3 (FP8) | 16 | 550.8 | 695.0 | 1.26× |
| GLM-5.3 (FP8) | 64 | 1104.6 | 1306.2 | 1.18× |
| GLM-5.3-W4A8 | 1 | 66.2 | 103.5 | 1.56× |
| GLM-5.3-W4A8 | 4 | 196.4 | 281.8 | 1.44× |
| GLM-5.3-W4A8 | 16 | 480.1 | 572.0 | 1.19× |
| GLM-5.3-W4A8 | 64 | 962.8 | 1190.1 | 1.24× |
PhalaCloud reported acceptance length ~2.93 (EAGLE steps=3), which is not
comparable to num_speculative_tokens=1 here.
5. PhalaCloud card items not reproduced locally
| Benchmark | PhalaCloud W4AFP8 | Why not reproduced |
|---|---|---|
| AA-LCR | 73.0 | Dataset and judge model not public |
| BFCL (45-item live subset) | 82.2 | Custom subset composition not published |
| NIAH @ ~930k tokens | 3/3 | See below |
NIAH is worth a note: W4A8 TP8 KV capacity is 1,257,280 tokens, enough for a 930k-token prompt; original FP8 only has 608,128, physically does not fit, so that item cannot be compared.
6. Performance and memory (previously measured, same protocol)
| Setup | Weight VRAM / GPU | KV capacity | prefill 8 conc. × 8K in | decode 128 conc. | decode 256 conc. |
|---|---|---|---|---|---|
| FP8 TP8 | ~88 GiB | 608,128 | 3675 | 1686 | 2094 |
| W4A8 TP8+EP | 52.6 GiB | 1,257,280 | 3441 | 1617 | 1779 |
| W4A8 2×(TP4+EP) | 93 GiB | 638,336 | 5993 | 2256 | 3249 |
Units are tok/s. No FP8 TP4 option: 704 GiB / 4 = 176 GiB per GPU, over the 141 GiB memory cap.