gemma-4-31B-MXFP8-FP8KV-CT-AutoRound

This repository contains a measured MXFP8 compressed-tensors checkpoint of Google's dense Gemma 4 31B model. The transformer Linear weights use 8-bit microscaling with group size 32; the export also includes calibrated static FP8 KV-cache scales. The checkpoint was tuned for 200 iterations with 128 calibration samples from NeelNanda/pile-10k. Embeddings and the vision tower were kept in higher precision.

On four full-sample text benchmarks, this deployment scored 0.78936 as the unweighted mean of PIQA, HellaSwag, MMLU, and GSM8K strict-match. Its BF16 reference scored 0.79055 on the same tasks. Read this first: the BF16 run used BF16 KV cache while this checkpoint used calibrated FP8 KV cache, so the score differences compare two end-to-end inference configurations and do not isolate weight quantization. No equivalence or statistical-significance claim is made.

The validated evaluation path used vLLM 0.29.0 and FlashInfer 0.7.0 on 8 NVIDIA RTX 5090 GPUs (SM120). It ran in text-only mode with a 131,072-token maximum context. The retained vision weights and multimodal path were not evaluated.

1. Model summary

Property	Base model	This checkpoint
Name and architecture	google/gemma-4-31B, Gemma4ForConditionalGeneration; dense, 60 text layers (50 sliding-attention and 10 full-attention)	Same architecture; 31B family
Weight format	BF16 safetensors	Compressed Tensors, mxfp8-quantized; MXFP8 weights with serialized quantization metadata
Safetensors	2 shards; 62,546,338,248 bytes (58.251 GiB)	7 shards; 34,174,889,744 bytes (31.828 GiB)
Storage ratio	1.00×	1.83× smaller by summed safetensors file bytes
Header census	1,188 tensor keys; BF16	1,772 tensor keys: 410 F8_E4M3, 410 U8, 778 BF16, and 174 F32
KV cache scales	Not present	87 K-scale and 87 V-scale entries; static FP8 scheme
Context	Configuration supports up to 262,144 tokens	131,072 tokens used in the validated evaluation
License	Apache 2.0	Same license as the base model

File sizes are the sums of the actual safetensors shard bytes, not du allocation estimates. Index-to-header verification found all 1,772 checkpoint keys in their indexed shards. The U8 tensors store MXFP8 block scales; the remaining BF16/F32 tensors are not represented as MXFP8 weight tensors.

The model configuration specifies 8-bit floating-point weights with symmetric groupwise scales (group size 32), dynamic 8-bit floating-point input activations, and a static tensorwise FP8 KV-cache scheme.

2. Precision plan

Module family	Checkpoint precision	Evidence and rationale
Text-backbone Linear weights	MXFP8 (F8_E4M3) with U8 group scales	410 quantized weight tensors and matching scale tensors in the safetensors headers
Token embeddings	BF16	embed_tokens is in the exported ignore list
Vision tower and vision projection	BF16	vision_tower and embed_vision are in the exported ignore list; the vision path was not evaluated
Other retained weights and parameters	BF16 or F32	Preserved at their exported higher precision; tied embeddings mean there is no standalone lm_head weight key
KV cache	Static FP8 at inference	Exported kv_cache_scheme plus 87 K and 87 V scale entries

2.1 Quantization plan and artifact

The run configuration and exported quantization_config.json agree on the explicit ignore families embed_tokens, vision_tower, and embed_vision. All index keys were found in the corresponding safetensors headers. The published precision census is based on those headers; it does not infer an unmeasured per-layer count from the file size.

2.2 Size accounting

Checkpoint	Safetensors bytes	GiB
BF16 base	62,546,338,248	58.251
MXFP8 + FP8 KV	34,174,889,744	31.828

The measured ratio is 1.830184× based on total shard bytes. KV-cache scales are included in the quantized checkpoint; runtime KV allocation is separate.

3. Evaluation results

Evaluation used lm-eval 0.4.13 with its vLLM backend, TP=8, FlashInfer, seed 42, text-only mode, and no sample limit. “±” is the lm-eval-reported standard error.

Task	Metric	Samples	MXFP8 + FP8 KV	BF16 + BF16 KV	Δ (quantized − BF16)
PIQA	acc,none	1,838	0.8269858542 ± 0.0088254177	0.8280739935 ± 0.0088034070	-0.108814 pp
HellaSwag	acc,none	10,042	0.6556462856 ± 0.0047418598	0.6596295559 ± 0.0047286535	-0.398327 pp
MMLU (57 subjects)	acc,none	14,042	0.8112804444 ± 0.0030857345	0.8117077339 ± 0.0030874353	-0.042729 pp
GSM8K	exact_match,strict-match	1,319	0.8635329795 ± 0.0094557420	0.8627748294 ± 0.0094778082	+0.075815 pp
Unweighted four-task mean	Descriptive mean	—	0.7893613909	0.7905465282	-0.118514 pp

The mean is the arithmetic mean of the four displayed task scores, not a sample-weighted aggregate. Deltas are in percentage points. These are separate evaluation runs, not a paired significance test.

3.1 What was held fixed

Both runs used the same four tasks, metrics, sample counts, seed, vLLM backend, FlashInfer attention backend, TP=8, 131,072 maximum context, max_num_seqs=1, 0.85 GPU-memory utilization, and text-only evaluation. GSM8K used 5-shot prompting and up to 2,048 generated tokens; chat-template and few-shot-as-multiturn modes were disabled because the local tokenizer had no chat template.

The checkpoint evaluation used kv_cache_dtype=fp8; the BF16 reference used kv_cache_dtype=auto, which resolved to BF16. All other recorded task and serving settings were held constant.

3.2 Functional evidence and limitations

The checkpoint loaded and completed all four full-sample evaluations on RTX 5090. PIQA, HellaSwag, and MMLU used log-likelihood scoring; GSM8K used generation and strict-match scoring. No gate was configured. The reported results establish behavior only for these tasks and this text-only protocol; they do not establish multimodal quality, long-context quality beyond 131,072 tokens, throughput, latency, or agentic performance.

4. Usage

4.1 vLLM

The evaluated vLLM load used the checkpoint's embedded Compressed Tensors configuration and calibrated FP8 KV scales:

vllm serve INCModel4/gemma-4-31B-MXFP8-FP8KV-CT-AutoRound \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --max-model-len 131072 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.85 \
  --attention-backend FLASHINFER \
  --language-model-only

The validated hardware was 8× RTX 5090. Use an accelerator/kernel combination that supports Gemma 4's 512-dimension full-attention head with FP8 KV. An earlier A100-SXM4-80GB (SM80) run with vLLM 0.30.0 failed before processing samples in the FP8-KV kernel path; do not infer A100 support from this checkpoint.

This command reflects the successfully evaluated settings. An external HTTP serving endpoint, multimodal requests, and performance/throughput were not separately benchmarked.

4.2 Transformers

The configuration can be inspected with Transformers, but loading this Compressed Tensors checkpoint for inference through Transformers was not validated. Use the vLLM path above for the measured inference results.

5. Reproducibility

5.1 Hardware and software

Phase	Measured environment
Quantization	2× NVIDIA A100-SXM4-80GB; quantization completed and exported the checkpoint
Evaluation	8× NVIDIA RTX 5090 (SM120), TP=8; Linux kernel 6.8; driver 610.57.04; CUDA runtime 13.0

CPU and system RAM inventory for the quantization host were not preserved in the run records and are not asserted here.

5.2 Package versions

Phase	Packages
Quantization	AutoRound 0.16.0.dev194+g20710256; PyTorch 2.13.0+cu130; Transformers 5.17.0; vLLM 0.30.0
Evaluation	lm-eval 0.4.13; vLLM 0.29.0; FlashInfer 0.7.0; PyTorch 2.13.0+cu130; Transformers 5.17.0

The original quantization setup also recorded lm-eval 0.4.12; the scored validation runs used lm-eval 0.4.13.

5.3 Environment variables

The validated runs used the following device visibility. No access token is required in the model card or command line:

# Quantization host
export CUDA_VISIBLE_DEVICES=0,1

# Evaluation host
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

The original quantization environment activated its Conda environment before setting LD_PRELOAD to that environment's lib/libstdc++.so.6. Follow the same ABI setup only if required by the installed environment.

6. Reproduce the artifact

6.1 Quantize

The measured quantization used the official base model, 200 tuning iterations, 128 samples from NeelNanda/pile-10k, and Compressed Tensors export:

MODEL_ID=google/gemma-4-31B
OUTPUT_DIR=./quantized

auto-round \
  --model_name "$MODEL_ID" \
  --scheme MXFP8 \
  --group_size 32 \
  --algorithm auto_round \
  --iters 200 \
  --nsamples 128 \
  --dataset NeelNanda/pile-10k \
  --static_kv_dtype fp8 \
  --ignore_layers embed_tokens,vision_tower,embed_vision \
  --device_map auto \
  --format llm_compressor \
  --output_dir "$OUTPUT_DIR"

The measured quantizer version is recorded in §5.2. This command reproduces the configured recipe; byte-for-byte output can vary with package builds and hardware.

6.2 Evaluate

Use the lm-eval vLLM backend with the settings from the full-sample evaluation:

MODEL_ID=INCModel4/gemma-4-31B-MXFP8-FP8KV-CT-AutoRound
COMMON_ARGS="pretrained=${MODEL_ID},tensor_parallel_size=8,max_model_len=131072,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,attention_backend=FLASHINFER,language_model_only=True,kv_cache_dtype=fp8,max_num_seqs=1"

lm_eval --model vllm --model_args "$COMMON_ARGS" \
  --tasks piqa,hellaswag,mmlu \
  --batch_size auto \
  --seed 42

lm_eval --model vllm --model_args "$COMMON_ARGS" \
  --tasks gsm8k \
  --batch_size 64 \
  --seed 42 \
  --num_fewshot 5

The GSM8K command uses strict-match as the headline metric. The BF16 reference is available at google/gemma-4-31B; when reproducing the comparison, set kv_cache_dtype=auto for that BF16 model and keep the remaining evaluation settings unchanged.

7. Known issues and caveats

• FP8 KV support is hardware- and kernel-dependent for the 512-dimension full-attention layers. The tested route is RTX 5090 with vLLM 0.29.0 and FlashInfer 0.7.0; an earlier A100 SM80 attempt failed before scoring.
• The configuration declares a 262,144-token context, but this checkpoint was evaluated only with max_model_len=131072.
• The vision tower and projection remain in the checkpoint at higher precision; no image or other multimodal request was evaluated.
• Score differences compare MXFP8 weights plus FP8 KV against BF16 weights plus BF16 KV. They are not a weight-only quantization delta.
• No throughput, latency, long-context, coding, agentic, or safety evaluation is claimed.

8. License and attribution

The base model is Gemma 4 31B by Google DeepMind and is distributed under Apache 2.0 according to the Gemma 4 license. The same license applies to this derivative checkpoint; users should review the linked terms before use or redistribution.

Quantization used Intel AutoRound and the checkpoint uses the Compressed Tensors format. Evaluation used vLLM, FlashInfer, and lm-evaluation-harness. The measured benchmark results and sample counts are listed in §3.