gemma-4-31B · AWQ INT4 (W4A16)

4-bit quantization of google/gemma-4-31B (commit 5bbc2fb), produced for efficient vLLM serving. Quantized by axteam.

Base model google/gemma-4-31B (commit 5bbc2fb)
Method AWQ INT4 (W4A16)
Format compressed-tensors pack-quantized
Weights / activations W4A16 — INT4 asymmetric weights, group size 32 (BF16 scale + INT4 zero-point), BF16 activations
Architecture Gemma 4 · Dense
Size on disk ~20.9 GB
Calibration 256 samples (Korean-heavy; -it includes 32 self-generated thinking-mode answers)
License Gemma 4 / Apache-2.0

Quantization details

  • Scheme — W4A16 — INT4 asymmetric weights, group size 32 (BF16 scale + INT4 zero-point), BF16 activations, stored as compressed-tensors pack-quantized.
  • Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), and all norms. These stay unquantized to preserve quality — about 6% of all parameters in this build.
  • Tokenizer / processor_config / generation_config are byte-identical to the original. This is a base (pre-trained) model with no chat template — use plain text completion, not chat. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.

  • RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which all apply AWQ (see each repo's recipe.yaml).
  • AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
  • QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark BF16 (original) FP8-dynamic AWQ-8bit NVFP4A16 NVFP4 AWQ-4bit (this repo)
MMLU (5-shot, generative) 84.59 84.71 (100.1%) 84.52 (99.9%) 83.70 (98.9%) 83.09 (98.2%) 84.02 (99.3%)
GSM8K (5-shot, strict) 84.46 83.09 (98.4%) 84.23 (99.7%) 83.78 (99.2%) 82.03 (97.1%) 84.15 (99.6%)
ARC-Challenge (0-shot) 69.62 68.94 (99.0%) 69.11 (99.3%) 68.43 (98.3%) 67.49 (96.9%) 69.11 (99.3%)
Winogrande (5-shot) 83.90 84.29 (100.5%) 84.29 (100.5%) 84.45 (100.7%) 82.64 (98.5%) 83.58 (99.6%)
KMMLU (5-shot, direct) 67.35 67.22 (99.8%) 67.39 (100.1%) 66.41 (98.6%) 65.22 (96.8%) 67.33 (100.0%)
HAE-RAE (0-shot) 82.31 82.03 (99.7%) 82.03 (99.7%) 81.94 (99.6%) 80.29 (97.6%) 81.85 (99.4%)
Average 78.71 78.38 (99.6%) 78.60 (99.9%) 78.12 (99.3%) 76.79 (97.6%) 78.34 (99.5%)

Evaluation (measured)

Perplexity was measured against the BF16 original on public text — the WikiText-2 test set (English) and a Korean Wikipedia sample, 2,048-token chunks. 8-bit builds (AWQ INT8, FP8) stayed within 0.5% of BF16; 4-bit builds (AWQ INT4, NVFP4) were 2.0–6.6% higher (range across both base models and both 4-bit formats). Next-token top-1 agreement was not measured for the base models.

Requirements

Runtime vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0 (text input only — the benchmarks above, run with image input disabled; image input not run for this build).
vLLM install vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory Weights: 20.9 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM) transformers>=5.5.0 (5.5.2+ recommended; not run here for 31B).
trust_remote_code not needed

Usage (vLLM)

vllm serve axteam/gemma-4-31B-AWQ-4bit
from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-31B-AWQ-4bit")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.generate("The three primary colors are", params)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-31B-AWQ-4bit --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights and tokenizer belong to Google DeepMind.


한국어 요약

google/gemma-4-31B의 AWQ INT4 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목 내용
원본 모델 google/gemma-4-31B (커밋 5bbc2fb)
양자화 방식 AWQ INT4 (W4A16)
저장 형식 compressed-tensors pack-quantized
모델 구조 Gemma 4 · Dense
용량 약 20.9 GB
보정 데이터 샘플 256개 (한국어 위주)

양자화 상세

  • 가중치: INT4 비대칭, 그룹 크기 32 (BF16 스케일 + INT4 영점)
  • 활성화: BF16 (양자화하지 않음)
  • 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), AWQ 방식 — 보정 데이터로 활성화가 큰 가중치 채널을 찾아 채널별 스케일로 보호한 뒤 INT4로 반올림
  • BF16 유지: 비전 타워, 비전 투영층, 토큰 임베딩(LM 헤드와 공유), 모든 정규화층 — 전체 파라미터의 약 6%
  • 원본 파일: 토크나이저·processor·generation config를 바이트 단위로 그대로 사용(사전학습 모델이라 chat template은 없음), config.json은 원본에 quantization_config 블록만 추가

품질 (실측)

  • 공개 텍스트 PPL: WikiText-2(영어)·한국어 위키 표본, 2,048토큰 단위 — 8비트(AWQ INT8·FP8)는 원본 대비 0.5% 이내, 4비트(AWQ INT4·NVFP4)는 2.0~6.6% 상승 (두 base 모델·두 4비트 형식을 합친 범위, top-1은 미측정)
  • 벤치마크 평균: 78.34 (원본 78.71, 99.5%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
  • 한국어 벤치마크: KMMLU 67.33 (원본 67.35, 100.0%) · HAE-RAE 81.85 (원본 82.31, 99.4%)
  • MMLU 방식: lm-eval 생성형 변형(mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가

요구사항

  • 실행 엔진: vLLM 0.19.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인 (텍스트 입력만 — 위 벤치마크, 이미지 입력은 끄고 측정, 이 빌드의 이미지 입력은 실행 미확인)
  • vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
  • GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 — 가중치 전용 정수 형식이라 FP8·FP4 전용 텐서코어 불필요 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
  • 메모리: 가중치 20.9 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
  • Transformers로 직접 사용: transformers 5.5.0 이상 (5.5.2 이상 권장) — 이 모델은 Transformers로 직접 실행해 보지 않음
  • trust_remote_code: 불필요

사용법

  • 서버 실행: vllm serve axteam/gemma-4-31B-AWQ-4bit
  • Python: LLM(model="axteam/gemma-4-31B-AWQ-4bit").generate(...) — 사전학습 모델이라 대화가 아닌 텍스트 이어쓰기
  • 권장 샘플링: temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)
  • KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

  • 라이선스: Apache-2.0 (Gemma 4 라이선스 안내) — 원본 모델의 라이선스를 그대로 따름
  • 권리: 원본 가중치·토크나이저는 Google DeepMind 소유, 이 리포는 양자화 파생물
Downloads last month
248
Safetensors
Model size
7B params
Tensor type
I32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for axteam/gemma-4-31B-AWQ-4bit

Quantized
(46)
this model

Collection including axteam/gemma-4-31B-AWQ-4bit