gemma-4-26B-A4B Β· AWQ INT8 (W8A16)

8-bit quantization of google/gemma-4-26B-A4B (commit 24548b6), produced for efficient vLLM serving. Quantized by axteam.

Base model google/gemma-4-26B-A4B (commit 24548b6)
Method AWQ INT8 (W8A16)
Format compressed-tensors pack-quantized
Weights / activations W8A16 β€” INT8 symmetric weights, group size 32, BF16 activations
Architecture Gemma 4 Β· Mixture-of-Experts (8 active / 128 total + 1 shared)
Size on disk ~29.2 GB
Calibration 256 samples (Korean-heavy; -it includes 32 self-generated thinking-mode answers)
License Gemma 4 / Apache-2.0

Quantization details

  • Scheme β€” W8A16 β€” INT8 symmetric weights, group size 32, BF16 activations, stored as compressed-tensors pack-quantized.
  • Kept in BF16 β€” vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms, plus the shared MLP. These stay unquantized to preserve quality β€” about 7% of all parameters in this build.
  • Tokenizer / processor_config / generation_config are byte-identical to the original. This is a base (pre-trained) model with no chat template β€” use plain text completion, not chat. config.json is the original plus a quantization_config block.

Quantization method β€” PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.

  • RTN (round-to-nearest) β€” weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which all apply AWQ (see each repo's recipe.yaml).
  • AWQ (activation-aware weight quantization) β€” a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm β†’ q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
  • QAT (quantization-aware training) β€” the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT β‰₯ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ β€” a calibration-based PTQ (256-sample calibration, no gradient).

Benchmarks (measured)

Every column was measured with the same harness and settings β€” lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark BF16 (original) FP8-dynamic AWQ-8bit (this repo) NVFP4A16 NVFP4 AWQ-4bit
MMLU (5-shot, generative) 77.77 77.25 (99.3%) 77.75 (100.0%) 77.18 (99.2%) 76.20 (98.0%) 76.82 (98.8%)
GSM8K (5-shot, strict) 70.89 71.34 (100.6%) 71.49 (100.9%) 69.98 (98.7%) 67.17 (94.8%) 69.98 (98.7%)
ARC-Challenge (0-shot) 68.34 68.60 (100.4%) 68.17 (99.8%) 68.52 (100.2%) 66.98 (98.0%) 68.69 (100.5%)
Winogrande (5-shot) 80.58 80.03 (99.3%) 80.58 (100.0%) 79.64 (98.8%) 78.22 (97.1%) 79.56 (98.7%)
KMMLU (5-shot, direct) 60.03 60.07 (100.1%) 60.15 (100.2%) 59.41 (99.0%) 58.14 (96.9%) 58.99 (98.3%)
HAE-RAE (0-shot) 77.27 76.99 (99.6%) 76.90 (99.5%) 77.18 (99.9%) 75.44 (97.6%) 75.80 (98.1%)
Average 72.48 72.38 (99.9%) 72.51 (100.0%) 71.98 (99.3%) 70.36 (97.1%) 71.64 (98.8%)

Evaluation (measured)

Perplexity was measured against the BF16 original on public text β€” the WikiText-2 test set (English) and a Korean Wikipedia sample, 2,048-token chunks. 8-bit builds (AWQ INT8, FP8) stayed within 0.5% of BF16; 4-bit builds (AWQ INT4, NVFP4) were 2.0–6.6% higher (range across both base models and both 4-bit formats). Next-token top-1 agreement was not measured for the base models.

Requirements

Runtime vLLM β€” the architecture is in vLLM β‰₯ 0.19.0; tested here with vLLM 0.32.0 (text input only β€” the benchmarks above, run with image input disabled; image input not run for this build).
vLLM install vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers β‰₯5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 β€” do not upgrade it separately).
GPU NVIDIA Turing (sm75) or newer β€” weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory Weights: 29.2 GB. Add KV cache (grows with context length Γ— concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM) Not loadable by Transformers (checked up to 5.22.1: no conversion rule for per-expert checkpoints, the experts stay uninitialised) β€” use vLLM.
trust_remote_code not needed

Usage (vLLM)

vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit
from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.generate("The three primary colors are", params)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here β€” the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights and tokenizer belong to Google DeepMind.


ν•œκ΅­μ–΄ μš”μ•½

google/gemma-4-26B-A4B의 AWQ INT8 (W8A16) μ–‘μžν™” λΉŒλ“œ Β· μ œμž‘ axteam

κ°œμš”

ν•­λͺ© λ‚΄μš©
원본 λͺ¨λΈ google/gemma-4-26B-A4B (컀밋 24548b6)
μ–‘μžν™” 방식 AWQ INT8 (W8A16)
μ €μž₯ ν˜•μ‹ compressed-tensors pack-quantized
λͺ¨λΈ ꡬ쑰 Gemma 4 Β· MoE (μ „λ¬Έκ°€ 128개 쀑 8개 ν™œμ„± + 항상 μ“°λŠ” 곡유 MLP)
μš©λŸ‰ μ•½ 29.2 GB
보정 데이터 μƒ˜ν”Œ 256개 (ν•œκ΅­μ–΄ μœ„μ£Ό)

μ–‘μžν™” 상세

  • κ°€μ€‘μΉ˜: INT8 λŒ€μΉ­, κ·Έλ£Ή 크기 32
  • ν™œμ„±ν™”: BF16 (μ–‘μžν™”ν•˜μ§€ μ•ŠμŒ)
  • μ–‘μžν™” 절차: PTQ(ν•™μŠ΅ ν›„ μ–‘μžν™”, μž¬ν•™μŠ΅ μ—†μŒ), AWQ 방식 β€” 보정 λ°μ΄ν„°λ‘œ ν™œμ„±ν™”κ°€ 큰 κ°€μ€‘μΉ˜ 채널을 μ°Ύμ•„ 채널별 μŠ€μΌ€μΌλ‘œ λ³΄ν˜Έν•œ λ’€ INT8둜 반올림
  • BF16 μœ μ§€: λΉ„μ „ νƒ€μ›Œ, λΉ„μ „ 투영측, 토큰 μž„λ² λ”©(LM ν—€λ“œμ™€ 곡유), MoE λΌμš°ν„°, λͺ¨λ“  μ •κ·œν™”μΈ΅, 곡유 MLP β€” 전체 νŒŒλΌλ―Έν„°μ˜ μ•½ 7%
  • 원본 파일: ν† ν¬λ‚˜μ΄μ €Β·processorΒ·generation configλ₯Ό λ°”μ΄νŠΈ λ‹¨μœ„λ‘œ κ·ΈλŒ€λ‘œ μ‚¬μš©(μ‚¬μ „ν•™μŠ΅ λͺ¨λΈμ΄λΌ chat template은 μ—†μŒ), config.json은 원본에 quantization_config λΈ”λ‘λ§Œ μΆ”κ°€

ν’ˆμ§ˆ (μ‹€μΈ‘)

  • 곡개 ν…μŠ€νŠΈ PPL: WikiText-2(μ˜μ–΄)Β·ν•œκ΅­μ–΄ μœ„ν‚€ ν‘œλ³Έ, 2,048토큰 λ‹¨μœ„ β€” 8λΉ„νŠΈ(AWQ INT8Β·FP8)λŠ” 원본 λŒ€λΉ„ 0.5% 이내, 4λΉ„νŠΈ(AWQ INT4Β·NVFP4)λŠ” 2.0~6.6% μƒμŠΉ (두 base λͺ¨λΈΒ·λ‘ 4λΉ„νŠΈ ν˜•μ‹μ„ ν•©μΉœ λ²”μœ„, top-1은 λ―ΈμΈ‘μ •)
  • 벀치마크 평균: 72.51 (원본 72.48, 100.0%) β€” MMLU(μƒμ„±ν˜•)Β·GSM8KΒ·ARC-ChallengeΒ·WinograndeΒ·KMMLUΒ·HAE-RAE 6μ’… 평균, lm-evaluation-harness + vLLM, chat template μ—†λŠ” μ™„μ„±ν˜• ν”„λ‘¬ν”„νŠΈ(κ³Όμ œλ³„ 0Β·5-shot, ν‘œ μ°Έκ³ )Β·greedy, λͺ¨λ“  λΉŒλ“œλ₯Ό 같은 μ„€μ •μœΌλ‘œ μΈ‘μ •
  • ν•œκ΅­μ–΄ 벀치마크: KMMLU 60.15 (원본 60.03, 100.2%) Β· HAE-RAE 76.90 (원본 77.27, 99.5%)
  • MMLU 방식: lm-eval μƒμ„±ν˜• λ³€ν˜•(mmlu_generative) β€” λͺ¨λΈμ΄ λ‹΅ κΈ€μžλ₯Ό μƒμ„±ν•˜κ³  첫 쀄이 μ •ν™•νžˆ μΌμΉ˜ν•΄μ•Ό μ •λ‹΅μœΌλ‘œ μ±„μ ν•˜λ―€λ‘œ, λ‹€λ₯Έ 방식(예: log-likelihood 채점)으둜 잰 MMLU μ μˆ˜μ™€ 직접 비ꡐ λΆˆκ°€

μš”κ΅¬μ‚¬ν•­

  • μ‹€ν–‰ μ—”μ§„: vLLM 0.19.0 이상(이 ꡬ쑰λ₯Ό μ§€μ›ν•˜λŠ” 첫 버전), 이 λΉŒλ“œλŠ” vLLM 0.32.0μ—μ„œ μ‹€ν–‰ 확인 (ν…μŠ€νŠΈ μž…λ ₯만 β€” μœ„ 벀치마크, 이미지 μž…λ ₯은 끄고 μΈ‘μ •, 이 λΉŒλ“œμ˜ 이미지 μž…λ ₯은 μ‹€ν–‰ 미확인)
  • vLLM 0.32 μ„€μΉ˜ 쑰건: Python 3.11~3.14, torch 2.13(CUDA 13.0 λΉŒλ“œ, CUDA 13 지원 λ“œλΌμ΄λ²„ ν•„μš”), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 β€” transformersΒ·compressed-tensorsλŠ” vLLM이 버전을 κ³ μ •ν•˜λ―€λ‘œ compressed-tensorsλ₯Ό λ”°λ‘œ μ—…κ·Έλ ˆμ΄λ“œν•˜μ§€ 말 것 (이 μ²΄ν¬ν¬μΈνŠΈλŠ” 0.19.0으둜 μ €μž₯λμ§€λ§Œ 0.17.0μ—μ„œλ„ λ‘œλ“œλ¨)
  • GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 β€” κ°€μ€‘μΉ˜ μ „μš© μ •μˆ˜ ν˜•μ‹μ΄λΌ FP8Β·FP4 μ „μš© ν…μ„œμ½”μ–΄ λΆˆν•„μš” (GPU별 지원 μ—¬λΆ€λŠ” vLLM 0.32 컀널 선택 μ½”λ“œ κΈ°μ€€, μ‹€μ œ 싀행은 B200μ—μ„œλ§Œ 확인)
  • λ©”λͺ¨λ¦¬: κ°€μ€‘μΉ˜ 29.2 GB + KV μΊμ‹œ(μ»¨ν…μŠ€νŠΈ 길이 Γ— λ™μ‹œ μš”μ²­ μˆ˜μ— λΉ„λ‘€) + ν™œμ„±ν™”Β·CUDA graph용 수 GB β€” GPU ν•œ μž₯에 λ‹€ λ“€μ–΄κ°€μ§€ μ•ŠμœΌλ©΄ --tensor-parallel-size N으둜 λΆ„μ‚°
  • Transformers둜 직접 μ‚¬μš©: λΆˆκ°€ β€” 5.22.1κΉŒμ§€ λ‘œλ“œλ˜μ§€ μ•ŠμŒ 확인 (전문가별 체크포인트 λ³€ν™˜ κ·œμΉ™μ΄ μ—†μ–΄ μ „λ¬Έκ°€κ°€ μ΄ˆκΈ°ν™”λ˜μ§€ μ•ŠμŒ), vLLM μ‚¬μš©
  • trust_remote_code: λΆˆν•„μš”

μ‚¬μš©λ²•

  • μ„œλ²„ μ‹€ν–‰: vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit
  • Python: LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit").generate(...) β€” μ‚¬μ „ν•™μŠ΅ λͺ¨λΈμ΄λΌ λŒ€ν™”κ°€ μ•„λ‹Œ ν…μŠ€νŠΈ 이어쓰기
  • ꢌμž₯ μƒ˜ν”Œλ§: temperature=1.0, top_p=0.95, top_k=64 (Google ꢌμž₯κ°’)
  • KV μΊμ‹œ 절감(선택): vllm serve에 --kv-cache-dtype fp8 μΆ”κ°€ β€” κΈ°λ³Έ BF16인 KV μΊμ‹œλ₯Ό FP8둜 μ €μž₯ν•΄ KV μΊμ‹œ λ©”λͺ¨λ¦¬λ₯Ό μ•½ 절반으둜 μ€„μž„ (κΈ΄ μ»¨ν…μŠ€νŠΈΒ·λ™μ‹œ μš”μ²­μ΄ λ§Žμ„ λ•Œ 유리, κ°€μ€‘μΉ˜ μ–‘μžν™”μ™€ λ³„κ°œμΈ μ‹€ν–‰ μ˜΅μ…˜)

λΌμ΄μ„ μŠ€

  • λΌμ΄μ„ μŠ€: Apache-2.0 (Gemma 4 λΌμ΄μ„ μŠ€ μ•ˆλ‚΄) β€” 원본 λͺ¨λΈμ˜ λΌμ΄μ„ μŠ€λ₯Ό κ·ΈλŒ€λ‘œ 따름
  • ꢌ리: 원본 κ°€μ€‘μΉ˜Β·ν† ν¬λ‚˜μ΄μ €λŠ” Google DeepMind μ†Œμœ , 이 λ¦¬ν¬λŠ” μ–‘μžν™” νŒŒμƒλ¬Ό
Downloads last month
191
Safetensors
Model size
9B params
Tensor type
I32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for axteam/gemma-4-26B-A4B-AWQ-8bit

Quantized
(38)
this model

Collection including axteam/gemma-4-26B-A4B-AWQ-8bit