gemma-4-E4B-it · FP8 (dynamic)

8-bit quantization of google/gemma-4-E4B-it, produced for efficient vLLM serving. Quantized by axteam.

Base model google/gemma-4-E4B-it
Method FP8 (dynamic)
Format compressed-tensors float-quantized (scheme FP8_DYNAMIC)
Weights / activations W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations
Architecture Gemma 4 · Dense
Size on disk ~12.0 GB
Calibration None (data-free static weight scales)
License Gemma 4 / Apache-2.0

Quantization details

  • Scheme — W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
  • Kept in BF16 — vision tower, audio tower, vision/audio projections, token embeddings (tied with lm_head), the per-layer embedding table (embed_tokens_per_layer), the per-layer input gates and projections (per_layer_input_gate, per_layer_projection, per_layer_model_projection), and all norms. These stay unquantized to preserve quality — about 51% of all parameters in this build.
  • Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.

  • RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used on purpose for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which apply AWQ; their first uploads (2026-10-11, replaced 2026-11-01) were effectively RTN by accident (AWQ smoothing silently did not apply) and have since been rebuilt.
  • AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
  • QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: FP8 dynamic — a calibration-free PTQ (static per-channel weight scales + runtime per-token activation scales).

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark BF16 (original) FP8-dynamic (this repo) AWQ-8bit NVFP4A16 NVFP4 AWQ-4bit
MMLU (5-shot, generative) 62.16 63.20 (101.7%) 62.40 (100.4%) 64.48 (103.7%) 62.14 (100.0%) 64.16 (103.2%)
GSM8K (5-shot, strict) 84.31 84.00 (99.6%) 84.08 (99.7%) 82.71 (98.1%) 79.83 (94.7%) 81.73 (96.9%)
ARC-Challenge (0-shot) 56.91 57.34 (100.7%) 56.66 (99.6%) 56.66 (99.6%) 55.20 (97.0%) 55.97 (98.4%)
Winogrande (5-shot) 66.30 65.75 (99.2%) 66.85 (100.8%) 65.51 (98.8%) 66.46 (100.2%) 65.43 (98.7%)
KMMLU (5-shot, direct) 51.46 51.24 (99.6%) 51.46 (100.0%) 50.15 (97.5%) 49.11 (95.4%) 51.31 (99.7%)
HAE-RAE (0-shot) 69.84 69.02 (98.8%) 69.57 (99.6%) 69.48 (99.5%) 68.10 (97.5%) 68.10 (97.5%)
Average 65.16 65.09 (99.9%) 65.17 (100.0%) 64.83 (99.5%) 63.47 (97.4%) 64.45 (98.9%)

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: FP8 dynamic (W8A8).

Metric This model BF16 original
Perplexity (lower is better) 1.3173 1.3032 (+1.1%)
Top-1 next-token agreement 96.0% 99.2%

These numbers measure fidelity to the base model, not absolute task scores.

Requirements

Runtime vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0 (text: the benchmarks above; image: an image-description check on 2026-11-02).
vLLM install vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory Weights: 12.0 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM) transformers>=5.5.2 (5.5.0/5.5.1 cannot run the KV-shared layers).
trust_remote_code not needed

Usage (vLLM)

vllm serve axteam/gemma-4-E4B-it-FP8-dynamic
from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-E4B-it-FP8-dynamic")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Thinking mode: add the <|think|> token at the start of the system prompt to enable step-by-step reasoning.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-E4B-it-FP8-dynamic --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.


한국어 요약

google/gemma-4-E4B-it의 FP8 (W8A8) 양자화 빌드 · 제작 axteam

개요

항목 내용
원본 모델 google/gemma-4-E4B-it
양자화 방식 FP8 동적 양자화 (W8A8)
저장 형식 compressed-tensors float-quantized (scheme FP8_DYNAMIC)
모델 구조 Gemma 4 · Dense
용량 약 12.0 GB
보정 데이터 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산)

양자화 상세

  • 가중치: FP8(E4M3), 출력 채널별 스케일
  • 활성화: 실행 시 토큰별 동적 FP8
  • 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
  • BF16 유지: 비전 타워, 오디오 타워, 비전·오디오 투영층, 토큰 임베딩(LM 헤드와 공유), 층별 임베딩 표(embed_tokens_per_layer), 층별 입력 게이트·투영층(per_layer_input_gate·per_layer_projection·per_layer_model_projection), 모든 정규화층 — 전체 파라미터의 약 51%
  • 참고: 같은 모델의 AWQ-8bit·AWQ-4bit·NVFP4A16·NVFP4 리포 첫 업로드(2026-10-11)는 AWQ 스무딩이 적용되지 않아 사실상 단순 반올림(RTN)이었고 2026-11-01 재양자화됨 (이 FP8 빌드와는 무관)
  • 원본 파일: 토크나이저·chat template(Google의 2026년 7월 수정본)·processor·generation config를 바이트 단위로 그대로 사용, config.json은 원본에 quantization_config 블록만 추가

품질 (실측)

  • 원본 대비 충실도: PPL 1.3173 (원본 1.3032, +1.1%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 96.0% (원본 자체 기준값 99.2%) — 내부 영어·한국어 혼합 평가셋, 원본(BF16)과 비교 측정
  • 벤치마크 평균: 65.09 (원본 65.16, 99.9%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
  • 한국어 벤치마크: KMMLU 51.24 (원본 51.46, 99.6%) · HAE-RAE 69.02 (원본 69.84, 98.8%)
  • MMLU 방식: lm-eval 생성형 변형(mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가

요구사항

  • 실행 엔진: vLLM 0.19.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인 (텍스트 — 위 벤치마크, 이미지 — 2026-11-02 그림 설명 점검)
  • vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
  • GPU: NVIDIA Turing(sm75, 예: T4) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
  • 메모리: 가중치 12.0 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
  • Transformers로 직접 사용: transformers 5.5.2 이상 (5.5.0·5.5.1은 KV 공유 층 실행 불가)
  • trust_remote_code: 불필요

사용법

  • 서버 실행: vllm serve axteam/gemma-4-E4B-it-FP8-dynamic
  • Python: LLM(model="axteam/gemma-4-E4B-it-FP8-dynamic").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
  • 권장 샘플링: temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)
  • thinking 모드: 기본 꺼짐 — 켜려면 시스템 프롬프트 맨 앞에 <|think|> 토큰 추가
  • KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

  • 라이선스: Apache-2.0 (Gemma 4 라이선스 안내) — 원본 모델의 라이선스를 그대로 따름
  • 권리: 원본 가중치·토크나이저·chat template은 Google DeepMind 소유, 이 리포는 양자화 파생물
Downloads last month
371
Safetensors
Model size
8B params
Tensor type
BF16
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for axteam/gemma-4-E4B-it-FP8-dynamic

Quantized
(462)
this model

Collection including axteam/gemma-4-E4B-it-FP8-dynamic