Instructions to use axteam/gemma-4-26B-A4B-it-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use axteam/gemma-4-26B-A4B-it-FP8-dynamic with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="axteam/gemma-4-26B-A4B-it-FP8-dynamic") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("axteam/gemma-4-26B-A4B-it-FP8-dynamic") model = AutoModelForMultimodalLM.from_pretrained("axteam/gemma-4-26B-A4B-it-FP8-dynamic", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use axteam/gemma-4-26B-A4B-it-FP8-dynamic with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "axteam/gemma-4-26B-A4B-it-FP8-dynamic" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/axteam/gemma-4-26B-A4B-it-FP8-dynamic
- SGLang
How to use axteam/gemma-4-26B-A4B-it-FP8-dynamic with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "axteam/gemma-4-26B-A4B-it-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "axteam/gemma-4-26B-A4B-it-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use axteam/gemma-4-26B-A4B-it-FP8-dynamic with Docker Model Runner:
docker model run hf.co/axteam/gemma-4-26B-A4B-it-FP8-dynamic
gemma-4-26B-A4B-it · FP8 (dynamic)
8-bit quantization of google/gemma-4-26B-A4B-it (commit 4d7ae49), produced for efficient vLLM serving. Quantized by axteam.
| Base model | google/gemma-4-26B-A4B-it (commit 4d7ae49) |
| Method | FP8 (dynamic) |
| Format | compressed-tensors float-quantized (scheme FP8_DYNAMIC) |
| Weights / activations | W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations |
| Architecture | Gemma 4 · Mixture-of-Experts (8 active / 128 total + 1 shared) |
| Size on disk | ~27.2 GB |
| Calibration | None (data-free static weight scales) |
| License | Gemma 4 / Apache-2.0 |
Quantization details
- Scheme — W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations, stored as compressed-tensors
float-quantized(schemeFP8_DYNAMIC). - Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These stay unquantized to preserve quality — about 5% of all parameters in this build.
- Tokenizer /
chat_template/processor_config/generation_configare byte-identical to the original, including Google's July-2026 correctedchat_template.config.jsonis the original plus aquantization_configblock.
Quantization method — PTQ (AWQ / RTN) vs QAT
This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
- RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which all apply AWQ (see each repo's
recipe.yaml). - AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
- QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: FP8 dynamic — a calibration-free PTQ (static per-channel weight scales + runtime per-token activation scales).
Benchmarks (measured)
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic (this repo) | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot, generative) | 40.23 | 35.77 (88.9%) | 38.72 (96.2%) | 30.44 (75.7%) | 31.39 (78.0%) | 33.21 (82.5%) |
| GSM8K (5-shot, strict) | 66.19 | 67.70 (102.3%) | 67.10 (101.4%) | 50.42 (76.2%) | 60.88 (92.0%) | 55.12 (83.3%) |
| ARC-Challenge (0-shot) | 45.39 | 45.82 (100.9%) | 44.88 (98.9%) | 44.62 (98.3%) | 44.28 (97.6%) | 44.62 (98.3%) |
| Winogrande (5-shot) | 58.48 | 60.22 (103.0%) | 59.91 (102.4%) | 58.64 (100.3%) | 57.77 (98.8%) | 58.48 (100.0%) |
| KMMLU (5-shot, direct) | 36.52 | 34.81 (95.3%) | 35.84 (98.1%) | 33.42 (91.5%) | 34.37 (94.1%) | 33.91 (92.9%) |
| HAE-RAE (0-shot) | 76.17 | 75.62 (99.3%) | 75.99 (99.8%) | 73.60 (96.6%) | 72.87 (95.7%) | 70.76 (92.9%) |
| Average | 53.83 | 53.32 (99.1%) | 53.74 (99.8%) | 48.53 (90.1%) | 50.26 (93.4%) | 49.35 (91.7%) |
Evaluation (measured)
Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: FP8 dynamic (W8A8).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower is better) | 1.1531 | 1.1284 (+2.2%) |
| Top-1 next-token agreement | 96.3% | 98.7% |
These numbers measure fidelity to the base model, not absolute task scores.
Requirements
| Runtime | vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0 (text: the benchmarks above; image: an image-description check on 2026-11-02). |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 27.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | Not loadable by Transformers (checked up to 5.22.1: no conversion rule for per-expert checkpoints, the experts stay uninitialised) — use vLLM. |
| trust_remote_code | not needed |
Usage (vLLM)
vllm serve axteam/gemma-4-26B-A4B-it-FP8-dynamic
from vllm import LLM, SamplingParams
llm = LLM(model="axteam/gemma-4-26B-A4B-it-FP8-dynamic")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
[{"role": "user", "content": "Write a short joke about saving RAM."}],
params,
)
print(out[0].outputs[0].text)
Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64. Thinking mode is off by default; turn it on withchat_template_kwargs={"enable_thinking": True}(Pythonllm.chat(..., chat_template_kwargs=...)or the request body), and add--reasoning-parser gemma4tovllm serveto return the reasoning separately.
Reducing KV-cache memory (optional, serve-time)
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
vllm serve axteam/gemma-4-26B-A4B-it-FP8-dynamic --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
License
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.
한국어 요약
google/gemma-4-26B-A4B-it의 FP8 (W8A8) 양자화 빌드 · 제작 axteam
개요
| 항목 | 내용 |
|---|---|
| 원본 모델 | google/gemma-4-26B-A4B-it (커밋 4d7ae49) |
| 양자화 방식 | FP8 동적 양자화 (W8A8) |
| 저장 형식 | compressed-tensors float-quantized (scheme FP8_DYNAMIC) |
| 모델 구조 | Gemma 4 · MoE (전문가 128개 중 8개 활성 + 항상 쓰는 공유 MLP) |
| 용량 | 약 27.2 GB |
| 보정 데이터 | 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산) |
양자화 상세
- 가중치: FP8(E4M3), 출력 채널별 스케일
- 활성화: 실행 시 토큰별 동적 FP8
- 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
- BF16 유지: 비전 타워, 비전 투영층, 토큰 임베딩(LM 헤드와 공유), MoE 라우터, 모든 정규화층 — 전체 파라미터의 약 5%
- 원본 파일: 토크나이저·chat template(Google의 2026년 7월 수정본)·processor·generation config를 바이트 단위로 그대로 사용,
config.json은 원본에quantization_config블록만 추가
품질 (실측)
- 원본 대비 충실도: PPL 1.1531 (원본 1.1284, +2.2%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 96.3% (원본 자체 기준값 98.7%) — 내부 영어·한국어 혼합 평가셋, 원본(BF16)과 비교 측정
- 벤치마크 평균: 53.32 (원본 53.83, 99.1%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
- 한국어 벤치마크: KMMLU 34.81 (원본 36.52, 95.3%) · HAE-RAE 75.62 (원본 76.17, 99.3%)
- MMLU 방식: lm-eval 생성형 변형(
mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가
요구사항
- 실행 엔진: vLLM 0.19.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인 (텍스트 — 위 벤치마크, 이미지 — 2026-11-02 그림 설명 점검)
- vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
- GPU: NVIDIA Turing(sm75, 예: T4) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
- 메모리: 가중치 27.2 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면
--tensor-parallel-size N으로 분산 - Transformers로 직접 사용: 불가 — 5.22.1까지 로드되지 않음 확인 (전문가별 체크포인트 변환 규칙이 없어 전문가가 초기화되지 않음), vLLM 사용
- trust_remote_code: 불필요
사용법
- 서버 실행:
vllm serve axteam/gemma-4-26B-A4B-it-FP8-dynamic - Python:
LLM(model="axteam/gemma-4-26B-A4B-it-FP8-dynamic").chat(...)— 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고 - 권장 샘플링:
temperature=1.0,top_p=0.95,top_k=64(Google 권장값) - thinking 모드: 기본 꺼짐 — 켜려면
chat_template_kwargs={"enable_thinking": True}전달(Python은llm.chat()인자, HTTP는 요청 본문의"chat_template_kwargs": {"enable_thinking": true}), 추론 과정을 답변과 분리해 받으려면vllm serve에--reasoning-parser gemma4추가 - KV 캐시 절감(선택):
vllm serve에--kv-cache-dtype fp8추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)
라이선스
- 라이선스: Apache-2.0 (Gemma 4 라이선스 안내) — 원본 모델의 라이선스를 그대로 따름
- 권리: 원본 가중치·토크나이저·chat template은 Google DeepMind 소유, 이 리포는 양자화 파생물
- Downloads last month
- 220