Instructions to use axteam/gemma-4-26B-A4B-AWQ-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use axteam/gemma-4-26B-A4B-AWQ-8bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="axteam/gemma-4-26B-A4B-AWQ-8bit")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("axteam/gemma-4-26B-A4B-AWQ-8bit") model = AutoModelForMultimodalLM.from_pretrained("axteam/gemma-4-26B-A4B-AWQ-8bit", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use axteam/gemma-4-26B-A4B-AWQ-8bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "axteam/gemma-4-26B-A4B-AWQ-8bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-AWQ-8bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/axteam/gemma-4-26B-A4B-AWQ-8bit
- SGLang
How to use axteam/gemma-4-26B-A4B-AWQ-8bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "axteam/gemma-4-26B-A4B-AWQ-8bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-AWQ-8bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "axteam/gemma-4-26B-A4B-AWQ-8bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-26B-A4B-AWQ-8bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use axteam/gemma-4-26B-A4B-AWQ-8bit with Docker Model Runner:
docker model run hf.co/axteam/gemma-4-26B-A4B-AWQ-8bit
gemma-4-26B-A4B Β· AWQ INT8 (W8A16)
8-bit quantization of google/gemma-4-26B-A4B (commit 24548b6), produced for efficient vLLM serving. Quantized by axteam.
| Base model | google/gemma-4-26B-A4B (commit 24548b6) |
| Method | AWQ INT8 (W8A16) |
| Format | compressed-tensors pack-quantized |
| Weights / activations | W8A16 β INT8 symmetric weights, group size 32, BF16 activations |
| Architecture | Gemma 4 Β· Mixture-of-Experts (8 active / 128 total + 1 shared) |
| Size on disk | ~29.2 GB |
| Calibration | 256 samples (Korean-heavy; -it includes 32 self-generated thinking-mode answers) |
| License | Gemma 4 / Apache-2.0 |
Quantization details
- Scheme β W8A16 β INT8 symmetric weights, group size 32, BF16 activations, stored as compressed-tensors
pack-quantized. - Kept in BF16 β vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms, plus the shared MLP. These stay unquantized to preserve quality β about 7% of all parameters in this build.
- Tokenizer /
processor_config/generation_configare byte-identical to the original. This is a base (pre-trained) model with no chat template β use plain text completion, not chat.config.jsonis the original plus aquantization_configblock.
Quantization method β PTQ (AWQ / RTN) vs QAT
This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
- RTN (round-to-nearest) β weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which all apply AWQ (see each repo's
recipe.yaml). - AWQ (activation-aware weight quantization) β a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm β q/k/v) before rounding. Still PTQ (no gradient, secondsβminutes), and far better than RTN at 4-bit.
- QAT (quantization-aware training) β the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.
Rough low-bit fidelity: QAT β₯ AWQ > RTN. Cost: QAT β« AWQ > RTN. This build: AWQ β a calibration-based PTQ (256-sample calibration, no gradient).
Benchmarks (measured)
Every column was measured with the same harness and settings β lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit (this repo) | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot, generative) | 77.77 | 77.25 (99.3%) | 77.75 (100.0%) | 77.18 (99.2%) | 76.20 (98.0%) | 76.82 (98.8%) |
| GSM8K (5-shot, strict) | 70.89 | 71.34 (100.6%) | 71.49 (100.9%) | 69.98 (98.7%) | 67.17 (94.8%) | 69.98 (98.7%) |
| ARC-Challenge (0-shot) | 68.34 | 68.60 (100.4%) | 68.17 (99.8%) | 68.52 (100.2%) | 66.98 (98.0%) | 68.69 (100.5%) |
| Winogrande (5-shot) | 80.58 | 80.03 (99.3%) | 80.58 (100.0%) | 79.64 (98.8%) | 78.22 (97.1%) | 79.56 (98.7%) |
| KMMLU (5-shot, direct) | 60.03 | 60.07 (100.1%) | 60.15 (100.2%) | 59.41 (99.0%) | 58.14 (96.9%) | 58.99 (98.3%) |
| HAE-RAE (0-shot) | 77.27 | 76.99 (99.6%) | 76.90 (99.5%) | 77.18 (99.9%) | 75.44 (97.6%) | 75.80 (98.1%) |
| Average | 72.48 | 72.38 (99.9%) | 72.51 (100.0%) | 71.98 (99.3%) | 70.36 (97.1%) | 71.64 (98.8%) |
Evaluation (measured)
Perplexity was measured against the BF16 original on public text β the WikiText-2 test set (English) and a Korean Wikipedia sample, 2,048-token chunks. 8-bit builds (AWQ INT8, FP8) stayed within 0.5% of BF16; 4-bit builds (AWQ INT4, NVFP4) were 2.0β6.6% higher (range across both base models and both 4-bit formats). Next-token top-1 agreement was not measured for the base models.
Requirements
| Runtime | vLLM β the architecture is in vLLM β₯ 0.19.0; tested here with vLLM 0.32.0 (text input only β the benchmarks above, run with image input disabled; image input not run for this build). |
| vLLM install | vLLM 0.32 itself needs Python 3.11β3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers β₯5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 β do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer β weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 29.2 GB. Add KV cache (grows with context length Γ concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | Not loadable by Transformers (checked up to 5.22.1: no conversion rule for per-expert checkpoints, the experts stay uninitialised) β use vLLM. |
| trust_remote_code | not needed |
Usage (vLLM)
vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit
from vllm import LLM, SamplingParams
llm = LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.generate("The three primary colors are", params)
print(out[0].outputs[0].text)
Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64.
Reducing KV-cache memory (optional, serve-time)
The KV cache is unquantized (BF16) here β the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
License
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights and tokenizer belong to Google DeepMind.
νκ΅μ΄ μμ½
google/gemma-4-26B-A4Bμ AWQ INT8 (W8A16) μμν λΉλ Β· μ μ axteam
κ°μ
| νλͺ© | λ΄μ© |
|---|---|
| μλ³Έ λͺ¨λΈ | google/gemma-4-26B-A4B (μ»€λ° 24548b6) |
| μμν λ°©μ | AWQ INT8 (W8A16) |
| μ μ₯ νμ | compressed-tensors pack-quantized |
| λͺ¨λΈ ꡬ쑰 | Gemma 4 Β· MoE (μ λ¬Έκ° 128κ° μ€ 8κ° νμ± + νμ μ°λ 곡μ MLP) |
| μ©λ | μ½ 29.2 GB |
| 보μ λ°μ΄ν° | μν 256κ° (νκ΅μ΄ μμ£Ό) |
μμν μμΈ
- κ°μ€μΉ: INT8 λμΉ, κ·Έλ£Ή ν¬κΈ° 32
- νμ±ν: BF16 (μμννμ§ μμ)
- μμν μ μ°¨: PTQ(νμ΅ ν μμν, μ¬νμ΅ μμ), AWQ λ°©μ β 보μ λ°μ΄ν°λ‘ νμ±νκ° ν° κ°μ€μΉ μ±λμ μ°Ύμ μ±λλ³ μ€μΌμΌλ‘ 보νΈν λ€ INT8λ‘ λ°μ¬λ¦Ό
- BF16 μ μ§: λΉμ νμ, λΉμ ν¬μμΈ΅, ν ν° μλ² λ©(LM ν€λμ 곡μ ), MoE λΌμ°ν°, λͺ¨λ μ κ·νμΈ΅, 곡μ MLP β μ 체 νλΌλ―Έν°μ μ½ 7%
- μλ³Έ νμΌ: ν ν¬λμ΄μ Β·processorΒ·generation configλ₯Ό λ°μ΄νΈ λ¨μλ‘ κ·Έλλ‘ μ¬μ©(μ¬μ νμ΅ λͺ¨λΈμ΄λΌ chat templateμ μμ),
config.jsonμ μλ³Έμquantization_configλΈλ‘λ§ μΆκ°
νμ§ (μ€μΈ‘)
- κ³΅κ° ν μ€νΈ PPL: WikiText-2(μμ΄)Β·νκ΅μ΄ μν€ νλ³Έ, 2,048ν ν° λ¨μ β 8λΉνΈ(AWQ INT8Β·FP8)λ μλ³Έ λλΉ 0.5% μ΄λ΄, 4λΉνΈ(AWQ INT4Β·NVFP4)λ 2.0~6.6% μμΉ (λ base λͺ¨λΈΒ·λ 4λΉνΈ νμμ ν©μΉ λ²μ, top-1μ λ―ΈμΈ‘μ )
- λ²€μΉλ§ν¬ νκ· : 72.51 (μλ³Έ 72.48, 100.0%) β MMLU(μμ±ν)Β·GSM8KΒ·ARC-ChallengeΒ·WinograndeΒ·KMMLUΒ·HAE-RAE 6μ’ νκ· , lm-evaluation-harness + vLLM, chat template μλ μμ±ν ν둬ννΈ(κ³Όμ λ³ 0Β·5-shot, ν μ°Έκ³ )Β·greedy, λͺ¨λ λΉλλ₯Ό κ°μ μ€μ μΌλ‘ μΈ‘μ
- νκ΅μ΄ λ²€μΉλ§ν¬: KMMLU 60.15 (μλ³Έ 60.03, 100.2%) Β· HAE-RAE 76.90 (μλ³Έ 77.27, 99.5%)
- MMLU λ°©μ: lm-eval μμ±ν λ³ν(
mmlu_generative) β λͺ¨λΈμ΄ λ΅ κΈμλ₯Ό μμ±νκ³ μ²« μ€μ΄ μ νν μΌμΉν΄μΌ μ λ΅μΌλ‘ μ±μ νλ―λ‘, λ€λ₯Έ λ°©μ(μ: log-likelihood μ±μ )μΌλ‘ μ° MMLU μ μμ μ§μ λΉκ΅ λΆκ°
μꡬμ¬ν
- μ€ν μμ§: vLLM 0.19.0 μ΄μ(μ΄ κ΅¬μ‘°λ₯Ό μ§μνλ 첫 λ²μ ), μ΄ λΉλλ vLLM 0.32.0μμ μ€ν νμΈ (ν μ€νΈ μ λ ₯λ§ β μ λ²€μΉλ§ν¬, μ΄λ―Έμ§ μ λ ₯μ λκ³ μΈ‘μ , μ΄ λΉλμ μ΄λ―Έμ§ μ λ ₯μ μ€ν λ―ΈνμΈ)
- vLLM 0.32 μ€μΉ 쑰건: Python 3.11~3.14, torch 2.13(CUDA 13.0 λΉλ, CUDA 13 μ§μ λλΌμ΄λ² νμ), transformers 5.16.1 μ΄μ 5.20 λ―Έλ§, compressed-tensors 0.17.0 β transformersΒ·compressed-tensorsλ vLLMμ΄ λ²μ μ κ³ μ νλ―λ‘ compressed-tensorsλ₯Ό λ°λ‘ μ κ·Έλ μ΄λνμ§ λ§ κ² (μ΄ μ²΄ν¬ν¬μΈνΈλ 0.19.0μΌλ‘ μ μ₯λμ§λ§ 0.17.0μμλ λ‘λλ¨)
- GPU: NVIDIA Turing(sm75) μ΄μ, V100(sm70) λ―Έμ§μ β κ°μ€μΉ μ μ© μ μ νμμ΄λΌ FP8Β·FP4 μ μ© ν μμ½μ΄ λΆνμ (GPUλ³ μ§μ μ¬λΆλ vLLM 0.32 컀λ μ ν μ½λ κΈ°μ€, μ€μ μ€νμ B200μμλ§ νμΈ)
- λ©λͺ¨λ¦¬: κ°μ€μΉ 29.2 GB + KV μΊμ(컨ν
μ€νΈ κΈΈμ΄ Γ λμ μμ² μμ λΉλ‘) + νμ±νΒ·CUDA graphμ© μ GB β GPU ν μ₯μ λ€ λ€μ΄κ°μ§ μμΌλ©΄
--tensor-parallel-size NμΌλ‘ λΆμ° - Transformersλ‘ μ§μ μ¬μ©: λΆκ° β 5.22.1κΉμ§ λ‘λλμ§ μμ νμΈ (μ λ¬Έκ°λ³ 체ν¬ν¬μΈνΈ λ³ν κ·μΉμ΄ μμ΄ μ λ¬Έκ°κ° μ΄κΈ°νλμ§ μμ), vLLM μ¬μ©
- trust_remote_code: λΆνμ
μ¬μ©λ²
- μλ² μ€ν:
vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit - Python:
LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit").generate(...)β μ¬μ νμ΅ λͺ¨λΈμ΄λΌ λνκ° μλ ν μ€νΈ μ΄μ΄μ°κΈ° - κΆμ₯ μνλ§:
temperature=1.0,top_p=0.95,top_k=64(Google κΆμ₯κ°) - KV μΊμ μ κ°(μ ν):
vllm serveμ--kv-cache-dtype fp8μΆκ° β κΈ°λ³Έ BF16μΈ KV μΊμλ₯Ό FP8λ‘ μ μ₯ν΄ KV μΊμ λ©λͺ¨λ¦¬λ₯Ό μ½ μ λ°μΌλ‘ μ€μ (κΈ΄ 컨ν μ€νΈΒ·λμ μμ²μ΄ λ§μ λ μ 리, κ°μ€μΉ μμνμ λ³κ°μΈ μ€ν μ΅μ )
λΌμ΄μ μ€
- λΌμ΄μ μ€: Apache-2.0 (Gemma 4 λΌμ΄μ μ€ μλ΄) β μλ³Έ λͺ¨λΈμ λΌμ΄μ μ€λ₯Ό κ·Έλλ‘ λ°λ¦
- κΆλ¦¬: μλ³Έ κ°μ€μΉΒ·ν ν¬λμ΄μ λ Google DeepMind μμ , μ΄ λ¦¬ν¬λ μμν νμλ¬Ό
- Downloads last month
- 191
Model tree for axteam/gemma-4-26B-A4B-AWQ-8bit
Base model
google/gemma-4-26B-A4B