gemma-4-E2B-it Β· GGUF

GGUF builds of google/gemma-4-E2B-it for llama.cpp / Ollama / LM Studio. Converted by axteam with llama.cpp (converted from the BF16 original; the chat template is embedded in the GGUF).

File Quant Size Notes
gemma-4-E2B-it-GGUF-BF16.gguf BF16 9.3 GB unquantized 16-bit (reference; for re-quantizing or max quality)
gemma-4-E2B-it-GGUF-Q4_K_M.gguf Q4_K_M 3.4 GB 4-bit, the usual size/quality trade-off β€” the most common choice
gemma-4-E2B-it-GGUF-Q8_0.gguf Q8_0 5.0 GB 8-bit, closest to BF16 (larger)

Q4_K_M is the usual default (size/quality trade-off). Q8_0 stays closest to BF16. BF16 is the unquantized reference. Measured agreement with the BF16 file is in the table below. Architecture: Gemma 4 Β· Dense (multimodal; the GGUF text model runs the language side).

Quantization method

These GGUF files are post-training quantization (PTQ) made with llama.cpp β€” the BF16 weights are quantized block-wise (llama.cpp k-quants: per-block scales with mixed bit-width allocation, a refined round-to-nearest), with no retraining. This is a different toolchain from the AWQ / NVFP4 vLLM builds, and it is not QAT (quantization-aware training, which fine-tunes the weights). Q8_0 stays closest to BF16, Q4_K_M is the usual size/quality trade-off (see the measured agreement below), and BF16 is the unquantized reference for re-quantizing.

Measured on GPU (llama.cpp)

llama.cpp b12005 (commit e0f142d, CUDA build) on an NVIDIA B200, 2026-11-02, all layers on the GPU, 16 windows of 2,048 tokens from the WikiText-2 test set and from a Korean Wikipedia sample. Raw-text perplexity of this instruction-tuned checkpoint is very high (the BF16 file scores 135 on WikiText-2 without its chat template), so it says little about quantization loss; each quantized file is compared with the BF16 file on the same tokens instead: mean KL divergence (lower = closer) and how often both pick the same top-1 token, for each quantized file.

File Mean KLD vs BF16 (EN / KO) Top-1 agreement with BF16 (EN / KO)
gemma-4-E2B-it-GGUF-Q4_K_M.gguf 0.4248 / 0.3995 77.7% / 79.9%
gemma-4-E2B-it-GGUF-Q8_0.gguf 0.0086 / 0.0085 96.5% / 96.9%

These figures are measured on the same raw text without the chat template, on which the BF16 file itself is far off (WikiText-2 perplexity 135), so they show how far the quantized files' predictions drift in that off-template setting β€” not chat quality.

Requirements

llama.cpp build β‰₯ b8637 β€” architecture gemma4 needs β‰₯ b8637, and the commands on this card use -fa on (β‰₯ b6325). Run here on 2026-11-02 with b12005 (commit e0f142d, CUDA build) on an NVIDIA B200: every file in this repo loaded with all layers on the GPU, each quantized file's agreement with the BF16 file was measured (see Measured on GPU above) and two short chat questions (capital of France / of Korea, one-word answer requested) were asked with every file; every reply named the right answer except gemma-4-E2B-it-GGUF-Q4_K_M.gguf on the Korean question (repeated the question instead of answering), but none of those replies was a direct one-word answer: they were reasoning-style notes that restate the request and contain the answer.
Memory the chosen GGUF file (sizes in the table) plus KV cache for the context (-c); --cache-type-k/v q8_0 -fa on halves the KV cache.
Input text only β€” this repo has no mmproj (vision projector) file.
Ollama / LM Studio not tested here.

Run with llama.cpp

# download one file (e.g. Q4_K_M) then:
llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -c 8192
# or a single answer and exit (-st):
llama-cli   -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -p "λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ”?" -st

Reduce KV-cache memory (optional, run-time)

KV-cache type is a run-time choice in llama.cpp (not baked into the file). Halve it with:

llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf --cache-type-k q8_0 --cache-type-v q8_0 -fa on

Ollama

ollama run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.

License

Quantized derivative of Google's Gemma 4 β€” governed by the Gemma 4 license. Weights, tokenizer, and chat template belong to Google DeepMind.


ν•œκ΅­μ–΄ μš”μ•½

google/gemma-4-E2B-it의 GGUF μ–‘μžν™” λΉŒλ“œ (llama.cppΒ·OllamaΒ·LM Studio용) Β· μ œμž‘ axteam

κ°œμš”

ν•­λͺ© λ‚΄μš©
원본 λͺ¨λΈ google/gemma-4-E2B-it
ν˜•μ‹ GGUF (llama.cpp)
λ³€ν™˜ 도ꡬ llama.cpp convert_hf_to_gguf.py β†’ llama-quantize
λͺ¨λΈ ꡬ쑰 Gemma 4 Β· Dense (원본은 λ©€ν‹°λͺ¨λ‹¬, GGUFλŠ” μ–Έμ–΄ λͺ¨λΈ λΆ€λΆ„λ§Œ μ‹€ν–‰)
파일 수 3개

μ–‘μžν™” 상세

  • μ–‘μžν™” 절차: PTQ(ν•™μŠ΅ ν›„ μ–‘μžν™”, μž¬ν•™μŠ΅ μ—†μŒ), llama.cpp 블둝 λ‹¨μœ„ μ–‘μžν™”(k-quant Q*_K, Q8_0)
  • chat template: GGUF νŒŒμΌμ— λ‚΄μž₯
  • BF16 μΌμΉ˜λ„(μ‹€μΈ‘, 2026-11-02): μ–‘μžν™” νŒŒμΌλ§ˆλ‹€ BF16 파일과의 μΌμΉ˜λ„(같은 ν…μŠ€νŠΈ μ˜μ–΄/ν•œκ΅­μ–΄ 2,048토큰×16, llama.cpp KL divergence 평균·top-1 토큰 일치율, B200 GPU) β€” gemma-4-E2B-it-GGUF-Q4_K_M.gguf KLD 0.4248/0.3995, top-1 일치 77.7%/79.9% Β· gemma-4-E2B-it-GGUF-Q8_0.gguf KLD 0.0086/0.0085, top-1 일치 96.5%/96.9% (chat template μ—†λŠ” 같은 ν…μŠ€νŠΈμ—μ„œ 잰 κ°’ β€” BF16 파일 μžμ²΄λ„ WikiText-2 PPL이 135둜 맀우 높은 쑰건이라 μ±„νŒ… ν’ˆμ§ˆμ΄ μ•„λ‹ˆλΌ κ·Έ μ‘°κ±΄μ—μ„œ 예츑이 BF16κ³Ό μ–Όλ§ˆλ‚˜ λ‹¬λΌμ§€λŠ”μ§€λ₯Ό λ‚˜νƒ€λƒ„)

제곡 파일

파일 크기 μ„€λͺ…
gemma-4-E2B-it-GGUF-BF16.gguf 9.3 GB μ–‘μžν™”ν•˜μ§€ μ•Šμ€ 16λΉ„νŠΈ β€” κΈ°μ€€ 파일 (μž¬μ–‘μžν™”μš©Β·μ΅œκ³  ν’ˆμ§ˆ)
gemma-4-E2B-it-GGUF-Q4_K_M.gguf 3.4 GB 4λΉ„νŠΈ β€” ν¬κΈ°Β·ν’ˆμ§ˆ κ· ν˜•, κ°€μž₯ 많이 μ“°λŠ” 선택
gemma-4-E2B-it-GGUF-Q8_0.gguf 5.0 GB 8λΉ„νŠΈ β€” BF16에 κ°€μž₯ κ°€κΉŒμ›€, μš©λŸ‰ 큼

μš”κ΅¬μ‚¬ν•­

  • llama.cpp: b8637 이상 (gemma4 ꡬ쑰 지원 b8637λΆ€ν„°, μ˜ˆμ‹œ λͺ…λ Ήμ˜ -fa on μ˜΅μ…˜ b6325λΆ€ν„°)
  • μ‹€ν–‰ 확인: 2026-11-02 b12005(컀밋 e0f142d) CUDA λΉŒλ“œλ‘œ NVIDIA B200μ—μ„œ 이 리포의 λͺ¨λ“  νŒŒμΌμ„ μ „ μΈ΅ GPU에 올렀 μ‹€ν–‰ β€” μ–‘μžν™” 파일과 BF16 파일의 μΌμΉ˜λ„ μΈ‘μ •(μœ„ μ˜μ–΄ λ³Έλ¬Έ ν‘œ), λͺ¨λ“  νŒŒμΌμ— 짧은 질문 2개(ν”„λž‘μŠ€Β·λŒ€ν•œλ―Όκ΅­ μˆ˜λ„, ν•œ 단어 λ‹΅ μš”μ²­)λ₯Ό 물음 β€” gemma-4-E2B-it-GGUF-Q4_K_M.gguf νŒŒμΌμ€ ν•œκ΅­μ–΄ μ§ˆλ¬Έμ—μ„œ λ‹΅ λŒ€μ‹  μ§ˆλ¬Έμ„ κ·ΈλŒ€λ‘œ λ˜ν’€μ΄ν•¨, λ‚˜λ¨Έμ§€ λ‹΅μ—λŠ” λͺ¨λ‘ 정닡이 λ“€μ–΄ 있음, λ‹€λ§Œ ν•œ λ‹¨μ–΄λ‘œ λ°”λ‘œ λ‹΅ν•œ κ²½μš°λŠ” μ—†κ³  λͺ¨λ‘ μš”μ²­μ„ ν’€μ–΄ μ“°λŠ” 좔둠식 λ¬Έμž₯ 속에 정닡이 λ“€μ–΄ μžˆλŠ” ν˜•νƒœ
  • λ©”λͺ¨λ¦¬: μ„ νƒν•œ GGUF 파일 크기 + μ»¨ν…μŠ€νŠΈ(-c)만큼의 KV μΊμ‹œ (KV μΊμ‹œ μ€„μ΄λŠ” 방법은 μ•„λž˜ μ‚¬μš©λ²• μ°Έκ³ )
  • μž…λ ₯: ν…μŠ€νŠΈ μ „μš© β€” λΉ„μ „ μΈμ½”λ”Β·νˆ¬μ˜μΈ΅ 파일(mmproj)이 μ—†μ–΄ 이미지 μž…λ ₯ λΆˆκ°€
  • OllamaΒ·LM Studio: μ œμž‘ μΈ‘μ—μ„œ μ‹€ν–‰ 미확인

μ‚¬μš©λ²•

  • 파일 λ°›κΈ°: hf download axteam/gemma-4-E2B-it-GGUF gemma-4-E2B-it-GGUF-Q4_K_M.gguf --local-dir . (μ•„λž˜ -m λͺ…령에 ν•„μš”, λ‹€λ₯Έ μ–‘μžν™”λŠ” 파일λͺ…μ˜ Q4_K_M만 λ³€κ²½)
  • μ„œλ²„ μ‹€ν–‰: llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -c 8192
  • ν•œ 번 질문 (λ‹΅λ³€ ν›„ μ’…λ£Œ): llama-cli -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -p "λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ”?" -st
  • Ollama: ollama run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
  • KV μΊμ‹œ 절감(선택): --cache-type-k q8_0 --cache-type-v q8_0 -fa on β€” KV μΊμ‹œ μ•½ 절반, GGUF 파일과 상관없이 μ‹€ν–‰ μ‹œ μ§€μ •
  • ꢌμž₯ μƒ˜ν”Œλ§: temperature=1.0, top_p=0.95, top_k=64 (Google ꢌμž₯κ°’)

λΌμ΄μ„ μŠ€

  • λΌμ΄μ„ μŠ€: Apache-2.0 (Gemma 4 λΌμ΄μ„ μŠ€ μ•ˆλ‚΄) β€” 원본 λͺ¨λΈμ˜ λΌμ΄μ„ μŠ€λ₯Ό κ·ΈλŒ€λ‘œ 따름
  • ꢌ리: 원본 κ°€μ€‘μΉ˜Β·ν† ν¬λ‚˜μ΄μ €Β·chat template은 Google DeepMind μ†Œμœ , 이 λ¦¬ν¬λŠ” μ–‘μžν™” νŒŒμƒλ¬Ό
Downloads last month
2,011
GGUF
Model size
5B params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for axteam/gemma-4-E2B-it-GGUF

Quantized
(472)
this model

Collection including axteam/gemma-4-E2B-it-GGUF