Instructions to use axteam/gemma-4-E2B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use axteam/gemma-4-E2B-it-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Use Docker
docker model run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use axteam/gemma-4-E2B-it-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "axteam/gemma-4-E2B-it-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "axteam/gemma-4-E2B-it-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
- Ollama
How to use axteam/gemma-4-E2B-it-GGUF with Ollama:
ollama run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use axteam/gemma-4-E2B-it-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "axteam/gemma-4-E2B-it-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use axteam/gemma-4-E2B-it-GGUF with Docker Model Runner:
docker model run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
- Lemonade
How to use axteam/gemma-4-E2B-it-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.gemma-4-E2B-it-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use axteam/gemma-4-E2B-it-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use axteam/gemma-4-E2B-it-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "axteam/gemma-4-E2B-it-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
gemma-4-E2B-it Β· GGUF
GGUF builds of google/gemma-4-E2B-it for llama.cpp / Ollama / LM Studio. Converted by axteam with llama.cpp (converted from the BF16 original; the chat template is embedded in the GGUF).
| File | Quant | Size | Notes |
|---|---|---|---|
gemma-4-E2B-it-GGUF-BF16.gguf |
BF16 | 9.3 GB | unquantized 16-bit (reference; for re-quantizing or max quality) |
gemma-4-E2B-it-GGUF-Q4_K_M.gguf |
Q4_K_M | 3.4 GB | 4-bit, the usual size/quality trade-off β the most common choice |
gemma-4-E2B-it-GGUF-Q8_0.gguf |
Q8_0 | 5.0 GB | 8-bit, closest to BF16 (larger) |
Q4_K_M is the usual default (size/quality trade-off). Q8_0 stays closest to BF16. BF16 is the unquantized reference. Measured agreement with the BF16 file is in the table below. Architecture: Gemma 4 Β· Dense (multimodal; the GGUF text model runs the language side).
Quantization method
These GGUF files are post-training quantization (PTQ) made with llama.cpp β the BF16 weights are quantized block-wise (llama.cpp k-quants: per-block scales with mixed bit-width allocation, a refined round-to-nearest), with no retraining. This is a different toolchain from the AWQ / NVFP4 vLLM builds, and it is not QAT (quantization-aware training, which fine-tunes the weights). Q8_0 stays closest to BF16, Q4_K_M is the usual size/quality trade-off (see the measured agreement below), and BF16 is the unquantized reference for re-quantizing.
Measured on GPU (llama.cpp)
llama.cpp b12005 (commit e0f142d, CUDA build) on an NVIDIA B200, 2026-11-02, all layers on the GPU, 16 windows of 2,048 tokens from the WikiText-2 test set and from a Korean Wikipedia sample. Raw-text perplexity of this instruction-tuned checkpoint is very high (the BF16 file scores 135 on WikiText-2 without its chat template), so it says little about quantization loss; each quantized file is compared with the BF16 file on the same tokens instead: mean KL divergence (lower = closer) and how often both pick the same top-1 token, for each quantized file.
| File | Mean KLD vs BF16 (EN / KO) | Top-1 agreement with BF16 (EN / KO) |
|---|---|---|
gemma-4-E2B-it-GGUF-Q4_K_M.gguf |
0.4248 / 0.3995 | 77.7% / 79.9% |
gemma-4-E2B-it-GGUF-Q8_0.gguf |
0.0086 / 0.0085 | 96.5% / 96.9% |
These figures are measured on the same raw text without the chat template, on which the BF16 file itself is far off (WikiText-2 perplexity 135), so they show how far the quantized files' predictions drift in that off-template setting β not chat quality.
Requirements
| llama.cpp | build β₯ b8637 β architecture gemma4 needs β₯ b8637, and the commands on this card use -fa on (β₯ b6325). Run here on 2026-11-02 with b12005 (commit e0f142d, CUDA build) on an NVIDIA B200: every file in this repo loaded with all layers on the GPU, each quantized file's agreement with the BF16 file was measured (see Measured on GPU above) and two short chat questions (capital of France / of Korea, one-word answer requested) were asked with every file; every reply named the right answer except gemma-4-E2B-it-GGUF-Q4_K_M.gguf on the Korean question (repeated the question instead of answering), but none of those replies was a direct one-word answer: they were reasoning-style notes that restate the request and contain the answer. |
| Memory | the chosen GGUF file (sizes in the table) plus KV cache for the context (-c); --cache-type-k/v q8_0 -fa on halves the KV cache. |
| Input | text only β this repo has no mmproj (vision projector) file. |
| Ollama / LM Studio | not tested here. |
Run with llama.cpp
# download one file (e.g. Q4_K_M) then:
llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -c 8192
# or a single answer and exit (-st):
llama-cli -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -p "λνλ―Όκ΅μ μλλ?" -st
Reduce KV-cache memory (optional, run-time)
KV-cache type is a run-time choice in llama.cpp (not baked into the file). Halve it with:
llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf --cache-type-k q8_0 --cache-type-v q8_0 -fa on
Ollama
ollama run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M
Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64.
License
Quantized derivative of Google's Gemma 4 β governed by the Gemma 4 license. Weights, tokenizer, and chat template belong to Google DeepMind.
νκ΅μ΄ μμ½
google/gemma-4-E2B-itμ GGUF μμν λΉλ (llama.cppΒ·OllamaΒ·LM Studioμ©) Β· μ μ axteam
κ°μ
| νλͺ© | λ΄μ© |
|---|---|
| μλ³Έ λͺ¨λΈ | google/gemma-4-E2B-it |
| νμ | GGUF (llama.cpp) |
| λ³ν λꡬ | llama.cpp convert_hf_to_gguf.py β llama-quantize |
| λͺ¨λΈ ꡬ쑰 | Gemma 4 Β· Dense (μλ³Έμ λ©ν°λͺ¨λ¬, GGUFλ μΈμ΄ λͺ¨λΈ λΆλΆλ§ μ€ν) |
| νμΌ μ | 3κ° |
μμν μμΈ
- μμν μ μ°¨: PTQ(νμ΅ ν μμν, μ¬νμ΅ μμ), llama.cpp λΈλ‘ λ¨μ μμν(k-quant
Q*_K,Q8_0) - chat template: GGUF νμΌμ λ΄μ₯
- BF16 μΌμΉλ(μ€μΈ‘, 2026-11-02): μμν νμΌλ§λ€ BF16 νμΌκ³Όμ μΌμΉλ(κ°μ ν
μ€νΈ μμ΄/νκ΅μ΄ 2,048ν ν°Γ16, llama.cpp KL divergence νκ· Β·top-1 ν ν° μΌμΉμ¨, B200 GPU) β
gemma-4-E2B-it-GGUF-Q4_K_M.ggufKLD 0.4248/0.3995, top-1 μΌμΉ 77.7%/79.9% Β·gemma-4-E2B-it-GGUF-Q8_0.ggufKLD 0.0086/0.0085, top-1 μΌμΉ 96.5%/96.9% (chat template μλ κ°μ ν μ€νΈμμ μ° κ° β BF16 νμΌ μ체λ WikiText-2 PPLμ΄ 135λ‘ λ§€μ° λμ 쑰건μ΄λΌ μ±ν νμ§μ΄ μλλΌ κ·Έ 쑰건μμ μμΈ‘μ΄ BF16κ³Ό μΌλ§λ λ¬λΌμ§λμ§λ₯Ό λνλ)
μ 곡 νμΌ
| νμΌ | ν¬κΈ° | μ€λͺ |
|---|---|---|
gemma-4-E2B-it-GGUF-BF16.gguf |
9.3 GB | μμννμ§ μμ 16λΉνΈ β κΈ°μ€ νμΌ (μ¬μμνμ©Β·μ΅κ³ νμ§) |
gemma-4-E2B-it-GGUF-Q4_K_M.gguf |
3.4 GB | 4λΉνΈ β ν¬κΈ°Β·νμ§ κ· ν, κ°μ₯ λ§μ΄ μ°λ μ ν |
gemma-4-E2B-it-GGUF-Q8_0.gguf |
5.0 GB | 8λΉνΈ β BF16μ κ°μ₯ κ°κΉμ, μ©λ νΌ |
μꡬμ¬ν
- llama.cpp: b8637 μ΄μ (
gemma4ꡬ쑰 μ§μ b8637λΆν°, μμ λͺ λ Ήμ-fa onμ΅μ b6325λΆν°) - μ€ν νμΈ: 2026-11-02 b12005(μ»€λ° e0f142d) CUDA λΉλλ‘ NVIDIA B200μμ μ΄ λ¦¬ν¬μ λͺ¨λ νμΌμ μ μΈ΅ GPUμ μ¬λ € μ€ν β μμν νμΌκ³Ό BF16 νμΌμ μΌμΉλ μΈ‘μ (μ μμ΄ λ³Έλ¬Έ ν), λͺ¨λ νμΌμ μ§§μ μ§λ¬Έ 2κ°(νλμ€Β·λνλ―Όκ΅ μλ, ν λ¨μ΄ λ΅ μμ²)λ₯Ό λ¬Όμ β
gemma-4-E2B-it-GGUF-Q4_K_M.ggufνμΌμ νκ΅μ΄ μ§λ¬Έμμ λ΅ λμ μ§λ¬Έμ κ·Έλλ‘ λνμ΄ν¨, λλ¨Έμ§ λ΅μλ λͺ¨λ μ λ΅μ΄ λ€μ΄ μμ, λ€λ§ ν λ¨μ΄λ‘ λ°λ‘ λ΅ν κ²½μ°λ μκ³ λͺ¨λ μμ²μ νμ΄ μ°λ μΆλ‘ μ λ¬Έμ₯ μμ μ λ΅μ΄ λ€μ΄ μλ νν - λ©λͺ¨λ¦¬: μ νν GGUF νμΌ ν¬κΈ° + 컨ν
μ€νΈ(
-c)λ§νΌμ KV μΊμ (KV μΊμ μ€μ΄λ λ°©λ²μ μλ μ¬μ©λ² μ°Έκ³ ) - μ
λ ₯: ν
μ€νΈ μ μ© β λΉμ μΈμ½λΒ·ν¬μμΈ΅ νμΌ(
mmproj)μ΄ μμ΄ μ΄λ―Έμ§ μ λ ₯ λΆκ° - OllamaΒ·LM Studio: μ μ μΈ‘μμ μ€ν λ―ΈνμΈ
μ¬μ©λ²
- νμΌ λ°κΈ°:
hf download axteam/gemma-4-E2B-it-GGUF gemma-4-E2B-it-GGUF-Q4_K_M.gguf --local-dir .(μλ-mλͺ λ Ήμ νμ, λ€λ₯Έ μμνλ νμΌλͺ μQ4_K_Mλ§ λ³κ²½) - μλ² μ€ν:
llama-server -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -c 8192 - ν λ² μ§λ¬Έ (λ΅λ³ ν μ’
λ£):
llama-cli -m gemma-4-E2B-it-GGUF-Q4_K_M.gguf -p "λνλ―Όκ΅μ μλλ?" -st - Ollama:
ollama run hf.co/axteam/gemma-4-E2B-it-GGUF:Q4_K_M - KV μΊμ μ κ°(μ ν):
--cache-type-k q8_0 --cache-type-v q8_0 -fa onβ KV μΊμ μ½ μ λ°, GGUF νμΌκ³Ό μκ΄μμ΄ μ€ν μ μ§μ - κΆμ₯ μνλ§:
temperature=1.0,top_p=0.95,top_k=64(Google κΆμ₯κ°)
λΌμ΄μ μ€
- λΌμ΄μ μ€: Apache-2.0 (Gemma 4 λΌμ΄μ μ€ μλ΄) β μλ³Έ λͺ¨λΈμ λΌμ΄μ μ€λ₯Ό κ·Έλλ‘ λ°λ¦
- κΆλ¦¬: μλ³Έ κ°μ€μΉΒ·ν ν¬λμ΄μ Β·chat templateμ Google DeepMind μμ , μ΄ λ¦¬ν¬λ μμν νμλ¬Ό
- Downloads last month
- 2,011
4-bit
8-bit
16-bit