Instructions to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Use Docker
docker model run hf.co/Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with Ollama:
ollama run hf.co/Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with Docker Model Runner:
docker model run hf.co/Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
- Lemonade
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.FineLlama-3.2-3B-Instruct-ead-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
FineLlama-3.2-3B-Instruct-ead-GGUF
GGUF quantized versions of Geraldine/FineLlama-3.2-3B-Instruct-ead model, optimized for efficient inference using llama.cpp.
Model Description
- Base Model: FineLlama-3.2-3B-Instruct-ead
- Quantization: Various GGUF formats
- Purpose: EAD tag generation and archival metadata encoding
- Framework: llama.cpp
Available Variants
The following quantized versions are available:
- Q2_K variant (1.36 GB)
- Q3_K_M variant (1.69 GB)
- Q4_K_M variant (2.02 GB)
- Q5_K_M variant (2.32 GB)
- Q6_K variant (2.64 GB)
- Q8_0 variant (3.42 GB)
- FP16 variant (6.43 GB)
Installation
- Download the desired GGUF model variant
- Install llama.cpp following the official instructions
- Place the model file in your llama.cpp models directory
Usage
# Example using Q4_K_M quantization
./main -m models/FineLlama-3.2-3B-Instruct-ead-Q4_K_M.gguf -n 1024 --repeat_penalty 1.1
# Example using server mode
./server -m models/FineLlama-3.2-3B-Instruct-ead-Q4_K_M.gguf -c 4096
Example using llama-cpp-python library
from llama_cpp import Llama
query = "..."
llm = Llama.from_pretrained(
repo_id="Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF",
filename="*Q8_0.gguf",
n_ctx=1024,
verbose=False
)
output = llm.create_chat_completion(
messages = [
{"role": "system", "content": "You are an archivist expert in EAD format."},
{
"role": "user",
"content": query
}
]
)
print(output["choices"][0]["message"]["content"])
Example using Ollama
ollama run hf.co/Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF:Q4_K_M
Quantization Details
- Q2_K: 2-bit quantization, optimized for efficiency
- Q3_K_M: 3-bit quantization with medium precision
- Q4_K_M: 4-bit quantization with medium precision
- Q5_K_M: 5-bit quantization with medium precision
- Q6_K: 6-bit quantization
- Q8_0: 8-bit quantization, highest precision among quantized versions
- FP16: Full 16-bit floating point, no quantization
Performance Considerations
- Lower bit quantizations (Q2_K, Q3_K_M) offer smaller file sizes but may have slightly reduced accuracy
- Higher bit quantizations (Q6_K, Q8_0) provide better accuracy but require more storage and memory
- FP16 provides full precision but requires significantly more resources
- Downloads last month
- 132
Hardware compatibility
Log In to add your hardware
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support