Instructions to use UraionLabs/Arete-27B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use UraionLabs/Arete-27B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use UraionLabs/Arete-27B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "UraionLabs/Arete-27B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "UraionLabs/Arete-27B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M
- Ollama
How to use UraionLabs/Arete-27B-GGUF with Ollama:
ollama run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use UraionLabs/Arete-27B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "UraionLabs/Arete-27B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use UraionLabs/Arete-27B-GGUF with Docker Model Runner:
docker model run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M
- Lemonade
How to use UraionLabs/Arete-27B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull UraionLabs/Arete-27B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Arete-27B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use UraionLabs/Arete-27B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default UraionLabs/Arete-27B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use UraionLabs/Arete-27B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf UraionLabs/Arete-27B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "UraionLabs/Arete-27B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Arete-27B ยท GGUF
Every llama.cpp quantization of Nous Research's Arete-27B
23 files from 8.4 GB to 53.8 GB ยท importance-matrix calibrated ยท every file validated
An independent conversion and quantization by Uraion Labs. Arete-27B was created and post-trained by Nous Research; this is not an official Nous Research release or a new fine-tune.
โก Quick start
llama-server -hf UraionLabs/Arete-27B-GGUF:Q4_K_M -c 8192
Then open http://127.0.0.1:8080, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Swap Q4_K_M for any format below.
๐งญ Which file should I use?
Pick a file 1โ2 GB smaller than your GPU memory (or unified memory on Apple Silicon), leaving room for the context cache.
| Your memory | โญ Our pick | Also fits |
|---|---|---|
| 12 GB | IQ2_M ยท 10.0 GB | IQ2_S, IQ2_XS, IQ2_XXS |
| 16 GB | IQ3_M ยท 12.6 GB | Q3_K_L, Q3_K_M, Q3_K_S, IQ3_XS, IQ3_XXS, Q2_K |
| 24 GB | Q5_K_M ยท 19.2 GB | Q5_K_S, Q4_1, Q4_K_M, IQ4_NL, Q4_K_S, Q4_0, IQ4_XS |
| 32 GB | Q8_0 ยท 28.6 GB | Q6_K_L, Q6_K |
| 64 GB+ | BF16 ยท 53.8 GB | โ |
Smaller files lose more quality; the 2-bit files are for tight memory only. For mixed-precision 2โ3-bit files that score better at the same size, see Arete-27B-GSQ-RCO-GGUF.
๐ฆ All files
| Bits | Format | Size | BPW | imatrix | Notes |
|---|---|---|---|---|---|
| 2-bit | IQ2_XXS | 8.4 GB | 2.51 | โ | Smallest; noticeable quality loss |
| IQ2_XS | 9.1 GB | 2.70 | โ | Very low bit; quality loss likely | |
| IQ2_S | 9.4 GB | 2.78 | โ | Low bit; quality loss likely | |
| IQ2_M โญ | 10.0 GB | 2.98 | โ | Best of the 2-bit i-quants | |
| Q2_K | 10.7 GB | 3.19 | โ | Classic 2-bit k-quant | |
| 3-bit | IQ3_XXS | 11.2 GB | 3.33 | โ | Compact 3-bit i-quant |
| IQ3_XS | 12.0 GB | 3.56 | โ | 3-bit i-quant | |
| Q3_K_S | 12.1 GB | 3.59 | โ | Small 3-bit k-quant | |
| IQ3_M โญ | 12.6 GB | 3.74 | โ | Strong 3-bit i-quant mix | |
| Q3_K_M | 13.3 GB | 3.96 | โ | 3-bit k-quant | |
| Q3_K_L | 14.3 GB | 4.27 | โ | Largest 3-bit k-quant | |
| 4-bit | IQ4_XS | 15.1 GB | 4.49 | โ | Compact 4-bit; good size/quality trade-off |
| Q4_0 | 15.5 GB | 4.62 | โ | Legacy 4-bit; broad backend support and ARM repacking | |
| Q4_K_S | 15.6 GB | 4.64 | โ | Smaller 4-bit k-quant | |
| IQ4_NL | 15.8 GB | 4.70 | โ | 4-bit non-linear; good for ARM/CPU repacking | |
| Q4_K_M โญ | 16.5 GB | 4.92 | โ | Balanced default: best all-rounder | |
| Q4_1 | 17.1 GB | 5.08 | โ | Legacy 4-bit with offsets | |
| 5-bit | Q5_K_S | 18.7 GB | 5.56 | โ | 5-bit k-quant |
| Q5_K_M โญ | 19.2 GB | 5.72 | โ | 5-bit k-quant; high quality | |
| 6-bit | Q6_K | 22.1 GB | 6.57 | โ | High precision; close to Q8_0 |
| Q6_K_L | 22.7 GB | 6.75 | โ | Q6_K with Q8_0 token embeddings and output tensor | |
| 8-bit | Q8_0 โญ | 28.6 GB | 8.51 | โ | Near-lossless |
| 16-bit | BF16 | 53.8 GB | 16.00 | โ | Original precision |
โญ = recommended. BPW is the true whole-file average bits per weight. All 22 quantized files passed three deterministic chat smoke tests; BF16 was validated structurally only.
๐ Running it
llama.cpp
# interactive chat
llama-cli -hf UraionLabs/Arete-27B-GGUF:Q4_K_M -c 8192
# or download one file first
hf download UraionLabs/Arete-27B-GGUF Arete-27B-Q4_K_M.gguf --local-dir .
llama-cli -m Arete-27B-Q4_K_M.gguf -ngl 99 -c 8192
Requires a recent llama.cpp with Qwen3.5 (qwen35) support.
Ollama ยท LM Studio
ollama run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M
In LM Studio, search for UraionLabs/Arete-27B-GGUF and pick a file.
Thinking mode & reasoning effort
The original chat template is embedded unchanged. Thinking is on by default at xhigh effort; the template also accepts medium and low. To turn thinking off for an API request:
{"chat_template_kwargs": {"enable_thinking": false}}
With llama-cli, pass --reasoning off.
Images (vision)
These files are text-only. The vision projector mmproj-Arete-27B-BF16.gguf in the companion repo is converted from Arete-27B's own vision tower, so it should pair with any file here via --mmproj. It has been tested only with the GSQ-RCO IQ3_XXS build so far.
About Arete-27B
Arete-27B is Nous Research's supervised fine-tune of Qwen3.8-27B, intended to change value framing while preserving general and agentic capabilities. See the upstream model card and the Nous Research announcement for intended uses, training and evaluations. Upstream benchmark results are not measurements of these quantizations.
๐ฌ Provenance & validation
How these files were made
- Source:
NousResearch/Arete-27B@108f0afbdcfaa1ceeda4a635004304dfff6d4da2โ BF16 GGUF (llama.cpp29c720eaf) โ every quant directly from that BF16. No requantization. - Importance matrix:
imatrix-Arete-27B.gguf, computed from Arete-27B itself (Q8_0, 129 ร 512-token chunks) over the public calibration_datav3 text. It was used for every format marked โ. The first-release Q2_K, Q4_K_M and Q6_K were built without one; Q8_0 doesn't use one. - Toolchain: imatrix formats built with llama.cpp
0.9.0 (build 12110, commit cd959bae9). Conversion used--no-nextn(the checkpoint has no MTP head).
What was checked
- Every file: architecture, layer count, tensor names and count, vocabulary size, exact chat-template preservation, and the SHA-256 of the uploaded copy on the Hub.
- Every quantized file: three deterministic, non-thinking chat smoke tests (factual recall, sequence completion, a Python function). Full transcripts are in
quantization_info.json. - Not validated: long context, thinking-mode output, tool calling, and task benchmarks. Smoke tests are not quality benchmarks.
Perplexity spot-check
wikitext-2 test split, first 100 ร 512-token chunks (lower is better). A quick sanity metric, not a task benchmark.
| Format | BPW | PPL |
|---|---|---|
| IQ2_XS | 2.70 | 8.07 ยฑ 0.12 |
| IQ2_S | 2.78 | 7.99 ยฑ 0.12 |
| IQ3_XXS | 3.33 | 7.22 ยฑ 0.11 |
| IQ3_XS | 3.56 | 7.06 ยฑ 0.11 |
| IQ3_M | 3.74 | 7.04 ยฑ 0.11 |
| Q8_0 | 8.51 | 6.75 ยฑ 0.10 |
Checksums: SHA256SUMS ยท full metadata, recipes and transcripts: quantization_info.json
License and credit
Apache-2.0, following the source repository; the original LICENSE is included.
Model development and post-training: Nous Research ยท Starting model: Qwen / Alibaba ยท Conversion, quantization, testing and packaging: Uraion Labs
- Downloads last month
- 3,545