Arete-27B ยท GGUF

Every llama.cpp quantization of Nous Research's Arete-27B
23 files from 8.4 GB to 53.8 GB ยท importance-matrix calibrated ยท every file validated

Base model llama.cpp imatrix GSQ-RCO License

An independent conversion and quantization by Uraion Labs. Arete-27B was created and post-trained by Nous Research; this is not an official Nous Research release or a new fine-tune.

โšก Quick start

llama-server -hf UraionLabs/Arete-27B-GGUF:Q4_K_M -c 8192

Then open http://127.0.0.1:8080, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Swap Q4_K_M for any format below.

๐Ÿงญ Which file should I use?

Pick a file 1โ€“2 GB smaller than your GPU memory (or unified memory on Apple Silicon), leaving room for the context cache.

Your memory โญ Our pick Also fits
12 GB IQ2_M ยท 10.0 GB IQ2_S, IQ2_XS, IQ2_XXS
16 GB IQ3_M ยท 12.6 GB Q3_K_L, Q3_K_M, Q3_K_S, IQ3_XS, IQ3_XXS, Q2_K
24 GB Q5_K_M ยท 19.2 GB Q5_K_S, Q4_1, Q4_K_M, IQ4_NL, Q4_K_S, Q4_0, IQ4_XS
32 GB Q8_0 ยท 28.6 GB Q6_K_L, Q6_K
64 GB+ BF16 ยท 53.8 GB โ€“

Smaller files lose more quality; the 2-bit files are for tight memory only. For mixed-precision 2โ€“3-bit files that score better at the same size, see Arete-27B-GSQ-RCO-GGUF.

๐Ÿ“ฆ All files

Bits Format Size BPW imatrix Notes
2-bit IQ2_XXS 8.4 GB 2.51 โœ“ Smallest; noticeable quality loss
IQ2_XS 9.1 GB 2.70 โœ“ Very low bit; quality loss likely
IQ2_S 9.4 GB 2.78 โœ“ Low bit; quality loss likely
IQ2_M โญ 10.0 GB 2.98 โœ“ Best of the 2-bit i-quants
Q2_K 10.7 GB 3.19 โ€“ Classic 2-bit k-quant
3-bit IQ3_XXS 11.2 GB 3.33 โœ“ Compact 3-bit i-quant
IQ3_XS 12.0 GB 3.56 โœ“ 3-bit i-quant
Q3_K_S 12.1 GB 3.59 โœ“ Small 3-bit k-quant
IQ3_M โญ 12.6 GB 3.74 โœ“ Strong 3-bit i-quant mix
Q3_K_M 13.3 GB 3.96 โœ“ 3-bit k-quant
Q3_K_L 14.3 GB 4.27 โœ“ Largest 3-bit k-quant
4-bit IQ4_XS 15.1 GB 4.49 โœ“ Compact 4-bit; good size/quality trade-off
Q4_0 15.5 GB 4.62 โœ“ Legacy 4-bit; broad backend support and ARM repacking
Q4_K_S 15.6 GB 4.64 โœ“ Smaller 4-bit k-quant
IQ4_NL 15.8 GB 4.70 โœ“ 4-bit non-linear; good for ARM/CPU repacking
Q4_K_M โญ 16.5 GB 4.92 โ€“ Balanced default: best all-rounder
Q4_1 17.1 GB 5.08 โœ“ Legacy 4-bit with offsets
5-bit Q5_K_S 18.7 GB 5.56 โœ“ 5-bit k-quant
Q5_K_M โญ 19.2 GB 5.72 โœ“ 5-bit k-quant; high quality
6-bit Q6_K 22.1 GB 6.57 โ€“ High precision; close to Q8_0
Q6_K_L 22.7 GB 6.75 โœ“ Q6_K with Q8_0 token embeddings and output tensor
8-bit Q8_0 โญ 28.6 GB 8.51 โ€“ Near-lossless
16-bit BF16 53.8 GB 16.00 โ€“ Original precision

โญ = recommended. BPW is the true whole-file average bits per weight. All 22 quantized files passed three deterministic chat smoke tests; BF16 was validated structurally only.

๐Ÿš€ Running it

llama.cpp
# interactive chat
llama-cli -hf UraionLabs/Arete-27B-GGUF:Q4_K_M -c 8192

# or download one file first
hf download UraionLabs/Arete-27B-GGUF Arete-27B-Q4_K_M.gguf --local-dir .
llama-cli -m Arete-27B-Q4_K_M.gguf -ngl 99 -c 8192

Requires a recent llama.cpp with Qwen3.5 (qwen35) support.

Ollama ยท LM Studio
ollama run hf.co/UraionLabs/Arete-27B-GGUF:Q4_K_M

In LM Studio, search for UraionLabs/Arete-27B-GGUF and pick a file.

Thinking mode & reasoning effort

The original chat template is embedded unchanged. Thinking is on by default at xhigh effort; the template also accepts medium and low. To turn thinking off for an API request:

{"chat_template_kwargs": {"enable_thinking": false}}

With llama-cli, pass --reasoning off.

Images (vision)

These files are text-only. The vision projector mmproj-Arete-27B-BF16.gguf in the companion repo is converted from Arete-27B's own vision tower, so it should pair with any file here via --mmproj. It has been tested only with the GSQ-RCO IQ3_XXS build so far.

About Arete-27B

Arete-27B is Nous Research's supervised fine-tune of Qwen3.8-27B, intended to change value framing while preserving general and agentic capabilities. See the upstream model card and the Nous Research announcement for intended uses, training and evaluations. Upstream benchmark results are not measurements of these quantizations.

๐Ÿ”ฌ Provenance & validation

How these files were made
  • Source: NousResearch/Arete-27B @ 108f0afbdcfaa1ceeda4a635004304dfff6d4da2 โ†’ BF16 GGUF (llama.cpp 29c720eaf) โ†’ every quant directly from that BF16. No requantization.
  • Importance matrix: imatrix-Arete-27B.gguf, computed from Arete-27B itself (Q8_0, 129 ร— 512-token chunks) over the public calibration_datav3 text. It was used for every format marked โœ“. The first-release Q2_K, Q4_K_M and Q6_K were built without one; Q8_0 doesn't use one.
  • Toolchain: imatrix formats built with llama.cpp 0.9.0 (build 12110, commit cd959bae9). Conversion used --no-nextn (the checkpoint has no MTP head).
What was checked
  • Every file: architecture, layer count, tensor names and count, vocabulary size, exact chat-template preservation, and the SHA-256 of the uploaded copy on the Hub.
  • Every quantized file: three deterministic, non-thinking chat smoke tests (factual recall, sequence completion, a Python function). Full transcripts are in quantization_info.json.
  • Not validated: long context, thinking-mode output, tool calling, and task benchmarks. Smoke tests are not quality benchmarks.
Perplexity spot-check

wikitext-2 test split, first 100 ร— 512-token chunks (lower is better). A quick sanity metric, not a task benchmark.

Format BPW PPL
IQ2_XS 2.70 8.07 ยฑ 0.12
IQ2_S 2.78 7.99 ยฑ 0.12
IQ3_XXS 3.33 7.22 ยฑ 0.11
IQ3_XS 3.56 7.06 ยฑ 0.11
IQ3_M 3.74 7.04 ยฑ 0.11
Q8_0 8.51 6.75 ยฑ 0.10

Checksums: SHA256SUMS ยท full metadata, recipes and transcripts: quantization_info.json

License and credit

Apache-2.0, following the source repository; the original LICENSE is included.

Model development and post-training: Nous Research ยท Starting model: Qwen / Alibaba ยท Conversion, quantization, testing and packaging: Uraion Labs

Downloads last month
3,545
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for UraionLabs/Arete-27B-GGUF

Base model

Qwen/Qwen3.8-27B
Quantized
(57)
this model