prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF

DeepSeek-V4.1-Flash-1017 GGUF: imatrix quants for one big workstation

Three imatrix-guided quants of DeepSeek-V4.1-Flash-1017, built and validated on one machine: an RTX PRO 6000 96 GB, a 16 GB second card, and 128 GB of RAM. The routed experts are split across GPU and RAM, and the ~195 GiB engram tables are memory-mapped and read from disk on demand, so they cost disk space, not RAM.

All three quants were measured against the release-precision experts, not against another quant, and all three answer all 16 retrieval and multi-hop questions exactly at depths up to 255K tokens.

These files need a DeepSeek-V4.1 build of llama.cpp. Upstream llama.cpp cannot load DeepSeek-V4.1 yet (#28696 is open). See Runtime.

Which one do I want?

	IQ3_XXS (recommended)	IQ2_XS	IQ2_XXS
Weights you must hold in VRAM + RAM	193.9 GiB	171.7 GiB	152.0 GiB
Engram tables (mmapped from disk)	194.6 GiB	194.6 GiB (identical files)	194.6 GiB (identical files)
Total download	388.4 GiB	366.3 GiB	346.5 GiB
Mean KL divergence vs release experts	0.141	0.183	0.229
Median KL divergence	0.023	0.033	0.046
Same top-1 token	88.5%	86.7%	85.1%
Perplexity vs reference	+5.2%	+10.4%	+14.1%
Depth ladder (needle + multi-hop, to 255K)	16/16	16/16	16/16
Decode, reference box (see below)	18.1-20.6 tok/s (16 at 255K)	20.4-22.9 tok/s (17 at 255K)	22.1-25.9 tok/s (19 at 255K)
Prefill, 32K-255K prompt	475-527 tok/s	529-592 tok/s	636-727 tok/s

• IQ3_XXS: pick this if your VRAM + RAM can hold ~194 GiB of weights plus context.
• IQ2_XS: pick this when you're ~20 GiB short. It costs 30% more KL divergence and ~1.8 points of top-1 agreement. On the same box it decodes ~10% faster and prefills ~15% faster, because fewer expert bytes sit in RAM.
• IQ2_XXS: pick this when you're ~40 GiB short of IQ3_XXS, or when speed matters more than fidelity. Against IQ3_XXS it costs 62% more KL divergence and 3.4 points of top-1 agreement; on the same box it decodes ~25% faster and prefills ~40% faster.

Shards -00008-, -00009- and -00010- (the two engram tables among them, 204.1 GiB together) are byte-identical in all three quants, and -00006- is also shared by IQ3_XXS and IQ2_XS (see SHA256SUMS). If you want more than one quant, download those once and copy or link them under the other quant's name.

Recipes

tensor group	IQ3_XXS	IQ2_XS	IQ2_XXS
Routed experts gate/up (80 tensors)	IQ3_XXS on 28 layers, IQ2_XS on 12	IQ2_XS	IQ2_XXS
Routed experts down (40)	IQ3_XXS	IQ3_XXS	Q2_K
Shared expert, attention, indexer/engram projections	Q8_0 (copied from the source, not re-quantized)	same	same
Router, mHC, compressor, norms, some indexer/engram parts	F32 / BF16, as in the source	same	same
token_embd, output	BF16	BF16	BF16
Engram tables (2 x 384M rows x 256)	Q8_0, see Engram precision	same	same

The routed experts are about 95% of the non-engram bytes, so their bit width decides quality.

Which 12 layers get IQ2_XS in the IQ3_XXS quant: 1, 12, 13, 20, 21, 27, 30, 31, 32, 33, 34, 35. We didn't pick them from imatrix energy, which on this family mostly tracks depth. Instead, a forward-pass probe measured, per layer, the size of the MoE output (absolute, and relative to the residual stream) on three kinds of text (general, reasoning traces, tool dialogues). The 12 layers that contribute least across all three were demoted. The three probes agree almost exactly. Layer 15 contributes about as much as the residual itself in every probe, so it stays at IQ3_XXS everywhere.

general.file_type reports Q8_0 because that was the quantizer's base type. The experts are per-tensor overrides, so trust the table above, not the Hub's type badge.

Engram precision

DeepSeek ships the engram tables in block-scaled FP8 (E4M3 values with one E8M0 scale per 32, i.e. MXFP8). GGUF has no FP8 storage type, so every GGUF of this model stores the engram in some other type. These files use Q8_0, taken from the source conversion. Sampling 16,384 rows against the release shows 0.55% relative RMS error, a worst-row cosine of 0.99998, and no element bit-identical.

Runtime

Use smalinin/llama.cpp, branch my_build. No patch is needed. The fork implements V4.1's two-level candidate prefilter, which is what makes context past 16,384 tokens faithful. It also carries upstream's fix for an out-of-bounds read in the CUDA routed-expert matmul (#29953); without that fix, CUDA builds can hit an Xid 31 fault. We haven't tested these files on other V4.1 runtimes (ik_llama.cpp, antirez's ds4).

git clone -b my_build https://github.com/smalinin/llama.cpp llama.cpp-dsv41
cd llama.cpp-dsv41 && git checkout 425badfe8   # the commit we tested
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

Which build the numbers come from. The quality, depth-ladder and speed results on this card were measured on smalinin's older my_build_deepseek41 branch at cfd8adcf6, plus a patch (still in this repo's history) that added the same MMQ fix and two changes these files don't use. We then re-tested IQ3_XXS on my_build at 425badfe8, on the same box with the same settings:

• Function: the shakedown (chat, thinking, tool calls, a tool round trip) and all 16 depth-ladder questions to 255K pass. GPU0 peaks at 92,552 MiB, about 100 MiB below the older build.
• Output: KL divergence between the two builds is 0.011 (same top-1 token 96.4%, 48 x 512 tokens of wikitext). That is what any rounding change does to this quant: on the older build alone, scoring 4 sequences per batch instead of 1 also gives 0.011, and so does switching the engram between Q8_0 and BF16. Identical runs repeat exactly (KL 0).
• Speed: within run-to-run variation. Short-prompt decode averaged 19.7 tok/s on both builds over two rounds in alternating order. Over the 16 ladder questions, run back to back on the same day, decode averaged 17.2 tok/s on my_build and 17.8 on the older build (-3%; per rung, from 0 to -13%), and prefill of the long prompts averaged 498 vs 477 tok/s.

The IQ2_XS and IQ2_XXS files weren't re-run on my_build.

Reference box and settings

RTX PRO 6000 Blackwell 96 GB (GPU0) + RTX 5070 Ti 16 GB (GPU1), Ryzen 9 9950X3D, 128 GB DDR5. The engram shards live on an NVMe RAID.

CUDA_VISIBLE_DEVICES=0,1 llama-server \
  --model DeepSeek-V4.1-Flash-1017-IQ3_XXS-00001-of-00010.gguf \
  --device CUDA0,CUDA1 --tensor-split 1,0 -ngl 999 \
  -ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.24\.ffn_down_exps\.weight=CPU' \
  -ncmoe 24 \
  --ctx-size 262144 -fa on -b 2048 -ub 2048 -np 1 --threads 12 \
  --jinja --temp 1.0 --top-p 1.0 --min-p 0.0

	GPU0	GPU1	RAM (CPU expert tier)	disk (engram)
Weights placed	78.7 GiB	8.1 GiB	105.8 GiB	194.6 GiB
Used after load, 256K ctx	92,474 MiB	10,154 MiB
Peak during the 255K rung	92,654 MiB (4.6 GiB free)

IQ2_XS on the same box: use -ncmoe 23 and drop the blk\.24\.ffn_down_exps override (-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1'). That places 77.3 GiB on GPU0 and 85.1 GiB in RAM; GPU0 holds 90,730 MiB after load and peaks at 90,940 MiB in the 255K rung.

IQ2_XXS on the same box: use -ncmoe 18 and send block 18's down-projection experts to RAM as well (-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.18\.ffn_down_exps\.weight=CPU'). That places 78.1 GiB on GPU0, 7.1 GiB on GPU1 and 65.5 GiB in RAM; GPU0 holds 91,002 MiB after load and peaks at 91,180 MiB in the 255K rung.

Settings that matter

• -np 1. We validated with one slot and haven't tested more. How sequences are batched changes the output slightly: on IQ3_XXS, scoring 4 sequences per batch instead of 1 moves it by KL divergence 0.011, the same as any rounding change, but on the release-precision master the same change gives ~0.1. To reproduce our KL numbers or imatrix, run one sequence per batch (-c = -b = -ub).
• Keep mmap on. No --load-mode none, no --mlock. The engram is read lazily from the file; loading it would need another ~195 GiB of RAM.
• --tensor-split 1,0 keeps every layer on GPU0 and moves only expert tensors to the second card. Splitting layers across cards disables the fused mHC ops.
• -ncmoe N keeps the experts of the first N layers in RAM. Raise it if GPU0 runs out of memory, lower it if RAM does. The compute buffer grows with context, so test at your full --ctx-size.
• Thinking is on by default. Turn it off per request with "chat_template_kwargs": {"enable_thinking": false}. Don't use {"thinking": false}, which renders a non-thinking prompt while the parser still expects a thinking block.
• Sampling: DeepSeek recommends temperature 1.0, top_p 1.0.
• The first long request after a load is slow. It pages the RAM expert tier back in from disk. After that, speeds are as below.

Chat template

The embedded template is a V4.1 template that reproduces all five of DeepSeek's official encoder test cases byte for byte, with one fix: top-level tools also render when the conversation starts with a user message. DeepSeek's reference encoder attaches tools to the system message, so in a conversation that starts with a user message they're silently dropped. No --chat-template-file is needed; the release files were tested with the embedded template.

Validation

Everything in this section was measured on the older my_build_deepseek41 build; Runtime has the re-test on my_build.

KL divergence. Reference = the source GGUF, whose routed experts are bit-identical to the release's MXFP4. Its trunk and engram are Q8_0. Text: wikitext-2 test, 40 chunks of 4,096 tokens, scoring the second half of each (81,920 tokens), one sequence per batch.

Depth ladder. Facts planted at 10/50/90% depth in a synthetic log, and four questions per rung: a verbatim needle, a single retrieval, a three-fact multi-hop sum, and a comparison. Thinking on, effort low. The IQ3_XXS rows were measured before the shard-1 metadata rewrite (same tensor bytes); the release files repeated the 32K rung, 4/4. The IQ2_XS and IQ2_XXS rows are the release files. Every rung scored 4/4 on all three quants.

prompt tokens	IQ3_XXS prefill / decode	IQ2_XS prefill / decode	IQ2_XXS prefill / decode
32,137	475 / 19.2-20.1 tok/s	592 / 21.7-22.9 tok/s	727 / 24.2-25.1 tok/s
64,141	527 / 19.2-20.6 tok/s	591 / 20.7-22.9 tok/s	720 / 23.9-25.9 tok/s
125,244	513 / 18.1-19.5 tok/s	570 / 20.4-21.5 tok/s	688 / 22.1-23.8 tok/s
255,340	479 / 15.6-16.1 tok/s	529 / 16.3-17.3 tok/s	636 / 18.8-19.4 tok/s

Functional checks: arithmetic, thinking open/close, tool calls with thinking on and off, and a tool-result round trip all pass on all three quants, using the release files and their embedded template. Short-prompt decode is 18.4-18.9 tok/s (IQ3_XXS), 21.4-22.4 tok/s (IQ2_XS) and 24.1-24.8 tok/s (IQ2_XXS).

Calibration

The imatrix (imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf) covers 950,272 tokens, computed as 116 single sequences of 8,192 tokens. All 46,080 routed-expert slots received activations. The corpus is 2.96M characters:

• general/scientific prose, plus an agentic, code and tool-use set
• source code: DeepSeek's reference inference and encoding code, and the llama.cpp runtime
• the documents behind pfeifferj's GSQ-RCO calibration: fineweb-edu, LLM compression calibration chats and OpenThoughts reasoning traces, with the chat and reasoning sets rendered in the V4.1 chat format
• Chinese Wikipedia (65 articles)
• DeepSeek's five official encoder test cases, and 48 dialogues rendered with the V4.1 template

Files

quant	files
IQ3_XXS	DeepSeek-V4.1-Flash-1017-IQ3_XXS-000{01..10}-of-00010.gguf
IQ2_XS	DeepSeek-V4.1-Flash-1017-IQ2_XS-000{01..10}-of-00010.gguf
IQ2_XXS	DeepSeek-V4.1-Flash-1017-IQ2_XXS-000{01..10}-of-00010.gguf
imatrix	imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf
checksums	SHA256SUMS (sha256sum -c SHA256SUMS --ignore-missing)

Shards 8 and 9 each hold one engram table (97.3 GiB apiece). Point llama.cpp at shard 1; it finds the rest. Downloading shards 8 and 9 with huggingface_hub / hf download needs Xet (hf_xet, the default); with HF_HUB_DISABLE_XET=1 set, the Hub refuses files this large.

Attribution

• Model: DeepSeek-AI, MIT License.
• Source conversion: mrfakename/DeepSeek-V4.1-Flash-1017-GGUF (Q8_0 trunk/engram, release MXFP4 experts unchanged).
• Runtime: smalinin's my_build (first validated on his my_build_deepseek41 with Nick Mermiges' MMQ padding fix). Upstream V4.1 support: vcruz305's #28696.
• Calibration document list: pfeifferj's GSQ-RCO manifest.