Instructions to use deepseek-ai/DeepSeek-V4.1-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4.1-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="deepseek-ai/DeepSeek-V4.1-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4.1-Flash", device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4.1-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4.1-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4.1-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4.1-Flash

SGLang

How to use deepseek-ai/DeepSeek-V4.1-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4.1-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4.1-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4.1-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4.1-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4.1-Flash with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4.1-Flash
```

Runs on one RTX 5090 (31.8 GiB) + 125.7 GiB RAM via a llama.cpp fork: GGUF, report and numbers

#37

by JigSawPT - opened Sep 11

Discussion

JigSawPT

Sep 11

•

edited Sep 11

If you want to run this model on a single consumer card: I ported it to a llama.cpp fork and measured what limits it. It runs exactly — logits within the reference implementation's own fp8 rounding floor — at 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache, on an RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM and a PCIe 5 NVMe.

How. The 269 GiB of routed experts stream from disk through a VRAM cache (18 GiB) and a pinned RAM tier (72 GiB); the 189 GiB engram tables stay memory-mapped on disk and are read by the host. Nothing fits in RAM + VRAM and nothing has to. The routed experts are a lossless repack of the released MXFP4 blocks (verified byte for byte), attention and dense weights are dequantized from fp8, and the engram tables travel as their raw fp8 bytes.

Where the token goes: 20 % compute, 26 % PCIe, 54 % NVMe. The disk is at 43 % utilisation; the limit is that only ~3.8 disk requests exist in flight per layer, because a layer's requests can only leave after its router has run. Ceiling with zero disk misses: 6.2 tokens/s; with everything resident, 21.

What didn't pay (measured, not guessed): a prefetch oracle with perfect knowledge of future routing gives +30 % but only 5 tokens ahead — at the depths a layer predictor can reach it loses; an n-gram draft is −33 %; the model's own DSpark draft head hits 51–97 % acceptance and is still neutral, because on a disk-bound machine a verification step pays for the union of experts of the block; encoder-only prefill isn't exact below 2 432 tokens. Also measured flat: more VRAM cache (13 → 22 GiB), a larger RAM tier (72 → 88 GiB, actually slower), and a smarter eviction policy (LRU against the optimal Belady policy: 0 % margin). A faster NVMe I did not test; the disk sits at 43 % utilisation, so I don't expect it to help.

One real defect found on the way: V4 normalises each attention head of q after wq_b; V4.1 does not. The inherited line never raised an error (norm 181.019 = √(64×512) gave it away).

GGUF with the engram tables (502 GB, 11 shards): https://huggingface.co/JigSawPT/DeepSeek-V4.1-Flash-GGUF
Draft head GGUF (8 GB): https://huggingface.co/JigSawPT/DeepSeek-V4.1-Flash-DSpark-GGUF
Fork branch: https://github.com/JigSawPT/llama.cpp/tree/dsv41-porte
Report, tools, raw results (MIT, DOI 10.5281/zenodo.22711369): https://github.com/JigSawPT/deepseek-v41-flash-on-5090
llama.cpp discussion: https://github.com/ggml-org/llama.cpp/discussions/28766

Command:

llama-server -m DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 --moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --reasoning off

Caveats: the GGUF runs only on the fork (upstream llama.cpp has no V4.1 runtime yet, and its open conversion PR stores the engram differently); Linux is untested on this branch; build instructions (Windows, CUDA 13.0) are in the branch README. Thanks to DeepSeek for the model and the reference implementation, which made the exactness checks possible.

krzysztofma

Sep 14

Thanks!
Regarding: "20 % compute, 26 % PCIe, 54 % NVMe."
What does it mean exactly ? 26% of the time is spent on weights transfer to the GPU via PCIE-5.0 x16 ?

"Ceiling with zero disk misses: 6.2 tokens/s"
So i.e. You mean with 100% memory misses?

If 54% is NVMe reads , then it’s the bottleneck, right? What if engram tables would be stored on a separate NVMe or model layers split between two or more disks?

konstantinnikol

Sep 16

Thanks!
Regarding: "20 % compute, 26 % PCIe, 54 % NVMe."
What does it mean exactly ? 26% of the time is spent on weights transfer to the GPU via PCIE-5.0 x16 ?

"Ceiling with zero disk misses: 6.2 tokens/s"
So i.e. You mean with 100% memory misses?

If 54% is NVMe reads , then it’s the bottleneck, right? What if engram tables would be stored on a separate NVMe or model layers split between two or more disks?

Engram stored on separate NVME does not give more than 2% gain in token speed based on my tests. But extra NVME help. You can see test results using several NVME here https://github.com/argonautlabsai/argodrive

krzysztofma

Sep 21

•

edited Sep 21

Engram stored on separate NVME does not give more than 2% gain in token speed based on my tests. But extra NVME help. You can see test results using several NVME here https://github.com/argonautlabsai/argodrive

Thanks @konstantinnikol for the link!. Very interesting article and repo . So it looks like offloading n-gram tables to separate SSD won't help much , but streaming the expert weight's replicas from few fast SSDs can almost double the performance. For DS V4.1 it was slightly less than 1.5x , but with the fork - it's more than double. I'm not sure it the commit applies only to Metal (the commit message suggests that) or CUDA too:

https://github.com/argonautlabsai/ds4-argodrive/commit/18fc7953c53d423606ac1c366dc82c0141cbd0e0

JigSawPT

Sep 23

Sorry for the slow answer, and thanks @konstantinnikol for stepping in.

20 % / 26 % / 54 %. This splits the stall per remap call. Once per layer per token, the six experts that layer chose are brought into the VRAM cache, and the graph waits for that. We swept the pinned RAM tier from 0 to 72 GiB with the VRAM cache fixed (its hit rate stayed at 62.74 %). That separates the stall into three parts:
- compute (20 %);
- experts that missed VRAM but were in pinned RAM and crossed PCIe 5.0 x16 (26 %);
- experts read from the NVMe (54 %).
So yes, the 26 % is host-to-GPU transfer.
6.2 tokens/s with zero disk misses means every expert missing from VRAM is found in the pinned RAM tier, so nothing is read from disk. It is not 100 % misses. A prefetch oracle with perfect knowledge of the next tokens got within 2 % of it. With everything resident in VRAM, the measured ceiling is 21.3 tokens/s.
Is the NVMe the bottleneck? It is the biggest share, but the drive was not saturated. It delivers 10.04 GB/s at the queue depth that decode produces, and the port used 4.33 GB/s (43 %). Only about 3.8 reads are in flight per layer, because a layer's reads can only be issued after that layer's router has run. An I/O-thread sweep plateaued at four workers.
Engram on a separate NVMe. The engram row addresses are known from the token ids, so we prefetch them. After that, the engram costs about 0.9 ms per token, a fraction of a percent of a token at 5 tokens/s, so a separate drive has little to gain. That is consistent with the ≤2 % above.
Layers split across drives: not tested. With about 3.8 reads in flight, the extra bandwidth would mostly go unused on the read path the report measured.

Details are in the report's sections "Speed: where the token goes" and "The deficit is in the graph, not the queue": https://github.com/JigSawPT/deepseek-v41-flash-on-5090

Update since the report: on the same machine, --moe-stream-direct (O_DIRECT for the expert reads) doubled decode on the card's command line whenever experts come from disk (3.9-4.9 -> 8.9-10.0 tokens/s on the first run of four prompts; resident decode unchanged at 19-22). So the buffered read path, not the drive, was part of that limit, and more drives may matter once it is bypassed. We have not measured that yet.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment