Qwen3.5-9B-NVFP4-GGUF

Qwen3.5-9B from Alibaba's Qwen team is a 9B-parameter dense multimodal language model featuring a hybrid Gated DeltaNet + Gated Attention architecture with 262K native context window (extensible to 1M+ tokens via RoPE scaling), 248K vocabulary supporting 201 languages, and early‑fusion training for unified text, image, and video understanding. It achieves SOTA performance across modalities with 89.2% OCRBench, 84.5% VideoMME, 78.9% MathVision, and 70.1% MMMU-Pro, while delivering production-level agentic capabilities including 66.1% BFCL-V4 and 79.1% TAU2-Bench for native tool calling, plus toggleable thinking mode for step-by-step reasoning on complex tasks. Apache 2.0-licensed and optimized for vLLM/SGLang/llama.cpp/Ollama deployment (~18GB VRAM BF16, ~5GB 4-bit), the instruction-tuned variant excels at repository-level coding, frontend development, document/PDF parsing, visual question answering, and multilingual chatbots as a scalable foundation for edge-to-server multimodal agents.

MTP — Speculative Decoding: Yes

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Qwen3.5-9B-NVFP4.gguf	NVFP4	8.46 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 9B parameter model.
Qwen3.5-9B.mmproj.gguf	mmproj	922 MB	Link	Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.5-9B-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric	Value
Average Prompt Processing Speed	258.57 tokens/s
Average Generation Speed	217.83 tokens/s
Total Wall Time	142.9 seconds
Context Length	16,384 tokens
KV Cache (K)	bf16
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity	96 GB
Peak GPU VRAM Usage	7.69 GiB
Peak GPU Utilization	90%
GPU Layers Offloaded	99 (-ngl 99)
Random Seed	42
Speculative Decoding	MTP (draft_n_max=2)
Test Runs	5

Test Shots

Shot	Test
Shot 1	C: Insert Node at End of Singly Linked List
Shot 2	Jogger and Train Passing Time
Shot 3	Two Pipes Filling a Tank
Shot 4	Students in Examination Rooms
Shot 5	Father and Son Ages

Run Inference

./build/bin/llama-cli \
  -m /path/to/models/Qwen3.5-9B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k bf16 \
  --seed 42 \
  -n 8192 \
  -st \
  -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

Snapshots (5)

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods.