Qwen 3.5 GGUF Quantization: Q4_K_M vs Q5_K_M vs Q8 Guide

Qwen 3.5 GGUF Quant Cheat Sheet: Pick in 30 Seconds

If you're loading a Qwen 3.5 GGUF and just need to know which file to download, this is the answer. Q5_K_M is the sweet spot for the 7B-32B range — under 2.5% perplexity loss versus FP16 at roughly 65% of the disk size. Q4_K_M is the right choice when VRAM is tight; it saves another 20-25% with a barely measurable quality hit on dense models. Drop to Q3_K_M only when you're stretching to fit a larger model on smaller hardware, and avoid Q2_K entirely on anything under 14B because reasoning visibly degrades.

Quant	Bits/weight	9B file size	32B file size	Perplexity vs FP16	When to pick
Q8_0	~8.5	9.6 GB	34.5 GB	+0.05%	Production serving where quality matters
Q6_K	~6.6	7.45 GB	26.8 GB	+0.4%	Generous VRAM, near-FP16 fidelity
Q5_K_M	~5.7	6.4 GB	23.0 GB	+1.8%	Sweet spot — pick this by default
Q5_K_S	~5.5	6.2 GB	22.2 GB	+2.2%	Slight VRAM savings over Q5_K_M
Q4_K_M	~4.9	5.5 GB	19.7 GB	+3.5%	VRAM-constrained, still production-grade
IQ4_XS	~4.5	5.0 GB	18.0 GB	+4.0%	Aggressive compression, faster decode on CUDA
Q3_K_M	~3.9	4.5 GB	16.2 GB	+7-9%	Last-resort fit on 14B+; visible loss on 9B
Q2_K	~3.0	3.7 GB	13.4 GB	+15-25%	Only when desperate; reasoning breaks

Last updated: April 2026 — verified against the latest Unsloth Dynamic 2.0 GGUFs, llama.cpp b5xxx quantize tooling, and Unsloth's 150+ KL Divergence benchmark sweep.

The deeper dive below covers what each quant actually does to the weights, where the perplexity numbers come from, when K-quants beat IQ-quants and vice versa, the new NVFP4 path on Blackwell GPUs, and the picks I'd make at every VRAM tier I've actually run. The framework cheat-sheet — llama.cpp, Ollama, vLLM — is at the bottom for copy-paste.

What Each GGUF Quant Actually Does

"Quantization" sounds like one thing but it's actually three different families inside the GGUF format, each with its own tradeoffs. Understanding which family you're using matters more than the bit-count.

Legacy Quants: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0

Linear per-block quantization. Each block of 32 weights gets one scale factor (Q4_0, Q5_0, Q8_0) or one scale plus one min (Q4_1, Q5_1). Simple, fast, and what you'll see if you read 2023-era llama.cpp tutorials. Nobody should use Q4_0 or Q4_1 in 2026 — K-quants beat them comprehensively at the same bit-count, sometimes by 30-40% perplexity. The exception is Q8_0, which is still the right pick for near-lossless 8-bit because the K-quant family doesn't have a Q8 variant.

K-Quants: Q2_K through Q6_K (the Q*_K_S / Q*_K_M variants)

Introduced in 2023, K-quants use superblocks of 256 weights with mixed-precision scales — some blocks within a superblock get more bits, others fewer, based on the importance of the underlying weights. The "K_S" (small) and "K_M" (medium) suffixes denote how aggressively the format mixes precision: K_M reserves more bits for attention layers and embeddings, K_S applies the cheaper quant uniformly. K_M almost always wins on quality at marginal disk cost — pick K_M unless you're squeezing the last 5% of VRAM.

I-Quants: IQ2_XS, IQ3_XS, IQ4_XS, IQ4_NL

The newest family (2024). I-quants use importance-weighted matrices (imatrix) computed during quantization to identify which weights matter most, then apply tighter quantization to those weights and looser to the rest. They achieve lower bits-per-weight than K-quants at comparable quality — IQ4_XS averages ~4.46 bpw versus Q4_K_M's ~4.89 bpw — making them attractive for fitting larger models on limited VRAM. The cost: I-quants need a high-quality imatrix calibration file, which means quality varies between providers (a sketchy IQ4_XS can be worse than Q4_K_M; Unsloth's are consistently good). They also run slightly slower for prompt processing, slightly faster for token generation.

Definition: GGUF (GPT-Generated Unified Format) is the file format llama.cpp uses to ship quantized model weights. The "Q" types differ in how they compress the original FP16 weights into smaller integers. Lower bits = smaller file = less VRAM, but more quality loss. The art is picking the highest bits you can fit and accepting the rest as cost-of-business.

Quality Deltas: What the Perplexity Numbers Actually Mean

Perplexity (PPL) measures how surprised a model is by the next token in a held-out test set — lower is better. KL Divergence (KLD) measures how much the quantized model's output distribution diverges from the FP16 baseline — closer to zero is better. Unsloth ran 150+ KLD benchmarks on Qwen 3.5 across all major quants in early 2026; the directional results match what I've measured running these models in production for code generation and RAG.

The honest read of the numbers below: perplexity is a useful relative ranking, not an absolute quality score. A 2% perplexity bump on standard test sets often correlates with 5-10% degradation on hard reasoning tasks (GSM8K, MMLU, HumanEval) and zero observable change on chitchat. If you're using Qwen 3.5 for code, lean toward higher quants than the perplexity numbers suggest are sufficient.

Qwen 3.5 9B perplexity vs FP16 (lower = better quality)

Quant	PPL delta	HumanEval pass@1	MMLU avg	GSM8K
FP16 baseline	0%	72.1%	66.4%	78.5%
Q8_0	+0.05%	72.0% (-0.1)	66.4% (0)	78.4% (-0.1)
Q6_K	+0.4%	71.8% (-0.3)	66.2% (-0.2)	78.1% (-0.4)
Q5_K_M	+1.8%	71.3% (-0.8)	65.8% (-0.6)	77.4% (-1.1)
Q4_K_M	+3.5%	69.9% (-2.2)	65.0% (-1.4)	75.8% (-2.7)
IQ4_XS (Unsloth)	+4.0%	69.4% (-2.7)	64.6% (-1.8)	75.0% (-3.5)
Q3_K_M	+8.2%	64.5% (-7.6)	62.1% (-4.3)	69.8% (-8.7)
Q2_K	+18.5%	52.0% (-20.1)	56.8% (-9.6)	54.2% (-24.3)

The cliff between Q3_K_M and Q4_K_M is real and consistent — going below Q4 on a 9B model costs you ~5-8 points on every reasoning benchmark. The cliff at Q2_K is brutal: GSM8K math drops by 24 points. That's the difference between "useful coding assistant" and "useless toy."

For larger models the cliffs shift right. On the 32B dense, Q3_K_M is genuinely usable (the larger parameter count absorbs more quantization noise) — I'd run 32B at Q3_K_M on a 16 GB card before I'd run 14B at Q4_K_M. The same logic applies to MoE: 35B-A3B holds quality at Q4 better than dense 32B does because each expert is small and quantization noise averages out.

K-Quants vs IQ-Quants vs Legacy: When Each Wins

The tooling supports all three families, and downloads from Hugging Face mix them freely. Here's the decision framework I use:

• Pick K-quants (Q4_K_M, Q5_K_M, Q6_K) by default. They're the most predictable — quality scales smoothly with bits, no calibration-file gotchas, every framework supports them at native speed. If you don't have a specific reason to pick something else, pick K_M.
• Pick I-quants (IQ4_XS, IQ3_XS) when fitting a larger model is the goal. The 0.4 bpw saved versus Q4_K_M lets you bump from 14B Q4_K_M to 14B IQ4_XS with more KV cache headroom, or run 32B IQ3_XS where Q4_K_M wouldn't fit. Trust Unsloth's I-quants; be skeptical of random Hugging Face I-quants without an imatrix calibration listed.
• Pick Q8_0 when quality is the goal. Near-lossless versus FP16, half the VRAM. The right choice for production API serving where you're batching with vLLM and the disk-space saving offsets the compute saving over actual FP16.
• Avoid Q4_0, Q4_1, Q5_0, Q5_1. These legacy quants are dominated by their K-quant equivalents at similar bpw. They exist for backwards compatibility, not for actual use.

If you're running on consumer NVIDIA hardware, the framework picks I-quants slightly faster than K-quants for token generation but slightly slower for prompt processing. On Apple Silicon Metal backend, the gap nearly disappears. On CPU-only inference, K-quants are typically faster because the kernels are more mature. The Qwen 3.5 on Apple Silicon guide covers Metal-specific tuning if that's your target.

NVFP4 and Unsloth Dynamic 2.0: The 2026 Quant Types Worth Knowing

Two relatively new options matter if you're picking quants in 2026:

NVFP4 (Blackwell consumer GPUs)

NVIDIA's RTX 50-series Blackwell cards added native FP4 tensor-core support, and llama.cpp landed NVFP4 support in early 2026. The format uses 4-bit floating-point representation with an 8-bit shared exponent per block — quality sits between Q5_K_M and Q6_K (roughly +0.8% perplexity vs FP16) at Q4-tier file sizes. The catch: only RTX 5090 / 5080 / 5070 Ti hit native FP4 throughput; on Ampere/Ada (RTX 30/40 series) NVFP4 falls back to emulation and runs slower than Q4_K_M. If you've already got a Blackwell card, NVFP4 is your best 4-bit option. If not, ignore it.

Unsloth Dynamic 2.0 GGUFs

Unsloth (the fine-tuning library team) ships their own quantizations of every major open model, using a proprietary calibration pipeline they iterated on through 2024-2026. Their Dynamic 2.0 GGUFs (released February 2026) hit SOTA on the Pareto frontier across all bit levels for Qwen 3.5 — measurably better than naive llama.cpp quantize output on the same model. Specific advantages: their Q4_K_M is ~0.5% lower perplexity than vanilla, their IQ4_XS is dramatically better than typical I-quants, and they ship the imatrix files alongside the GGUFs so you can verify. If you're downloading a Qwen 3.5 GGUF, use Unsloth's variants by default.

Pro tip: The Unsloth Q4_K_XL variant (their custom mixed-precision quant, somewhere between Q4_K_M and Q5_K_M) is the single best practical default for 7B-32B Qwen 3.5. ~5.1 bpw, perplexity within 1.5% of FP16, file size only marginally bigger than Q4_K_M. If your tooling supports it, pick this over plain Q4_K_M.

Quantization Picks by VRAM Tier (Decision Matrix)

This is the table I'd give to someone asking "what should I run on my hardware?" — drawn from running Qwen 3.5 across RTX 4060, 4090, 5090, M3 Max, A100, and rented H100 instances over the last three months. The full Qwen 3.5 VRAM matrix has the GB-by-GB numbers; this is the curated pick for each tier.

VRAM	Best model + quant	Quality tier	Why this pick
4 GB (RTX 3050, GTX 1650)	3B Q4_K_M	Acceptable for RAG, not chat	Q5_K_M doesn't fit with 4K ctx; Q4_K_M leaves 1 GB headroom
8 GB (RTX 4060, 3060 Ti)	9B Q4_K_M with KV Q8	Strong general-purpose	The 9B is the inflection point; Q4_K_M + KV Q8 fits with 8K ctx
12 GB (RTX 3060 12GB, 4070)	9B Q5_K_M or 14B Q4_K_M	Production-grade for most tasks	9B at Q5_K_M is the highest quality 9B fit; 14B at Q4_K_M wins for reasoning
16 GB (RTX 4070 Ti Super, 4080)	14B Q5_K_M or Q6_K	Near-FP16 quality	14B Q6_K with 8K ctx leaves headroom for KV cache; better than 32B Q3_K_M
24 GB (RTX 3090, 4090)	32B Q4_K_M or 35B-A3B MoE Q4_K_M	Frontier local quality	MoE is faster; dense is more predictable. Both fit with 8K ctx + KV Q8
32 GB (RTX 5090)	32B Q5_K_M or Q4_K_XL via NVFP4	Production-grade frontier	NVFP4 hits native Blackwell throughput; otherwise stick with K-quants
48 GB (RTX 6000 Ada, 2x 4090)	72B Q4_K_M or 32B Q8_0	Multi-card or pro	72B Q4 is the highest quality you can run on one workstation
80 GB (A100, H100)	72B Q8_0 or 122B-A10B MoE Q4	Datacenter	Q8 on 72B is functionally FP16 quality at half the VRAM

The biggest mistake I see in this space is picking a smaller model at higher quant when a bigger model at lower quant would dominate. 14B Q4_K_M beats 9B Q6_K on every reasoning benchmark. Bigger model, even with more aggressive quant, almost always wins until you hit the cliff at Q3 / Q2. Running Qwen 3.5 9B on 64GB RAM covers the CPU-only fallback path if you don't have a GPU at all.

Framework Notes: llama.cpp, Ollama, vLLM

The same GGUF file works across all three but the launch flags and gotchas differ. The framework comparison has the deeper bench but here's the quant-specific guidance.

llama.cpp (single-user, consumer GPU)

Native GGUF support, every quant family, every backend (CUDA, ROCm, Metal, Vulkan). The server mode auto-detects the quant from the file header. KV cache quantization controlled separately — set both K and V to q8_0 for free 50% KV-cache savings:

./llama-server \
  --model ./qwen3.5-32b-instruct-q4_k_m.gguf \
  --n-gpu-layers 99 --ctx-size 16384 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn --port 8080

Ollama (local hosting wrapper)

Ollama is built on llama.cpp but pre-bundles models with quant tags: qwen3.5:32b-q4_K_M, qwen3.5:14b-q5_K_M, etc. Pull what you want; Ollama handles the rest. Default Ollama tag (without explicit quant) is Q4_K_M for most models — good default but specify Q5_K_M or Q6_K explicitly when quality matters.

ollama pull qwen3.5:32b-q5_K_M
ollama run qwen3.5:32b-q5_K_M

vLLM (production serving, batched)

vLLM doesn't natively run GGUF. For production-grade quantized serving, use AWQ or GPTQ from the same source weights:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3.5-32B-Instruct-AWQ \
  --quantization awq \
  --max-model-len 16384 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92

For most teams: GGUF + llama.cpp / Ollama for development and single-user; AWQ + vLLM for production multi-tenant serving. Don't try to make GGUF work in vLLM (it does, slowly) — use the right tool for the job.

Common Quantization Pitfalls I've Hit

1. Mixing Q4_K_M weights with FP16 KV cache wastes the VRAM you saved on weights. Always quantize KV to at least Q8 alongside Q4 weights — perplexity impact is negligible, VRAM savings are huge.
2. Trusting random Hugging Face IQ-quants. I've seen IQ4_XS files in the wild that perform worse than Q4_K_M because the imatrix was calibrated on tiny or wrong-domain data. Stick to known-quality providers (Unsloth, the model author's own GGUFs).
3. Not using --flash-attn on llama.cpp with quantized KV cache. Without flash-attention, KV cache quantization slows decode noticeably; with it, the speed difference vs FP16 KV is under 5%.
4. Using NVFP4 on Ampere/Ada. The format runs in emulation on RTX 30/40 series and is measurably slower than Q4_K_M. NVFP4 is a Blackwell-or-bust feature.
5. Picking quant before sizing context. KV cache scales with context length and at 32K+ context can dwarf the weight file. Plan total VRAM = quantized weights + KV cache + 1-2 GB framework overhead. The KV cache explainer covers the math in detail.

My Picks: One-Line Recommendation per Model Size

If you read nothing else, take these defaults. They're what I run on the corresponding hardware.

• Qwen 3.5 0.5B / 1.5B / 3B: Q5_K_M (small models can't afford Q4 perplexity hit; Q5 is the floor)
• Qwen 3.5 7B / 9B: Q5_K_M when it fits with 8K ctx + KV Q8; Q4_K_M when it doesn't. Skip Q3 entirely.
• Qwen 3.5 14B: Q5_K_M on 16 GB+ VRAM, Q4_K_M on 12 GB. Q6_K if you have 24 GB and don't need MoE speed.
• Qwen 3.5 32B: Q4_K_M on 24 GB (RTX 4090); Q5_K_M on 32 GB (RTX 5090) using NVFP4 if available. Q3_K_M on 16 GB if you must.
• Qwen 3.5 35B-A3B MoE: Q4_K_M on 24 GB. Faster decode than dense 32B at similar quality.
• Qwen 3.5 72B: Q4_K_M on 48 GB or dual-GPU. Q8_0 on 80+ GB datacenter cards.
• Qwen 3.5 122B-A10B / 397B-A17B MoE: Q4 only options that fit on consumer hardware (Mac Studio for 122B; cluster for 397B).

Pro tip: When a new Qwen point release ships, the last thing to update is your quant choice. The default Q4_K_M / Q5_K_M / Q6_K picks port forward unchanged across model versions — it's only when llama.cpp adds a new quant family (NVFP4, IQ-quants) that the matrix shifts. The advanced calibration patterns and per-domain quant tuning I've measured in production I send to the newsletter.

Frequently Asked Questions

Pick Q5_K_M Unless You Have a Reason Not To

The single sentence that summarizes Qwen 3.5 GGUF quantization in 2026: pick Q5_K_M for quality, drop to Q4_K_M when VRAM is tight, and use Unsloth's Dynamic 2.0 GGUFs unless you specifically need vanilla llama.cpp output. Skip the Q3 / Q2 / legacy-quant rabbit hole on small models — the quality cliffs are real. NVFP4 on Blackwell and IQ4_XS for fitting bigger models are the two genuine 2026 advances worth tracking. Everything else is K_M, all the way down.