LLAMA-QUANTIZE(1)

📦
version 10438 · 2 versions
, in ubuntu stonking

💖 LLAMA-QUANTIZE(1) 💖

okay bestie, welcome to User Commands ✨ this is the group chat where your giant f32 model gets a whole new body. it's literally called llama-quantize and it's giving make it smol energy 🦋

🌸 NAME 🌸

llama-quantize — no cap, that's the whole name, that IS the vibe 💅

✨ DESCRIPTION ✨

here's how you call it, copy this exactly okay:

usage: llama-quantize [--help] [--allow-requantize]
[--leave-output-tensor] [--pure] [--imatrix] [--include-weights]

[--exclude-weights] [--output-tensor-type]
[--token-embedding-type] [--tensor-type] [--tensor-type-file]
[--prune-layers] [--keep-split] [--override-kv] [--dry-run]
model-f32.gguf [model-quant.gguf] type [nthreads]

so like… you hand it a model-f32.gguf, you may hand it a model-quant.gguf output name, you tell it a type, and may tell it nthreads. that's the whole plot 🎀

🌈 OPTIONS 🌈

fr fr read the warnings, bestie, I'm not being dramatic, the manpage is being dramatic and I'm just the messenger 💀
  • --help — shows you the help. classic icon behavior 🫶
  • --allow-requantize — lets you requantize tensors that have already been quantized.
WARNING: this can severely reduce quality compared to quantizing from 16bit or 32bit! like. severely severely. 🔥
  • --leave-output-tensor — leaves output.weight un(re)quantized. this makes the model bigger but might also make it better, especially when you're requantizing. main character tradeoff energy ✨
  • --pure — disables k-quant mixtures and quantizes all tensors to the same type. one type to rule them all, no blending, periodt 💅
  • --imatrix file_name — use the data in file_name as the importance matrix for quant optimizations. iykyk 🌸
  • --include-weights tensor_name — use the importance matrix for this/these tensor(s). 🎀
  • --exclude-weights tensor_name — do not use the importance matrix for this/these tensor(s). 🙈
  • --output-tensor-type ggml_type — use this ggml_type for the output.weight tensor. 👑
  • --token-embedding-type ggml_type — use this ggml_type for the token embeddings tensor. 🧚
  • --tensor-type tensor_name=ggml_type — quantize this tensor to this ggml_type. this is an advanced option to selectively quantize tensors. may be specified multiple times. example: --tensor-type attn_q=q8_0 ⭐
  • --tensor-type-file tensor_types.txt — a list of tensors to quantize to a specific ggml_type. this is an advanced option to selectively quantize a long list of tensors. the file should use the same format as above, separated by spaces or newlines. 📄
  • --prune-layers L0,L1,L2... — comma-separated list of layer numbers to prune from the model.
WARNING: this is an advanced option, use with care. 🫣
  • --keep-split — generate the quantized model in the same shards as input. no re-sharding drama, we keep the fit 💖
  • --override-kv KEY=TYPE:VALUE — override model metadata by key in the quantized model. may be specified multiple times.
WARNING: this is an advanced option, use with care. 😭
  • --dry-run — calculate and show the final quantization size without performing quantization. example: llama-quantize --dry-run model-f32.gguf Q4_K 🔍
note: --include-weights and --exclude-weights cannot be used together. pick a lane bestie 🚫

🦋 allowed quantization types 🦋

okay this is the menu, the whole menu, nothing skipped, let's go 👑

  • 40 or Q1_0 — 1.125 bpw quantization
  • 41 or Q2_0 — 2.25 bpw quantization (group 64)
  • 2 or Q4_0 — 4.34G, +0.4685 ppl @ Llama-3-8B
  • 3 or Q4_1 — 4.78G, +0.4511 ppl @ Llama-3-8B
  • 38 or MXFP4_MOE — MXFP4 MoE
  • 8 or Q5_0 — 5.21G, +0.1316 ppl @ Llama-3-8B
  • 9 or Q5_1 — 5.65G, +0.1062 ppl @ Llama-3-8B
  • 19 or IQ2_XXS — 2.06 bpw quantization
  • 20 or IQ2_XS — 2.31 bpw quantization
  • 28 or IQ2_S — 2.5 bpw quantization
  • 29 or IQ2_M — 2.7 bpw quantization
  • 24 or IQ1_S — 1.56 bpw quantization
  • 31 or IQ1_M — 1.75 bpw quantization
  • 36 or TQ1_0 — 1.69 bpw ternarization
  • 37 or TQ2_0 — 2.06 bpw ternarization
  • 10 or Q2_K — 2.96G, +3.5199 ppl @ Llama-3-8B
  • 21 or Q2_K_S — 2.96G, +3.1836 ppl @ Llama-3-8B
  • 23 or IQ3_XXS — 3.06 bpw quantization
  • 26 or IQ3_S — 3.44 bpw quantization
  • 27 or IQ3_M — 3.66 bpw quantization mix
  • 12 or Q3_K — alias for Q3_K_M
  • 22 or IQ3_XS — 3.3 bpw quantization
  • 11 or Q3_K_S — 3.41G, +1.6321 ppl @ Llama-3-8B
  • 12 or Q3_K_M — 3.74G, +0.6569 ppl @ Llama-3-8B
  • 13 or Q3_K_L — 4.03G, +0.5562 ppl @ Llama-3-8B
  • 25 or IQ4_NL — 4.50 bpw non-linear quantization
  • 30 or IQ4_XS — 4.25 bpw non-linear quantization
  • 15 or Q4_K — alias for Q4_K_M
  • 14 or Q4_K_S — 4.37G, +0.2689 ppl @ Llama-3-8B
  • 15 or Q4_K_M — 4.58G, +0.1754 ppl @ Llama-3-8B
  • 17 or Q5_K — alias for Q5_K_M
  • 16 or Q5_K_S — 5.21G, +0.1049 ppl @ Llama-3-8B
  • 17 or Q5_K_M — 5.33G, +0.0569 ppl @ Llama-3-8B
  • 18 or Q6_K — 6.14G, +0.0217 ppl @ Llama-3-8B
  • 7 or Q8_0 — 7.96G, +0.0026 ppl @ Llama-3-8B
  • 1 or F16 — 14.00G, +0.0020 ppl @ Mistral-7B
  • 32 or BF16 — 14.00G, -0.0050 ppl @ Mistral-7B
  • 0 or F32 — 26.00G @ 7B
  • COPY — only copy tensors, no quantizing

aaand that's the whole page bestie, every flag, every number, every warning, nothing left behind 💜

this lil manual lives under debian, dated September 2026, and goes by LLAMA-QUANTIZE(1) — mwah, go quantize something, you got this 👑✨

📦 where this page comes from