Quantization

• Hub-first quant selection
• Quantization Formats
• Converting Models
• K-Quantization Methods
• Quality Testing
• Use Case Guide
• Model Size Scaling
• Finding Pre-Quantized Models
• Importance Matrices (imatrix)
• Troubleshooting

Hub-first quant selection

Before using generic tables, open the model repo with:

Prefer the exact quant labels and sizes shown in the Hardware compatibility section of the fetched ?local-app=llama.cpp page text or HTML. Then confirm the matching filenames in:

Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.

Quantization Formats

GGUF (GPT-Generated Unified Format) - Standard format for llama.cpp models.

Format Comparison

Recommendation: Use Q4_K_M for best balance of quality and speed.

Converting Models

Hugging Face to GGUF

Batch quantization

K-Quantization Methods

K-quants use mixed precision for better quality:

• Attention weights: Higher precision
• Feed-forward weights: Lower precision

Variants:

• _S (Small): Faster, lower quality
• _M (Medium): Balanced (recommended)
• _L (Large): Better quality, larger size

Example: Q4_K_M

• Q4: 4-bit quantization
• K: Mixed precision method
• M: Medium quality

Quality Testing

Use Case Guide

General purpose (chatbots, assistants)

Code generation

Creative writing

Technical/medical

Edge devices (Raspberry Pi)

Model Size Scaling

7B parameter models

13B parameter models

70B parameter models

Recommendation for 70B: Use Q3_K_M or Q4_K_S to fit in consumer hardware.

Finding Pre-Quantized Models

Use the Hub search with the llama.cpp app filter:

For a specific repo, open:

Then launch directly from the Hub without extra Hub tooling:

If you need the exact file name from the tree API:

Importance Matrices (imatrix)

What: Calibration data to improve quantization quality.

Benefits:

• 10-20% perplexity improvement with Q4
• Essential for Q3 and below

Usage:

Calibration data:

• Use domain-specific text (e.g., code for code models)
• ~100MB of representative text
• Higher quality data = better quantization

Troubleshooting

Model outputs gibberish:

• Quantization too aggressive (Q2_K)
• Try Q4_K_M or Q5_K_M
• Verify model converted correctly

Out of memory:

• Use lower quantization (Q4_K_S instead of Q5_K_M)
• Offload fewer layers to GPU (-ngl)
• Use smaller context (-c 2048)

Slow inference:

• Higher quantization uses more compute
• Q8_0 much slower than Q4_K_M
• Consider speed vs quality trade-off