Gguf Conversion

What is GGUF?

GGUF (GPT-Generated Unified Format):

• Optimized format for CPU/GPU inference with llama.cpp
• Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size
• Compatible with: Ollama, LM Studio, Jan, GPT4All, llama.cpp
• Typically 2-8GB for 7B models (vs 14GB unquantized)

When to Convert to GGUF

Convert when:

• Running models locally with Ollama or LM Studio
• Using CPU-optimized inference
• Reducing model size with quantization
• Deploying to edge devices
• Sharing models for local-first use

Critical Success Factors

Based on production testing, these are essential for reliable conversion:

1. ✅ Install Build Tools FIRST

Before cloning llama.cpp, install build dependencies:

Why: The quantization tool requires gcc and cmake. Installing after cloning doesn’t help.

2. ✅ Use CMake (Not Make)

Build the quantize tool with CMake:

Why: CMake is more reliable than make and produces consistent binary paths.

3. ✅ Include All Dependencies

PEP 723 header must include:

Why: sentencepiece and protobuf are critical for tokenizer conversion. Missing them causes silent failures.

4. ✅ Verify Names Before Use

Always verify repos exist:

Why: Non-existent dataset/model names cause job failures that could be caught in seconds.

Complete Conversion Script

See scripts/convert_to_gguf.py for the complete, production-ready script.

Key features:

• ✅ All dependencies in PEP 723 header
• ✅ Build tools installed automatically
• ✅ CMake build process (reliable)
• ✅ Comprehensive error handling
• ✅ Environment variable configuration
• ✅ Automatic README generation

Quick Conversion Job

Conversion Process

The script performs these steps:

1. Load and Merge - Load base model and LoRA adapter, merge them
2. Install Build Tools - Install gcc, cmake (CRITICAL: before cloning llama.cpp)
3. Setup llama.cpp - Clone repo, install Python dependencies
4. Convert to GGUF - Create FP16 GGUF using llama.cpp converter
5. Build Quantize Tool - Use CMake to build llama-quantize
6. Quantize - Create Q4_K_M, Q5_K_M, Q8_0 versions
7. Upload - Upload all versions + README to Hub

Quantization Options

Common quantization formats (from smallest to largest):

Recommendation: Create Q4_K_M, Q5_K_M, and Q8_0 versions to give users options.

Hardware Requirements

For conversion:

• Small models (<1B): CPU-basic works, but slow
• Medium models (1-7B): a10g-large recommended
• Large models (7B+): a10g-large or a100-large

Time estimates:

• 0.5B model: ~15-25 minutes on A10G
• 3B model: ~30-45 minutes on A10G
• 7B model: ~45-60 minutes on A10G

Using GGUF Models

GGUF models work on both CPU and GPU. They’re optimized for CPU inference but can also leverage GPU acceleration when available.

With Ollama (auto-detects GPU)

With llama.cpp

With LM Studio

1. Download the .gguf file
2. Import into LM Studio
3. Start chatting

Best Practices

✅ DO:

1. Verify repos exist before submitting jobs (use hub_repo_details)
2. Install build tools FIRST before cloning llama.cpp
3. Use CMake for building quantize tool (not make)
4. Include all dependencies in PEP 723 header (especially sentencepiece, protobuf)
5. Create multiple quantizations - Give users choice
6. Test on known models before production use
7. Use A10G GPU for faster conversion

❌ DON’T:

1. Assume repos exist - Always verify with hub tools
2. Use make instead of CMake - Less reliable
3. Remove dependencies to “simplify” - They’re all needed
4. Skip build tools - Quantization will fail silently
5. Use default paths - CMake puts binaries in build/bin/

Common Issues

Out of memory during merge

Fix:

• Use larger GPU (a10g-large or a100-large)
• Ensure device_map="auto" for automatic placement
• Use dtype=torch.float16 or torch.bfloat16

Conversion fails with architecture error

Fix:

• Ensure llama.cpp supports the model architecture
• Check for standard architecture (Qwen, Llama, Mistral, etc.)
• Update llama.cpp to latest: git clone --depth 1 https://github.com/ggerganov/llama.cpp.git
• Check llama.cpp documentation for model support

Quantization fails

Fix:

• Verify build tools installed: apt-get install build-essential cmake
• Use CMake (not make) to build quantize tool
• Check binary path: /tmp/llama.cpp/build/bin/llama-quantize
• Verify FP16 GGUF exists before quantizing

Missing sentencepiece error

Fix:

• Add to PEP 723 header: "sentencepiece>=0.1.99", "protobuf>=3.20.0"
• Don’t remove dependencies to “simplify” - all are required

Upload fails or times out

Fix:

• Large models (>2GB) need longer timeout: "timeout": "1h"
• Upload quantized versions separately if needed
• Check network/Hub status

Lessons Learned

These are from production testing and real failures:

1. Always Verify Before Use

Lesson: Don’t assume repos/datasets exist. Check first.

Prevented failures: Non-existent dataset names, typos in model names

2. Prioritize Reliability Over Performance

Lesson: Default to what’s most likely to succeed.

• Use CMake (not make) - more reliable
• Disable CUDA in build - faster, not needed
• Include all dependencies - don’t “simplify”

Prevented failures: Build failures, missing binaries

3. Create Atomic, Self-Contained Scripts

Lesson: Don’t remove dependencies or steps. Scripts should work as a unit.

• All dependencies in PEP 723 header
• All build steps included
• Clear error messages

Prevented failures: Missing tokenizer libraries, build tool failures

References

In this skill:

• scripts/convert_to_gguf.py - Complete, production-ready script

External:

• llama.cpp Repository (opens in a new tab)
• GGUF Specification (opens in a new tab)
• Ollama Documentation (opens in a new tab)
• LM Studio (opens in a new tab)

Summary

Critical checklist for GGUF conversion:

• Not done: Verify adapter and base models exist on Hub
• Not done: Use production script from scripts/convert_to_gguf.py
• Not done: All dependencies in PEP 723 header (including sentencepiece, protobuf)
• Not done: Build tools installed before cloning llama.cpp
• Not done: CMake used for building quantize tool (not make)
• Not done: Correct binary path: /tmp/llama.cpp/build/bin/llama-quantize
• Not done: A10G GPU selected for reasonable conversion time
• Not done: Timeout set to 45m minimum
• Not done: HF_TOKEN in secrets for Hub upload

The script in scripts/convert_to_gguf.py incorporates all these lessons and has been tested successfully in production.