Subchapter 109.1

Export Commands

references/export-commands.mdMarkdown12 KBView on GitHub

Last verified: 2026-07-14

Export Commands

Complete command sequences for every format on the SKILL.md Format Map, plus the smoke-test script skeleton. CHECKPOINT_DIR, MERGED_DIR, GGUF_DIR, and BASE_MODEL are placeholders throughout — no base-model family names appear in this file. Fill each with the promoted checkpoint’s actual path/repo before running.

Export Commands · Quantized Export · wshobson/agents · Skills Docs

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=CHECKPOINT_DIR,
    max_seq_length=4096,
    load_in_4bit=False,  # load full precision before merge
)

# fp16/bf16 merged export — safe default, no quant loss at this step
model.save_pretrained_merged(
    MERGED_DIR,
    tokenizer,
    save_method="merged_16bit",
)

# 4-bit merged export — only if the serving stack
# consumes bitsandbytes 4-bit directly (rare; most
# deployments quantize downstream instead — see the
# AWQ and GGUF sections below)
model.save_pretrained_merged(
    MERGED_DIR + "-4bit",
    tokenizer,
    save_method="merged_4bit",
)

# 1. Convert HF safetensors to GGUF (f16, no quant yet)
python convert_hf_to_gguf.py \
    "$MERGED_DIR" \
    --outfile "$GGUF_DIR/model-f16.gguf" \
    --outtype f16

# 2. Generate the importance matrix from a calibration
#    corpus — domain-representative text, several
#    hundred KB minimum; a generic corpus (e.g. the
#    llama.cpp wikitext sample) works if no
#    domain corpus is available
./llama-imatrix \
    -m "$GGUF_DIR/model-f16.gguf" \
    -f calibration-corpus.txt \
    -o "$GGUF_DIR/imatrix.dat" \
    --chunks 200

# 3. Quantize using the imatrix — Q4_K_M is the
#    edge/llama.cpp default from the Format Map
./llama-quantize \
    --imatrix "$GGUF_DIR/imatrix.dat" \
    "$GGUF_DIR/model-f16.gguf" \
    "$GGUF_DIR/model-Q4_K_M.gguf" \
    Q4_K_M

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

model = AutoAWQForCausalLM.from_pretrained(MERGED_DIR)
tokenizer = AutoTokenizer.from_pretrained(MERGED_DIR)

# calibration data: a few hundred domain-representative
# samples; reuses the same calibration-corpus concept
# as the llama.cpp imatrix step above
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(MERGED_DIR + "-awq")
tokenizer.save_pretrained(MERGED_DIR + "-awq")

vllm serve "$MERGED_DIR" \
    --quantization fp8 \
    --served-model-name checkpoint-fp8 \
    --port 8000 > vllm-fp8-load.log 2>&1 &
VLLM_PID=$!

# Bounded wait for readiness — up to 60s, not a blind sleep.
ready=0
for _ in $(seq 1 30); do
    if curl -sf http://localhost:8000/v1/models | grep -q checkpoint-fp8; then
        ready=1
        break
    fi
    sleep 2
done

if [ "$ready" -ne 1 ]; then
    echo "FAILED: endpoint did not come up within 60s"
    kill "$VLLM_PID" 2>/dev/null
    exit 1
fi

# Endpoint liveness alone does not prove FP8 loaded — vLLM can
# silently fall back to bf16. Confirm the actual dtype from the
# startup log before trusting the deployment.
if grep -qi 'fp8' vllm-fp8-load.log; then
    echo "FP8 endpoint up and confirmed FP8 in startup log"
else
    echo "FAILED: endpoint up but startup log does not confirm FP8 (possible silent bf16 fallback)"
    kill "$VLLM_PID" 2>/dev/null
    exit 1
fi

import json
import sys

# Deterministic decoding, persisted and reused for both the
# pre-export run (that produced pre-export-outputs.jsonl) and the
# post-export run below — greedy (temperature 0) with a fixed seed.
# Any drift in these settings between the two runs can flip an
# otherwise-valid export into a spurious mismatch.
SMOKE_TEST_GENERATION_KWARGS = {"temperature": 0, "seed": 0, "max_new_tokens": 512}

def load_pre_export_outputs(path: str) -> dict:
    """goldens.jsonl-keyed pre-export generations,
    produced from the promoted checkpoint before any

Export Commands

Export Commands

Export Commands

Export Commands

Unsloth: Merged Safetensors

Unsloth: GGUF with Quant Method

llama.cpp: Build, Convert, imatrix + Quantize

AWQ Export Sketch

vLLM: FP8 Load Check

Smoke-Test Script Skeleton