Setting the file. One moment.
Skill 14 · Huggingface LLM Trainer
Subchapter 14.5
references/reliability_principles.mdMarkdown11 KBView on GitHub
These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution.
Scripts
Convert To GgufRule: Never assume repos, datasets, or resources exist. Verify with tools first.
Before submitting ANY job:
# Verify dataset exists
dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5})
hub_repo_details(["author/dataset-name"], repo_type="dataset")
# Verify model exists
hub_repo_details(["org/model-name"], repo_type="model")
# Check script/file paths (for URL-based scripts)
# Verify before using: https://github.com/user/repo/blob/main/script.pyExamples that would have caught errors:
# ❌ WRONG: Assumed dataset exists
hf_jobs("uv", {
"script": """...""",
"env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"} # Doesn't exist!
})
# ✅ CORRECT: Verify first
dataset_search({"query": "argilla dpo", "author": "trl-lib"})
# Would show: "trl-lib/ultrafeedback_binarized" is the correct name
hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset")
# Confirms it exists before usingTime cost: 5-10 seconds
Time saved: Hours of failed job time + debugging
Rule: Default to what is most likely to succeed, not what is theoretically fastest.
Choose reliability:
# ❌ RISKY: Aggressive optimization that may fail
SFTConfig(
torch_compile=True, # Can fail on T4, A10G GPUs
optim="adamw_bnb_8bit", # Requires specific setup
fp16=False, # May cause training instability
...
)
# ✅ SAFE: Proven defaults
SFTConfig(
# torch_compile=True, # Commented with note: "Enable on H100 for 20% speedup"
optim="adamw_torch", # Standard, always works
fp16=True, # Stable and fast
...
)For build processes:
# ❌ UNRELIABLE: Uses make (platform-dependent)
subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"], check=True)
# ✅ RELIABLE: Uses CMake (consistent, documented)
subprocess.run([
"cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
"-DGGML_CUDA=OFF" # Disable CUDA for faster, more reliable build
], check=True)
subprocess.run([
"cmake", "--build", "/tmp/llama.cpp/build",
"--target", "llama-quantize", "-j", "4"
], check=True)The torch.compile failure:
Result: Reliability > 20% performance gain
Performance loss: 10-20% in best case
Reliability gain: 95%+ success rate vs 60-70%
Rule: Scripts should work as complete, independent units. Don’t remove parts to “simplify.”
Complete dependency specifications:
# ❌ INCOMPLETE: "Simplified" by removing dependencies
# /// script
# dependencies = [
# "transformers",
# "peft",
# "torch",
# ]
# ///
# ✅ COMPLETE: All dependencies explicit
# /// script
# dependencies = [
# "transformers>=4.36.0",
# "peft>=0.7.0",
# "torch>=2.0.0",
# "accelerate>=0.24.0",
# "huggingface_hub>=0.20.0",
# "sentencepiece>=0.1.99", # Required for tokenizers
# "protobuf>=3.20.0", # Required for tokenizers
# "numpy",
# "gguf",
# ]
# ///Complete build processes:
# ❌ INCOMPLETE: Assumes build tools exist
subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"]) # FAILS: no gcc/make
# ✅ COMPLETE: Installs all requirements
subprocess.run(["apt-get", "update", "-qq"], check=True)
subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True)
subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
# ... then buildThe sentencepiece failure:
Result: Don’t remove dependencies without thorough testing
Complexity: Slightly longer scripts
Reliability: Scripts “just work” every time
Rule: When things fail, make it obvious what went wrong and how to fix it.
Wrap subprocess calls:
# ❌ UNCLEAR: Silent failure
subprocess.run([...], check=True, capture_output=True)
# ✅ CLEAR: Shows what failed
try:
result = subprocess.run(
[...],
check=True,
capture_output=True,
text=True
)
print(result.stdout)
if result.stderr:
print("Warnings:", result.stderr)
except subprocess.CalledProcessError as e:
print(f"❌ Command failed!")
print("STDOUT:", e.stdout)
print("STDERR:", e.stderr)
raiseValidate inputs:
# ❌ UNCLEAR: Fails later with cryptic error
model = load_model(MODEL_NAME)
# ✅ CLEAR: Fails fast with clear message
if not MODEL_NAME:
raise ValueError("MODEL_NAME environment variable not set!")
print(f"Loading model: {MODEL_NAME}")
try:
model = load_model(MODEL_NAME)
print(f"✅ Model loaded successfully")
except Exception as e:
print(f"❌ Failed to load model: {MODEL_NAME}")
print(f"Error: {e}")
print("Hint: Check that model exists on Hub")
raiseRule: Before using new code in production, test with inputs you know work.
Known-good test inputs:
# For training
TEST_DATASET = "trl-lib/Capybara" # Small, well-formatted, widely used
TEST_MODEL = "Qwen/Qwen2.5-0.5B" # Small, fast, reliable
# For GGUF conversion
TEST_ADAPTER = "evalstate/qwen-capybara-medium" # Known working model
TEST_BASE = "Qwen/Qwen2.5-0.5B" # Compatible baseTesting workflow:
Time cost: 5-10 minutes for test run
Debugging time saved: Hours
Before submitting ANY job:
Following these principles transforms job success rate from ~60-70% to ~95%+
Sometimes reliability and performance conflict. Here’s how to choose:
| Scenario | Choose | Rationale |
|---|---|---|
| Demo/test | Reliability | Fast failure is worse than slow success |
| Production (first run) | Reliability | Prove it works before optimizing |
| Production (proven) | Performance | Safe to optimize after validation |
| Time-critical | Reliability | Failures cause more delay than slow runs |
| Cost-critical | Balanced | Test with small model, then optimize |
General rule: Reliability first, optimize second.
troubleshooting.md - Common issues and fixestraining_patterns.md - Proven training configurationsgguf_conversion.md - Production GGUF workflow