Setting the file. One moment.
Chapter 22 · Huggingface Vision Trainer
Subchapter 22.5
references/reliability_principles.mdMarkdown9 KBView on GitHub
These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution.
Rule: Never assume repos, datasets, or resources exist. Verify with tools first.
Before submitting ANY job:
# Verify dataset exists
dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5})
hub_repo_details(["author/dataset-name"], repo_type="dataset")
# Verify model exists
hub_repo_details(["org/model-name"], repo_type="model")
# Check script/file paths (for URL-based scripts)
# Verify before using: https://github.com/user/repo/blob/main/script.pyExamples that would have caught errors:
# ❌ WRONG: Assumed dataset exists
hf_jobs("uv", {
"script": """...""",
"env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"} # Doesn't exist!
})
# ✅ CORRECT: Verify first
dataset_search({"query": "argilla dpo", "author": "trl-lib"})
# Would show: "trl-lib/ultrafeedback_binarized" is the correct name
hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset")
# Confirms it exists before usingTime cost: 5-10 seconds
Time saved: Hours of failed job time + debugging
Rule: Default to what is most likely to succeed, not what is theoretically fastest.
Choose reliability:
# ❌ RISKY: Aggressive optimization that may fail
TrainingArguments(
torch_compile=True, # Can fail on T4, A10G GPUs
optim="adamw_bnb_8bit", # Requires specific setup
dataloader_num_workers=8, # May cause OOM on small instances
...
)
# ✅ SAFE: Proven defaults
TrainingArguments(
# torch_compile=True, # Commented with note: "Enable on H100 for 20% speedup"
optim="adamw_torch", # Standard, always works
fp16=True, # Stable and fast on T4/A10G
dataloader_num_workers=4, # Conservative, reliable
...
)The torch.compile failure:
Result: Reliability > 20% performance gain
Performance loss: 10-20% in best case
Reliability gain: 95%+ success rate vs 60-70%
Rule: Scripts should work as complete, independent units. Don’t remove parts to “simplify.”
Complete dependency specifications:
# ❌ INCOMPLETE: "Simplified" by removing dependencies
# /// script
# dependencies = [
# "transformers",
# "torch",
# "datasets",
# ]
# ///
# ✅ COMPLETE: All dependencies explicit
# /// script
# dependencies = [
# "transformers>=5.2.0",
# "accelerate>=1.1.0",
# "albumentations>=1.4.16", # Required for augmentation + bbox handling
# "timm", # Required for vision backbones
# "datasets>=4.0",
# "torchmetrics", # Required for mAP/mAR computation
# "pycocotools", # Required for COCO evaluation
# "trackio", # Required for metrics monitoring
# "huggingface_hub",
# ]
# ///The albumentations failure:
Result: Don’t remove dependencies without thorough testing
Complexity: Slightly longer scripts
Reliability: Scripts “just work” every time
Rule: When things fail, make it obvious what went wrong and how to fix it.
Wrap subprocess calls:
# ❌ UNCLEAR: Silent failure
subprocess.run([...], check=True, capture_output=True)
# ✅ CLEAR: Shows what failed
try:
result = subprocess.run(
[...],
check=True,
capture_output=True,
text=True
)
print(result.stdout)
if result.stderr:
print("Warnings:", result.stderr)
except subprocess.CalledProcessError as e:
print(f"❌ Command failed!")
print("STDOUT:", e.stdout)
print("STDERR:", e.stderr)
raiseValidate inputs:
# ❌ UNCLEAR: Fails later with cryptic error
model = load_model(MODEL_NAME)
# ✅ CLEAR: Fails fast with clear message
if not MODEL_NAME:
raise ValueError("MODEL_NAME environment variable not set!")
print(f"Loading model: {MODEL_NAME}")
try:
model = load_model(MODEL_NAME)
print(f"✅ Model loaded successfully")
except Exception as e:
print(f"❌ Failed to load model: {MODEL_NAME}")
print(f"Error: {e}")
print("Hint: Check that model exists on Hub")
raiseRule: Before using new code in production, test with inputs you know work.
Before submitting ANY job:
login(token=hf_token) and sets training_args.hub_token = hf_token BEFORE Trainer() initFollowing these principles transforms job success rate from ~60-70% to ~95%+
Sometimes reliability and performance conflict. Here’s how to choose:
| Scenario | Choose | Rationale |
|---|---|---|
| Demo/test | Reliability | Fast failure is worse than slow success |
| Production (first run) | Reliability | Prove it works before optimizing |
| Production (proven) | Performance | Safe to optimize after validation |
| Time-critical | Reliability | Failures cause more delay than slow runs |
| Cost-critical | Balanced | Test with small model, then optimize |
General rule: Reliability first, optimize second.