Setting the file. One moment.
Skill 14 · Huggingface LLM Trainer
Subchapter 14.6
references/trackio_guide.mdMarkdown6 KBView on GitHub
Trackio is an experiment tracking library that provides real-time metrics visualization for remote training on Hugging Face Jobs infrastructure.
⚠️ IMPORTANT: For Jobs training (remote cloud GPUs):
Scripts
Convert To Ggufprivate=True to trackio.init() if the metrics should not be publicStep 1: Add trackio dependency
# /// script
# dependencies = [
# "trl>=0.12.0",
# "trackio", # Required!
# ]
# ///Step 2: Create a Trackio Space (one-time setup)
Option A: Let Trackio auto-create (Recommended)
Pass a space_id to trackio.init() and Trackio will automatically create the Space if it doesn’t exist.
Option B: Create manually
hf repos create my-trackio-dashboard --type space --space-sdk gradioStep 3: Initialize Trackio with space_id
import trackio
trackio.init(
project="my-training",
space_id="username/trackio", # CRITICAL for Jobs! Replace 'username' with your HF username
private=True, # Spaces are PUBLIC by default; omit for a shareable dashboard
config={
"model": "Qwen/Qwen2.5-0.5B",
"dataset": "trl-lib/Capybara",
"learning_rate": 2e-5,
}
)Step 4: Configure TRL to use Trackio
SFTConfig(
report_to="trackio",
# ... other config
)Step 5: Finish tracking
trainer.train()
trackio.finish() # Ensures final metrics are syncedTrackio automatically logs:
nvidia-ml-py is installed — true for standard Jobs GPU flavors)space_id, or pinned with bucket_id=) every ~30 secondstrackio.finish() drains any pending metrics so everything is persistedUse sensible defaults for trackio configuration unless user requests otherwise.
import trackio
trackio.init(
project="qwen-capybara-sft",
name="baseline-run", # Descriptive name user will recognize
space_id="username/trackio", # Default space: {username}/trackio
private=True, # Spaces are PUBLIC by default; omit for a shareable dashboard
config={
# Keep config minimal - hyperparameters and model/dataset info only
"model": "Qwen/Qwen2.5-0.5B",
"dataset": "trl-lib/Capybara",
"learning_rate": 2e-5,
"num_epochs": 3,
}
)Key principles:
{username}/trackio with “trackio” as default space nameThe group parameter helps organize related runs together in the dashboard sidebar. This is useful when user is running multiple experiments with different configurations but wants to compare them together:
# Example: Group runs by experiment type
trackio.init(project="my-project", run_name="baseline-run-1", group="baseline")
trackio.init(project="my-project", run_name="augmented-run-1", group="augmented")
trackio.init(project="my-project", run_name="tuned-run-1", group="tuned")Runs with the same group name can be grouped together in the sidebar, making it easier to compare related experiments. You can group by any configuration parameter:
# Hyperparameter sweep - group by learning rate
trackio.init(project="hyperparam-sweep", run_name="lr-0.001-run", group="lr_0.001")
trackio.init(project="hyperparam-sweep", run_name="lr-0.01-run", group="lr_0.01")You can configure trackio using environment variables instead of passing parameters to trackio.init(). This is useful for managing configuration across multiple jobs.
HF_TOKEN
Required for creating Spaces and writing metrics to the HF Bucket (passed via secrets):
hf_jobs("uv", {
"script": "...",
"secrets": {
"HF_TOKEN": "$HF_TOKEN" # Enables Space creation and Hub push
}
})hf_jobs("uv", {
"script": """
# Training script - trackio config from environment
import trackio
from datetime import datetime
# Auto-generate run name
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M")
run_name = f"sft_qwen25_{timestamp}"
# Project and space_id can come from environment variables
trackio.init(run_name=run_name, group="SFT")
# ... training code ...
trackio.finish()
""",
"flavor": "a10g-large",
"timeout": "2h",
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
})When to use environment variables:
When to use direct parameters:
After starting training:
https://huggingface.co/spaces/username/trackio