--- title: "huggingface/skills" description: "Give your agents the power of the Hugging Face ecosystem" source: https://github.com/huggingface/skills ref: main license: Apache-2.0 licenseName: "Apache License 2.0" canonical: https://skillsdocs.com/huggingface/skills base: https://github.com/huggingface/skills/blob/main/ chapters: 26 inlined: 26 withheld: 0 words: 30525 updated: 2026-08-07T13:44:54Z generator: "Skills Docs" --- > **huggingface/skills** — every Agent Skill in this repository, inlined verbatim. > > Canonical HTML: https://skillsdocs.com/huggingface/skills > Per-chapter Markdown: https://skillsdocs.com/huggingface/skills/.md > Machine manifest: https://skillsdocs.com/huggingface/skills/.well-known/agent-skills/index.json > JSON: https://skillsdocs.com/api/v1/books/huggingface/skills > Install: `npx skills add huggingface/skills` > Upstream: https://github.com/huggingface/skills @ `main` > Licence: Apache-2.0 > > Content is mirrored from GitHub and © its authors, served unmodified. Takedown: https://github.com/kyleledbetter/skillsdocs/issues/new?labels=takedown&title=Takedown+request # huggingface/skills Give your agents the power of the Hugging Face ecosystem - **Chapters:** 26 - **Inlined:** 26 (licence detected) - **Words:** 30,525 - **Reading time:** 139 min - **Stars:** 10,907 ## Table of contents 1. [hf-mcp](https://skillsdocs.com/huggingface/skills/hf-mcp.md) — Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spa… 2. [hf-cli](https://skillsdocs.com/huggingface/skills/hf-cli.md) — Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub.… 3. [hf-cloud-aws-context-discovery](https://skillsdocs.com/huggingface/skills/hf-cloud-aws-context-discovery.md) — Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS w… 4. [hf-cloud-python-env-setup](https://skillsdocs.com/huggingface/skills/hf-cloud-python-env-setup.md) — Set up an isolated Python environment for SageMaker / AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be ex… 5. [hf-cloud-sagemaker-deployment-planner](https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-deployment-planner.md) — Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageM… 6. [hf-cloud-sagemaker-iam-preflight](https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-iam-preflight.md) — Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job… 7. [hf-cloud-sagemaker-production-defaults](https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-production-defaults.md) — Create a SageMaker endpoint (real-time, real-time scale-to-zero, or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill… 8. [hf-cloud-serving-image-selection](https://skillsdocs.com/huggingface/skills/hf-cloud-serving-image-selection.md) — Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMake… 9. [hf-mem](https://skillsdocs.com/huggingface/skills/hf-mem.md) — Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub 10. [huggingface-best](https://skillsdocs.com/huggingface/skills/huggingface-best.md) — Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmar… 11. [huggingface-community-evals](https://skillsdocs.com/huggingface/skills/huggingface-community-evals.md) — Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between… 12. [huggingface-datasets](https://skillsdocs.com/huggingface/skills/huggingface-datasets.md) — Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs… 13. [huggingface-gradio](https://skillsdocs.com/huggingface/skills/huggingface-gradio.md) — Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots. 14. [huggingface-llm-trainer](https://skillsdocs.com/huggingface/skills/huggingface-llm-trainer.md) — Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. Covers SFT, DPO,… 15. [huggingface-local-models](https://skillsdocs.com/huggingface/skills/huggingface-local-models.md) — Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGU… 16. [huggingface-lora-space-builder](https://skillsdocs.com/huggingface/skills/huggingface-lora-space-builder.md) — Build and publish a Gradio demo on Hugging Face Spaces for a user-provided LoRA. Use when someone asks to create, generate, ship, or publish a Space, demo, Gra… 17. [huggingface-paper-publisher](https://skillsdocs.com/huggingface/skills/huggingface-paper-publisher.md) — Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating p… 18. [huggingface-papers](https://skillsdocs.com/huggingface/skills/huggingface-papers.md) — Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github re… 19. [huggingface-spaces](https://skillsdocs.com/huggingface/skills/huggingface-spaces.md) — Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buck… 20. [huggingface-tool-builder](https://skillsdocs.com/huggingface/skills/huggingface-tool-builder.md) — Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. This is especially useful whe… 21. [huggingface-trackio](https://skillsdocs.com/huggingface/skills/huggingface-trackio.md) — Track and visualize ML training experiments with Trackio. Use when logging metrics during training (Python API), firing alerts for training diagnostics, or ret… 22. [huggingface-vision-trainer](https://skillsdocs.com/huggingface/skills/huggingface-vision-trainer.md) — Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet,… 23. [huggingface-zerogpu](https://skillsdocs.com/huggingface/skills/huggingface-zerogpu.md) — AI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. Use when writing or reviewing code that uses `@spaces.GPU`, configuring `python_ve… 24. [train-sentence-transformers](https://skillsdocs.com/huggingface/skills/train-sentence-transformers.md) — Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clusterin… 25. [transformers-js](https://skillsdocs.com/huggingface/skills/transformers-js.md) — Use Transformers.js to run state-of-the-art machine learning models directly in JavaScript/TypeScript. Supports NLP (text classification, translation, summariz… 26. [trl-training](https://skillsdocs.com/huggingface/skills/trl-training.md) — Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training v… ## Front matter _The repository README, verbatim except that relative links are resolved against https://github.com/huggingface/skills/blob/main/._ # Hugging Face Skills Hugging Face Skills are definitions for AI/ML tasks like dataset creation, model training, and evaluation. The client plugin marketplaces expose the `hf-cli` skill as the bootstrap path for core Hub operations; additional workflow skills can be installed on demand with `hf skills add ` or discovered by skill-aware clients over CLI/MCP integrations. The skills in this repository follow the standardized [Agent Skills](https://agentskills.io/home) format. > [!NOTE] > **Just want to give your agent access to the Hugging Face Hub?** Start with [`hf-cli`](https://huggingface.co/docs/hub/agents-cli). It's the recommended first Skill to install: it teaches your agent every `hf` command (search models, manage datasets and buckets, launch Spaces, run jobs) and is generated from your locally installed CLI so it stays current. ## How do Skills work? In practice, skills are self-contained folders that package instructions, scripts, and resources together for an AI agent to use on a specific use case. Each folder includes a `SKILL.md` file with YAML frontmatter (name and description) followed by the guidance your coding agent follows while the skill is active. > [!TIP] > If your agent doesn't support skills, you can use [`agentsmd/AGENTS.md`](https://github.com/huggingface/skills/blob/main/agentsmd/AGENTS.md) directly as a fallback. The `hf-cli` skill in this repository is also available through: - Cursor Marketplace (https://cursor.com/marketplace/huggingface) - Codex Plugins Directory (https://developers.openai.com/codex/plugins) ## Installation Hugging Face skills are compatible with Claude Code, Codex, Gemini CLI, and Cursor. ### Claude Code 1. Register the repository as a plugin marketplace: ``` /plugin marketplace add huggingface/skills ``` 2. Install the CLI skill: ``` /plugin install hf-cli@huggingface/skills ``` 3. To install another Hugging Face skill, use the `hf` CLI: ``` hf skills add ``` ### Codex 1. Copy or symlink any skills you want to use from this repository's `skills/` directory into one of Codex's standard `.agents/skills` locations (for example, `$REPO_ROOT/.agents/skills` or `$HOME/.agents/skills`) as described in the [Codex Skills guide](https://developers.openai.com/codex/skills/). 2. Once a skill is available in one of those locations, Codex will discover it using the Agent Skills standard and load the `SKILL.md` instructions when it decides to use that skill or when you explicitly invoke it. 3. If your Codex setup still relies on `AGENTS.md`, you can use the generated [`agentsmd/AGENTS.md`](https://github.com/huggingface/skills/blob/main/agentsmd/AGENTS.md) file in this repo as a fallback bundle of instructions. ### Gemini CLI 1. This repo includes `gemini-extension.json` to integrate with the Gemini CLI. 2. Install locally: ``` gemini extensions install . --consent ``` or use the GitHub URL: ``` gemini extensions install https://github.com/huggingface/skills.git --consent ``` 4. See [Gemini CLI extensions docs](https://geminicli.com/docs/extensions/#installing-an-extension) for more help. ### Cursor This repository includes Cursor plugin manifests: - `.cursor-plugin/plugin.json` - `.mcp.json` (configured with the Hugging Face MCP server URL) Install from repository URL (or local checkout) via the Cursor plugin flow. The marketplace entry is intentionally limited to `hf-cli`; use `hf skills add ` to install additional workflow skills. For contributors, regenerate manifests with: ```bash ./scripts/publish.sh ``` ## Skills This repository contains a few skills to get you started. You can also contribute your own skills to the repository. ### Available skills | Name | Description | Documentation | |------|-------------|---------------| | `hf-cli` | Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cli/SKILL.md) | | `hf-cloud-aws-context-discovery` | Discover the user''s local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-aws-context-discovery/SKILL.md) | | `hf-cloud-python-env-setup` | Set up an isolated Python environment for SageMaker / AWS work, with the right Python version and current boto3. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-python-env-setup/SKILL.md) | | `hf-cloud-sagemaker-deployment-planner` | Plan and coordinate the deployment of a model to Amazon SageMaker AI. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-deployment-planner/SKILL.md) | | `hf-cloud-sagemaker-iam-preflight` | Ensure a usable SageMaker execution role exists before deploying or training. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-iam-preflight/SKILL.md) | | `hf-cloud-sagemaker-production-defaults` | Create a SageMaker endpoint (real-time, real-time scale-to-zero, or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-production-defaults/SKILL.md) | | `hf-cloud-serving-image-selection` | Pick the right serving container for a SageMaker model deployment and find its current image URI. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-cloud-serving-image-selection/SKILL.md) | | `hf-mem` | Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/hf-mem/SKILL.md) | | `huggingface-best` | Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-best/SKILL.md) | | `huggingface-community-evals` | Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-community-evals/SKILL.md) | | `huggingface-datasets` | Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-datasets/SKILL.md) | | `huggingface-gradio` | Build Gradio web UIs and demos in Python. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-gradio/SKILL.md) | | `huggingface-llm-trainer` | Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-llm-trainer/SKILL.md) | | `huggingface-local-models` | Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-local-models/SKILL.md) | | `huggingface-lora-space-builder` | Build and publish a Gradio demo on Hugging Face Spaces for a user-provided LoRA. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-lora-space-builder/SKILL.md) | | `huggingface-paper-publisher` | Publish and manage research papers on Hugging Face Hub. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-paper-publisher/SKILL.md) | | `huggingface-papers` | Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-papers/SKILL.md) | | `huggingface-spaces` | Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buckets, inference providers, community grants. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-spaces/SKILL.md) | | `huggingface-tool-builder` | Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-tool-builder/SKILL.md) | | `huggingface-trackio` | Track and visualize ML training experiments with Trackio. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-trackio/SKILL.md) | | `huggingface-vision-trainer` | Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3 — plus any Transformers classifier), and SAM/SAM2 segmentation using Hugging Face Transformers on Hugging Face Jobs cloud GPUs. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-vision-trainer/SKILL.md) | | `huggingface-zerogpu` | AI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/huggingface-zerogpu/SKILL.md) | | `train-sentence-transformers` | Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval / pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/train-sentence-transformers/SKILL.md) | | `transformers-js` | Use Transformers.js to run state-of-the-art machine learning models directly in JavaScript/TypeScript. | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/transformers-js/SKILL.md) | | `trl-training` | Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). | [SKILL.md](https://github.com/huggingface/skills/blob/main/skills/trl-training/SKILL.md) | ### Using skills in your coding agent Once a skill is installed, mention it directly while giving your coding agent instructions: - "Use the HF LLM trainer skill to estimate the GPU memory needed for a 70B model run." - "Use the HF model evaluation skill to launch `run_eval_job.py` on the latest checkpoint." - "Use the HF dataset creator skill to draft new few-shot classification templates." - "Use the HF paper publisher skill to index my arXiv paper and link it to my model." Your coding agent automatically loads the corresponding `SKILL.md` instructions and helper scripts while it completes the task. ### Contribute or customize a skill 1. Copy one of the existing skill folders (for example, `hf-datasets/`) and rename it. 2. Update the new folder's `SKILL.md` frontmatter: ```markdown --- name: my-skill-name description: Describe what the skill does and when to use it --- # Skill Title Guidance + examples + guardrails ``` 3. Add or edit supporting scripts, templates, and documents referenced by your instructions. 4. Do not add the skill to `.claude-plugin/marketplace.json` by default. Client marketplaces are intentionally limited to `hf-cli`; the full Hub CLI marketplace is generated at `.claude-plugin/marketplace-internal.json`. 5. Run: ```bash ./scripts/publish.sh ``` to regenerate and validate all generated metadata. 6. Reinstall or reload the skill bundle in your coding agent so the updated folder is available. ### Marketplace The `.claude-plugin/marketplace.json` and `.cursor-plugin/marketplace.json` files intentionally expose only `hf-cli` for client marketplace installation. This keeps install-time manifests focused on core Hub operations and points users to `hf skills add ` for the rest of the repository. The generated `.claude-plugin/marketplace-internal.json` file contains the full skill list. Publish automation uploads it to the Hub bucket as `marketplace.json` so `hf skills list`, `hf skills add`, and `hf skills update` continue to see every available skill. Newer skill-aware integrations can also pull capabilities dynamically. Hugging Face's discovery flow lets clients search skills, MCP servers, and Spaces, while Skills-over-MCP work is standardizing how skills are discovered and consumed through MCP resources. ### Additional references - Browse the latest instructions, scripts, and templates directly at [huggingface/skills](https://github.com/huggingface/skills). - Review Hugging Face documentation for the specific libraries or workflows you reference inside each skill. --- ## 1. hf-mcp - **Source:** https://github.com/huggingface/skills/blob/main/hf-mcp/skills/hf-mcp/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/hf-mcp/skills/hf-mcp/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-mcp.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: hf-mcp description: Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server. --- # Hugging Face MCP Server Connect AI assistants to the Hugging Face Hub. Setup: https://huggingface.co/settings/mcp ## Use Cases & Examples ### Find the Best Model for a Task ``` User: "Find the best model for code generation" 1. model_search(task="text-generation", query="code", sort="trendingScore", limit=10) 2. hub_repo_details(repo_ids=["top-result-id"], include_readme=true) ``` ### Compare Models from Different Providers ``` User: "Compare Llama vs Qwen for text generation" 1. model_search(author="meta-llama", task="text-generation", sort="downloads", limit=5) 2. model_search(author="Qwen", task="text-generation", sort="downloads", limit=5) 3. hub_repo_details(repo_ids=["meta-llama/Llama-3.2-1B", "Qwen/Qwen3-8B"], include_readme=true) ``` ### Find Training Datasets ``` User: "Find datasets for sentiment analysis in English" 1. dataset_search(query="sentiment", tags=["language:en", "task_categories:text-classification"], sort="downloads") 2. hub_repo_details(repo_ids=["top-dataset-id"], repo_type="dataset", include_readme=true) ``` ### Discover AI Tools (MCP Spaces) ``` User: "Find a tool that can remove image backgrounds" 1. space_search(query="background removal", mcp=true) 2. dynamic_space(operation="view_parameters", space_name="result-space-id") 3. dynamic_space(operation="invoke", space_name="result-space-id", parameters="{...}") ``` ### Generate Images ``` User: "Create an image of a robot reading a book" 1. dynamic_space(operation="discover") # See available tasks 2. gr1_flux1_schnell_infer(prompt="a robot sitting in a library reading a book, warm lighting, detailed") ``` ### Research a Topic ``` User: "What are the latest papers on RLHF?" 1. paper_search(query="reinforcement learning from human feedback", results_limit=10) 2. hub_repo_details(repo_ids=["paper-linked-model"], include_readme=true) # If paper links to models ``` ### Learn How to Use a Library ``` User: "How do I fine-tune with LoRA using PEFT?" 1. hf_doc_search(query="LoRA fine-tuning", product="peft") 2. hf_doc_fetch(doc_url="https://huggingface.co/docs/peft/...") ``` ### Run a Quick GPU Job ``` User: "Run this Python script on a GPU" hf_jobs(operation="uv", args={ "script": "# /// script\n# dependencies = [\"torch\"]\n# ///\nimport torch\nprint(torch.cuda.is_available())", "flavor": "t4-small" }) ``` ### Train a Model on Cloud GPU ``` User: "Run my training script on an A10G" hf_jobs(operation="run", args={ "image": "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime", "command": ["/bin/sh", "-lc", "pip install transformers trl && python train.py"], "flavor": "a10g-small", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ### Check Job Status ``` User: "What's happening with my training job?" 1. hf_jobs(operation="ps") 2. hf_jobs(operation="logs", args={"job_id": "job-xxxxx"}) ``` ### Explore What's Trending ``` User: "What models are trending right now?" model_search(sort="trendingScore", limit=20) ``` ### Get Model Card Details ``` User: "Tell me about Mistral-7B" hub_repo_details(repo_ids=["mistralai/Mistral-7B-v0.1"], include_readme=true) ``` ### Find Quantized Models ``` User: "Find GGUF versions of Llama 3" model_search(query="Llama 3 GGUF", sort="downloads", limit=10) ``` ### Use a Gradio Space as a Tool ``` User: "Transcribe this audio file" 1. space_search(query="speech to text transcription", mcp=true) 2. dynamic_space(operation="view_parameters", space_name="openai/whisper") 3. dynamic_space(operation="invoke", space_name="openai/whisper", parameters="{\"audio\": \"...\"}") ``` ### Schedule Recurring Jobs ``` User: "Run this data sync every day at midnight" hf_jobs(operation="scheduled uv", args={ "script": "...", "cron": "0 0 * * *", "flavor": "cpu-basic" }) ``` ## Tool Selection Guide | Goal | Tool | |------|------| | Find models | `model_search` | | Find datasets | `dataset_search` | | Find Spaces/apps | `space_search` | | Find papers | `paper_search` | | Get repo README/details | `hub_repo_details` | | Learn library usage | `hf_doc_search` → `hf_doc_fetch` | | Run code on GPU/CPU | `hf_jobs` | | Use Gradio apps as tools | `dynamic_space` | | Generate images | `gr1_flux1_schnell_infer` or `dynamic_space` | | Check auth | `hf_whoami` | ## Tips - Use `sort="trendingScore"` to find what's popular now - Use `sort="downloads"` to find battle-tested options - Set `mcp=true` in `space_search` to find Spaces usable as tools - Use `include_readme=true` in `hub_repo_details` for full model/dataset documentation - For jobs accessing private repos, always include `secrets: {"HF_TOKEN": "$HF_TOKEN"}` - Use `dynamic_space(operation="discover")` to see all available Space-based tasks --- ## 2. hf-cli - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cli/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cli/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cli.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: hf-cli description: "Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`." --- Install: `curl -LsSf https://hf.co/cli/install.sh | bash -s`. The Hugging Face Hub CLI tool `hf` is available. IMPORTANT: The `hf` command replaces the deprecated `huggingface-cli` command. Use `hf --help` to view available functions. Note that auth commands are now all under `hf auth` e.g. `hf auth whoami`. Generated with `huggingface_hub v1.27.0`. Run `hf skills add --force` to regenerate. ## Commands - `hf cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` - `hf download REPO_ID` — Download files from the Hub. `[--type [model|dataset|space] --revision TEXT --include TEXT --exclude TEXT --cache-dir TEXT --local-dir TEXT --force-download --dry-run --max-workers INTEGER --format [auto|human|agent|json|quiet]]` - `hf env` — Print information about the environment. `[--format [auto|human|agent|json|quiet]]` - `hf sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]` - `hf update` — Update the `hf` CLI to the latest version. `[--format [auto|human|agent|json|quiet]]` - `hf upload REPO_ID` — Upload a file or a folder to the Hub. Recommended for single-commit uploads. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --delete TEXT --commit-message TEXT --commit-description TEXT --create-pr --every FLOAT --format [auto|human|agent|json|quiet]]` - `hf upload-large-folder REPO_ID LOCAL_PATH` — [Deprecated] Upload a large folder to the Hub. Use `hf upload` instead. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --num-workers INTEGER --no-report --no-bars --format [auto|human|agent|json|quiet]]` - `hf version` — Print information about the hf version. `[--format [auto|human|agent|json|quiet]]` ### `hf auth` — Manage authentication (login, logout, etc.). - `hf auth list` — List all stored access tokens. `[--format [auto|human|agent|json|quiet]]` - `hf auth login` — Login from your browser, or using a token from huggingface.co/settings/tokens. `[--add-to-git-credential --force --format [auto|human|agent|json|quiet]]` - `hf auth logout` — Logout from a specific token. `[--token-name TEXT --format [auto|human|agent|json|quiet]]` - `hf auth switch` — Switch between access tokens. `[--token-name TEXT --add-to-git-credential --format [auto|human|agent|json|quiet]]` - `hf auth token` — Print the current access token to stdout. `[--format [auto|human|agent|json|quiet]]` - `hf auth whoami` — Find out which huggingface.co account you are logged in as. `[--format [auto|human|agent|json|quiet]]` ### `hf buckets` — Commands to interact with buckets. - `hf buckets cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` - `hf buckets create BUCKET_ID` — Create a new bucket. `[--private --region [us|eu] --exist-ok --format [auto|human|agent|json|quiet]]` - `hf buckets delete BUCKET_ID` — Delete a bucket. `[--yes --missing-ok --format [auto|human|agent|json|quiet]]` - `hf buckets info BUCKET_ID` — Get info about a bucket. `[--format [auto|human|agent|json|quiet]]` - `hf buckets list` — List buckets or files in a bucket. `[--human-readable --tree --recursive --search TEXT --format [auto|human|agent|json|quiet]]` - `hf buckets move FROM_ID TO_ID` — Move (rename) a bucket to a new name or namespace. `[--format [auto|human|agent|json|quiet]]` - `hf buckets remove ARGUMENT` — Remove files from a bucket. `[--recursive --yes --dry-run --include TEXT --exclude TEXT --format [auto|human|agent|json|quiet]]` - `hf buckets sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]` ### `hf cache` — Manage local cache directory. - `hf cache list` — List cached repositories or revisions. `[--cache-dir TEXT --revisions --filter TEXT --sort [accessed|accessed:asc|accessed:desc|modified|modified:asc|modified:desc|name|name:asc|name:desc|size|size:asc|size:desc] --limit INTEGER --show-warnings --format [auto|human|agent|json|quiet]]` - `hf cache prune` — Remove detached revisions and incomplete downloads from the cache. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]` - `hf cache rm TARGETS` — Remove cached repositories or revisions. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]` - `hf cache verify REPO_ID` — Verify checksums for a single repo revision from cache or a local directory. `[--type [model|dataset|space] --revision TEXT --cache-dir TEXT --local-dir TEXT --fail-on-missing-files --fail-on-extra-files --format [auto|human|agent|json|quiet]]` ### `hf collections` — Interact with collections on the Hub. - `hf collections add-item COLLECTION_SLUG ITEM_ID ITEM_TYPE` — Add an item to a collection. `[--note TEXT --exists-ok --format [auto|human|agent|json|quiet]]` - `hf collections create TITLE` — Create a new collection on the Hub. `[--namespace TEXT --description TEXT --private --exists-ok --format [auto|human|agent|json|quiet]]` - `hf collections delete COLLECTION_SLUG` — Delete a collection from the Hub. `[--missing-ok --format [auto|human|agent|json|quiet]]` - `hf collections delete-item COLLECTION_SLUG ITEM_OBJECT_ID` — Delete an item from a collection. `[--missing-ok --format [auto|human|agent|json|quiet]]` - `hf collections info COLLECTION_SLUG` — Get info about a collection on the Hub. `[--format [auto|human|agent|json|quiet]]` - `hf collections list` — List collections on the Hub. `[--owner TEXT --item TEXT --sort [lastModified|trending|upvotes] --limit INTEGER --format [auto|human|agent|json|quiet]]` - `hf collections update COLLECTION_SLUG` — Update a collection's metadata on the Hub. `[--title TEXT --description TEXT --position INTEGER --private --theme TEXT --format [auto|human|agent|json|quiet]]` - `hf collections update-item COLLECTION_SLUG ITEM_OBJECT_ID` — Update an item in a collection. `[--note TEXT --position INTEGER --format [auto|human|agent|json|quiet]]` ### `hf datasets` — Interact with datasets on the Hub. - `hf datasets card DATASET_ID` — Get the dataset card (README) for a dataset on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` - `hf datasets info DATASET_ID` — Get info about a dataset on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` - `hf datasets leaderboard DATASET_ID` — List model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards. `[--limit INTEGER --format [auto|human|agent|json|quiet]]` - `hf datasets list` — List datasets on the Hub, or files in a dataset repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` - `hf datasets parquet DATASET_ID` — List parquet file URLs available for a dataset. `[--subset TEXT --split TEXT --format [auto|human|agent|json|quiet]]` - `hf datasets sql SQL` — Execute a raw SQL query with DuckDB against dataset parquet URLs. `[--format [auto|human|agent|json|quiet]]` ### `hf discussions` — Manage discussions and pull requests on the Hub. - `hf discussions close REPO_ID NUM` — Close a discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions comment REPO_ID NUM` — Comment on a discussion or pull request. `[--body TEXT --body-file PATH --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions create REPO_ID --title TEXT` — Create a new discussion or pull request on a repo. `[--body TEXT --body-file PATH --pull-request --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions diff REPO_ID NUM` — Show the diff of a pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions edit REPO_ID NUM COMMENT_ID` — Edit an existing comment on a discussion or pull request. `[--body TEXT --body-file PATH --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions info REPO_ID NUM` — Get info about a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions list REPO_ID` — List discussions and pull requests on a repo. `[--status [open|closed|merged|draft|all] --kind [all|discussion|pull_request] --author TEXT --limit INTEGER --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions merge REPO_ID NUM` — Merge a pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions rename REPO_ID NUM NEW_TITLE` — Rename a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf discussions reopen REPO_ID NUM` — Reopen a closed discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` ### `hf endpoints` — Manage Hugging Face Inference Endpoints. - `hf endpoints catalog deploy --repo TEXT` — Deploy an Inference Endpoint from the Model Catalog. `[--name TEXT --accelerator TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf endpoints catalog list` — List available Catalog models. `[--format [auto|human|agent|json|quiet]]` - `hf endpoints delete NAME` — Delete an Inference Endpoint permanently. `[--namespace TEXT --yes --format [auto|human|agent|json|quiet]]` - `hf endpoints deploy NAME --repo TEXT --framework TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --region TEXT --vendor TEXT` — Deploy an Inference Endpoint from a Hub repository. `[--namespace TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --revision TEXT --custom-image TEXT --health-route TEXT --port INTEGER --container-command TEXT --container-args TEXT --env TEXT --env-file TEXT --secrets TEXT --secrets-file TEXT --type [public|protected|authenticated|private] --format [auto|human|agent|json|quiet]]` - `hf endpoints describe NAME` — Get information about an existing endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf endpoints list` — Lists all Inference Endpoints for the given namespace. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf endpoints pause NAME` — Pause an Inference Endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf endpoints resume NAME` — Resume an Inference Endpoint. `[--namespace TEXT --fail-if-already-running --format [auto|human|agent|json|quiet]]` - `hf endpoints scale-to-zero NAME` — Scale an Inference Endpoint to zero. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf endpoints update NAME` — Update an existing endpoint. `[--namespace TEXT --repo TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --framework TEXT --revision TEXT --task TEXT --container-command TEXT --container-args TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --format [auto|human|agent|json|quiet]]` ### `hf extensions` — Manage hf CLI extensions. - `hf extensions exec NAME` — Execute an installed extension. - `hf extensions install REPO_ID` — Install an extension from a public GitHub repository. `[--force --format [auto|human|agent|json|quiet]]` - `hf extensions list` — List installed extension commands. `[--format [auto|human|agent|json|quiet]]` - `hf extensions remove NAME` — Remove an installed extension. `[--format [auto|human|agent|json|quiet]]` - `hf extensions search` — Search extensions available on GitHub (tagged with 'hf-extension' topic). `[--format [auto|human|agent|json|quiet]]` - `hf extensions update` — Update installed extension(s) to their latest version. `[--format [auto|human|agent|json|quiet]]` ### `hf jobs` — Run and manage Jobs on the Hub. - `hf jobs cancel JOB_ID` — Cancel a Job `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs hardware` — List available hardware options for Jobs `[--format [auto|human|agent|json|quiet]]` - `hf jobs inspect JOB_IDS` — Display detailed information on one or more Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs labels JOB_ID` — Update labels on a Job. Passing --label replaces all existing labels; passing --name alone keeps them. `[--name TEXT --label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs list` — List Jobs. `[--all --status [COMPLETED|CANCELED|ERROR|DELETED|SCHEDULING|RUNNING] --label TEXT --name TEXT --limit INTEGER --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs logs JOB_ID` — Fetch the logs of a Job. `[--follow --tail INTEGER --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs run IMAGE COMMAND` — Run a Job. `[--env TEXT --secrets TEXT --name TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --detach --expose INTEGER --ssh --resource-group-id TEXT --namespace TEXT]` - `hf jobs scheduled delete SCHEDULED_JOB_ID` — Delete a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled inspect SCHEDULED_JOB_IDS` — Display detailed information on one or more scheduled Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled labels SCHEDULED_JOB_ID` — Update labels on a scheduled Job. Passing --label replaces all existing labels; passing --name alone keeps them. `[--name TEXT --label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled list` — List scheduled Jobs `[--all --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled resume SCHEDULED_JOB_ID` — Resume (unpause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled run SCHEDULE IMAGE COMMAND` — Schedule a Job. `[--suspend --concurrency --env TEXT --secrets TEXT --name TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --expose INTEGER --resource-group-id TEXT --namespace TEXT]` - `hf jobs scheduled suspend SCHEDULED_JOB_ID` — Suspend (pause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled trigger SCHEDULED_JOB_ID` — Trigger a scheduled Job to run immediately (does not change the schedule). `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs scheduled uv run SCHEDULE SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--suspend --concurrency --image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --name TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --expose INTEGER --resource-group-id TEXT --namespace TEXT --with TEXT --python TEXT]` - `hf jobs ssh JOB_ID` — SSH into a running Job. `[--identity-file PATH --dry-run --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs stats` — Fetch the resource usage statistics and metrics of Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf jobs uv run SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --name TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --detach --expose INTEGER --ssh --resource-group-id TEXT --namespace TEXT --with TEXT --python TEXT]` - `hf jobs wait JOB_IDS` — Wait for one or more Jobs to reach a terminal state. `[--timeout TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]` ### `hf models` — Interact with models on the Hub. - `hf models card MODEL_ID` — Get the model card (README) for a model on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` - `hf models info MODEL_ID` — Get info about a model on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` - `hf models list` — List models on the Hub, or files in a model repo. `[--search TEXT --author TEXT --filter TEXT --pipeline-tag TEXT --gated --apps TEXT --num-parameters TEXT --inference-provider [baseten|cerebras|cohere|deepinfra|fal-ai|featherless-ai|fireworks-ai|groq|hf-inference|novita|nscale|openai|ovhcloud|publicai|replicate|scaleway|together|wavespeed|zai-org] --warm --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` ### `hf papers` — Interact with papers on the Hub. - `hf papers info PAPER_ID` — Get info about a paper on the Hub. `[--format [auto|human|agent|json|quiet]]` - `hf papers list` — List daily papers on the Hub. `[--date TEXT --week TEXT --month TEXT --submitter TEXT --sort [publishedAt|trending] --limit INTEGER --format [auto|human|agent|json|quiet]]` - `hf papers read PAPER_ID` — Read a paper as markdown. `[--format [auto|human|agent|json|quiet]]` - `hf papers search QUERY` — Search papers on the Hub. `[--limit INTEGER --format [auto|human|agent|json|quiet]]` ### `hf repos` — Manage repos on the Hub. - `hf repos branch create REPO_ID BRANCH` — Create a new branch for a repo on the Hub. `[--revision TEXT --type [model|dataset|space] --exist-ok --format [auto|human|agent|json|quiet]]` - `hf repos branch delete REPO_ID BRANCH` — Delete a branch from a repo on the Hub. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf repos cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` - `hf repos create REPO_ID` — Create a new repo on the Hub. `[--type [model|dataset|space] --sdk TEXT --template TEXT --private --public --protected --exist-ok --resource-group-id TEXT --region [us|eu] --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]` - `hf repos delete REPO_ID` — Delete a repo from the Hub. This is an irreversible operation. `[--type [model|dataset|space] --missing-ok --yes --format [auto|human|agent|json|quiet]]` - `hf repos delete-files REPO_ID PATTERNS` — Delete files from a repo on the Hub. `[--type [model|dataset|space] --revision TEXT --commit-message TEXT --commit-description TEXT --create-pr --format [auto|human|agent|json|quiet]]` - `hf repos duplicate FROM_ID` — Duplicate a repo on the Hub (model, dataset, or Space). `[--type [model|dataset|space] --private --public --protected --exist-ok --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]` - `hf repos list` — List all repos (models, datasets, spaces, buckets) with storage info. `[--namespace TEXT --type [model|dataset|space|bucket] --search TEXT --limit INTEGER --explore --format [auto|human|agent|json|quiet]]` - `hf repos move FROM_ID TO_ID` — Move a repository from a namespace to another namespace. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf repos settings REPO_ID` — Update the settings of a repository. `[--gated [auto|manual|false] --private --public --protected --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf repos tag create REPO_ID TAG` — Create a tag for a repo. `[--message TEXT --revision TEXT --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf repos tag delete REPO_ID TAG` — Delete a tag for a repo. `[--yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - `hf repos tag list REPO_ID` — List tags for a repo. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` ### `hf sandbox` — Run and manage sandboxes on Hugging Face Jobs. - `hf sandbox cp SRC DST` — Copy a file between the local machine and a sandbox (docker-style). `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox create` — Create a sandbox: a dedicated VM by default, or a cheap shared one with `--pool`. `[--pool TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --idle-timeout TEXT --env TEXT --secrets TEXT --env-file TEXT --secrets-file TEXT --volume TEXT --namespace TEXT --forward-hf-token --format [auto|human|agent|json|quiet]]` - `hf sandbox exec SANDBOX_ID COMMAND` — Run a command in a sandbox, streaming output. Exits with the command's exit code. `[--workdir TEXT --env TEXT --env-file TEXT --timeout FLOAT --namespace TEXT]` - `hf sandbox kill` — Terminate a sandbox, a whole shared host, or everything (--all). `[--all --yes --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox pool create` — Warm a pool: boot one host VM now, tagged so it can be found later by its pool id. `[--flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --per-host INTEGER RANGE --max-hosts INTEGER RANGE --idle-timeout TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox pool delete POOL_ID` — Terminate every host VM of a pool (and therefore all its sandboxes). `[--yes --namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox pool ls` — List running sandbox pools (grouped from their host VMs). `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox process kill SANDBOX_ID PID` — Stop a background process running in a sandbox. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox process ls SANDBOX_ID` — List the background processes running in a sandbox (started with `hf sandbox spawn`). `[--namespace TEXT --format [auto|human|agent|json|quiet]]` - `hf sandbox spawn SANDBOX_ID COMMAND` — Start a long-running command in the background and return its pid (don't wait). `[--workdir TEXT --env TEXT --env-file TEXT --namespace TEXT]` ### `hf skills` — Manage skills for AI assistants. - `hf skills add` — Install a Hugging Face skill for an AI assistant. `[--claude --global --dest PATH --force --format [auto|human|agent|json|quiet]]` - `hf skills list` — List available skills from the Hugging Face marketplace. `[--format [auto|human|agent|json|quiet]]` - `hf skills preview` — Print the generated `hf-cli` SKILL.md to stdout. `[--format [auto|human|agent|json|quiet]]` - `hf skills update` — Update installed Hugging Face marketplace skills. `[--claude --global --dest PATH --format [auto|human|agent|json|quiet]]` ### `hf spaces` — Interact with spaces on the Hub. - `hf spaces card SPACE_ID` — Get the Space card (README) for a Space on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` - `hf spaces dev-mode SPACE_ID` — Enable or disable dev mode on a Space. `[--stop --format [auto|human|agent|json|quiet]]` - `hf spaces hardware` — List available hardware options for Spaces. `[--format [auto|human|agent|json|quiet]]` - `hf spaces hot-reload SPACE_ID` — Hot-reload any Python file of a Space without a full rebuild + restart. `[--local-file PATH --skip-checks --skip-summary --format [auto|human|agent|json|quiet]]` - `hf spaces info SPACE_ID` — Get info about a space on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` - `hf spaces list` — List spaces on the Hub, or files in a space repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` - `hf spaces logs SPACE_ID` — Fetch the run or build logs of a Space. `[--build --follow --tail INTEGER --format [auto|human|agent|json|quiet]]` - `hf spaces pause SPACE_ID` — Pause a Space. `[--format [auto|human|agent|json|quiet]]` - `hf spaces restart SPACE_ID` — Restart a Space. `[--factory-reboot --format [auto|human|agent|json|quiet]]` - `hf spaces search QUERY` — Search spaces on the Hub using semantic search. `[--filter TEXT --sdk TEXT --include-non-running --description --limit INTEGER --format [auto|human|agent|json|quiet]]` - `hf spaces secrets add SPACE_ID` — Add or update secrets for a Space. `[--secrets TEXT --secrets-file TEXT --format [auto|human|agent|json|quiet]]` - `hf spaces secrets delete SPACE_ID KEY` — Remove a secret from a Space. `[--yes --format [auto|human|agent|json|quiet]]` - `hf spaces secrets list SPACE_ID` — List secrets for a Space. Secret values are write-only and not returned. `[--format [auto|human|agent|json|quiet]]` - `hf spaces settings SPACE_ID` — Update the settings of a Space. `[--sleep-time INTEGER --hardware [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --format [auto|human|agent|json|quiet]]` - `hf spaces ssh SPACE_ID` — SSH into a Space's Dev Mode container. `[--identity-file PATH --dry-run --auto --format [auto|human|agent|json|quiet]]` - `hf spaces templates` — List the available Space templates. `[--format [auto|human|agent|json|quiet]]` - `hf spaces variables add SPACE_ID` — Add or update environment variables for a Space. `[--env TEXT --env-file TEXT --format [auto|human|agent|json|quiet]]` - `hf spaces variables delete SPACE_ID KEY` — Remove an environment variable from a Space. `[--yes --format [auto|human|agent|json|quiet]]` - `hf spaces variables list SPACE_ID` — List environment variables for a Space. `[--format [auto|human|agent|json|quiet]]` - `hf spaces volumes delete SPACE_ID` — Remove all volumes from a Space. `[--yes --format [auto|human|agent|json|quiet]]` - `hf spaces volumes list SPACE_ID` — List volumes mounted in a Space. `[--format [auto|human|agent|json|quiet]]` - `hf spaces volumes set SPACE_ID` — Set (replace) volumes for a Space. `[--volume TEXT --format [auto|human|agent|json|quiet]]` - `hf spaces wait SPACE_ID` — Wait for a Space to finish building/starting. `[--timeout TEXT --format [auto|human|agent|json|quiet]]` ### `hf webhooks` — Manage webhooks on the Hub. - `hf webhooks create --watch TEXT` — Create a new webhook. `[--url TEXT --job-id TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]` - `hf webhooks delete WEBHOOK_ID` — Delete a webhook permanently. `[--yes --format [auto|human|agent|json|quiet]]` - `hf webhooks disable WEBHOOK_ID` — Disable an active webhook. `[--format [auto|human|agent|json|quiet]]` - `hf webhooks enable WEBHOOK_ID` — Enable a disabled webhook. `[--format [auto|human|agent|json|quiet]]` - `hf webhooks info WEBHOOK_ID` — Show full details for a single webhook. `[--format [auto|human|agent|json|quiet]]` - `hf webhooks list` — List all webhooks for the current user. `[--format [auto|human|agent|json|quiet]]` - `hf webhooks update WEBHOOK_ID` — Update an existing webhook. Only provided options are changed. `[--url TEXT --watch TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]` ## Common options - `--format` — Output format: `--format json` (or `--json`) or `--format table` (default). - `-q / --quiet` — Quiet output (one ID per line). - `--revision` — Git revision id which can be a branch name, a tag, or a commit hash. - `--token` — Use a User Access Token. Prefer setting `HF_TOKEN` env var instead of passing `--token`. - `--type` — The type of repository (model, dataset, or space). ## Mounting repos as local filesystems To mount Hub repositories or buckets as local filesystems — no download, no copy, no waiting — use `hf-mount`. Files are fetched on demand. GitHub: https://github.com/huggingface/hf-mount Install: `curl -fsSL https://raw.githubusercontent.com/huggingface/hf-mount/main/install.sh | sh` Some command examples: - `hf-mount start repo openai-community/gpt2 /tmp/gpt2` — mount a repo (read-only) - `hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data` — mount a bucket (read-write) - `hf-mount status` / `hf-mount stop /tmp/data` — list or unmount ## Tips - Use `hf --help` for full options, descriptions, usage, and real-world examples - Authenticate with `HF_TOKEN` env var (recommended) or with `--token` - Update the CLI with `hf update` (uses the correct command for the detected install method) --- ## 3. hf-cloud-aws-context-discovery - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-aws-context-discovery/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-aws-context-discovery/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-aws-context-discovery.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: hf-cloud-aws-context-discovery description: Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like "use my AWS account", "deploy to AWS", "use my profile", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first. --- # AWS Context Discovery Before doing any AWS work, read the user's local AWS config. Don't guess the region, and don't ask the user for things their config already answers. ## What to discover Run these at the start of the AWS work and remember the results for the rest of the session. ### 1. Active profile `AWS_PROFILE` env var, else `default`. If the user mentioned a profile in their prompt, that overrides. If the named profile doesn't exist in `~/.aws/config`, surface that clearly. ### 2. Region Resolution order — stop at the first one that produces a value: 1. Region the user explicitly named in this conversation 2. `AWS_REGION` env var 3. `AWS_DEFAULT_REGION` env var 4. `region` field on the active profile in `~/.aws/config` 5. Ask the user — but only after the first four have failed Do not fall back to `us-east-1` or any other hardcoded default. ### 3. Credentials, account ID, caller ARN ```bash aws sts get-caller-identity --profile --region ``` Three purposes in one call: confirms credentials are valid (stop if not), returns the `Account` ID (needed for ARN construction), returns the `Arn` of the caller. ### 4. Identify SSO / assumed-role principals The `Arn` field tells you what kind of principal this is. The pattern matters because it determines what IAM operations the caller can do. | ARN pattern | Type | IAM write capability | |---|---|---| | `arn:aws:iam:::user/` | IAM user | Depends on attached policies | | `arn:aws:sts:::assumed-role/AWSReservedSSO_<...>/` | **SSO assumed-role** | Typically **none** — can't create/modify IAM roles | | `arn:aws:sts:::assumed-role//` | Regular assumed-role | Depends on the role | **If the caller is SSO**, surface this immediately before later skills hit `iam:CreateRole` and fail: > Heads up: you're authenticated via SSO (`AWSReservedSSO__...`). SSO principals usually can't create IAM roles directly. If we need a SageMaker execution role, I'll look for an existing one first — if none exists, you'll need to ask whoever manages your AWS access to create one. This is the highest-leverage thing this skill does. Surfacing it now turns a confusing mid-deployment error into a five-second conversation. ## Commands to run ```bash # Effective profile and region (faster than parsing config files) aws configure list # Validate credentials and get identity aws sts get-caller-identity aws sts get-caller-identity --profile # if a profile was named ``` `aws configure list` handles env-var overrides and shows the resolved effective values. Prefer it over parsing `~/.aws/config` yourself. If you need to read raw config (e.g. to list profiles), `~/.aws/config` and `~/.aws/credentials` are plain INI files — read-only. ## What to report back One or two lines, not a wall of text: > Working with profile `my-profile` in `eu-west-1`, account `123456789012`. You're authenticated via SSO, so we'll need to use an existing IAM role rather than create one. Don't ask the user to confirm the region you just read from their config — they configured it; that is the confirmation. If something is wrong (credentials expired, profile doesn't exist, no region anywhere), stop and surface the specific error before continuing. --- ## 4. hf-cloud-python-env-setup - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-python-env-setup/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-python-env-setup/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-python-env-setup.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (3), referenced from this skill's directory: - `requirements.txt` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-python-env-setup/requirements.txt - `scripts/check_versions.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-python-env-setup/scripts/check_versions.py - `scripts/setup_env.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-python-env-setup/scripts/setup_env.py --- name: hf-cloud-python-env-setup description: 'Set up an isolated Python environment for SageMaker / AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to "set up the environment". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.' --- # Python Environment Setup for SageMaker Most SageMaker deployment failures that look like AWS problems are actually Python environment problems: wrong Python version, broken dependency resolution, stale SDK that doesn't know about a current API. This skill makes env setup boring and correct. ## Core rules 1. **Never use the system Python.** Always work inside an isolated environment. 2. **Pin the Python version, not the package versions.** Use 3.10, 3.11, or 3.12. Avoid 3.13+ — ML libraries lag on wheel availability and dependency resolution breaks in confusing ways. 3. **Install the latest of each package.** Don't defensively pin `boto3` or `awscli`. Newer ones have current API surfaces and security fixes. Only pin if the user explicitly requires a specific version. 4. **Check installed versions correctly.** Use `importlib.metadata.version("package-name")`, never `module.__version__`. The latter is inconsistent across packages. 5. **The bundled scripts use `boto3` directly.** The SageMaker Python SDK is a valid alternative — see "boto3 vs the SageMaker SDK" below. ## boto3 vs the SageMaker SDK The bundled deploy scripts (`deploy.py`, `deploy_async.py`, `teardown.py`) use `boto3` directly and read image URIs from [AWS's published Deep Learning Containers catalog](https://aws.github.io/deep-learning-containers/reference/available_images/). That fits this workflow's explicit-stages design — each skill produces a concrete value (region, role ARN, image URI) that the next one consumes — and `boto3` is the stable underlying API client. The SageMaker Python SDK (v3) is fine to use when the user prefers it or their project already does. Since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960) (June 2026), `ModelBuilder` auto-routes HuggingFace models to the current containers (text-generation → HuggingFace vLLM, multimodal → vLLM-Omni, embeddings → TEI). Don't avoid the SDK over stale-image or wrong-container concerns — that routing is fixed. Two specific SDK cases that still need care: - **Generative rerankers**: the SDK routes the `text-ranking` task to TEI unconditionally, which is wrong for causal-LM rerankers like Qwen3-Reranker — those need vLLM (see `hf-cloud-serving-image-selection`). Pass the container explicitly for these models. - **SSO assumed-role credentials**: v3 has had credential-resolution regressions in `ModelTrainer` / `FrameworkProcessor` under SSO profiles. If SDK calls fail with credential errors while `aws sts get-caller-identity` succeeds in the same shell, suspect this rather than your AWS config. If you use the SDK, install it into the isolated env like everything else (`.venv/bin/python -m pip install sagemaker`). The bundled scripts don't require it. ## How to set up The fastest path is the bundled script — it's Python, so it runs the same on Windows, macOS, and Linux: ```bash python3 scripts/setup_env.py # macOS / Linux python scripts/setup_env.py # Windows (PowerShell / cmd) ``` This script detects `uv` and uses it if available (faster), falls back to the stdlib `venv` module, creates `.venv/` with Python 3.12 (override: `python3 setup_env.py .venv 3.11`), refuses unsupported Python versions, installs from the bundled `requirements.txt`, and is idempotent. It also prints the correct interpreter path for the host OS (see below). Manual equivalent: ```bash # Preferred: uv uv venv --python 3.12 .venv uv pip install --python .venv/bin/python --upgrade boto3 awscli # Windows: .venv\Scripts\python.exe # Fallback: stdlib venv python3.12 -m venv .venv .venv/bin/python -m pip install --upgrade pip boto3 awscli ``` After setup, **invoke the env's Python explicitly** rather than activating the venv. The interpreter path differs by platform: ```bash .venv/bin/python deploy.py # macOS / Linux .venv\Scripts\python.exe deploy.py # Windows ``` This works the same in scripts, interactive shells, and agent tool calls. The rest of this skill writes `.venv/bin/python` for brevity — on Windows substitute `.venv\Scripts\python.exe`. ## Verifying ```bash .venv/bin/python scripts/check_versions.py ``` Prints versions of `boto3`, `botocore`, `awscli`. Uses `importlib.metadata.version()` so it works on every package, including ones without `__version__`. Pass arbitrary names: `... check_versions.py transformers huggingface_hub`. ## Deployment-specific extras Default `requirements.txt` covers SageMaker orchestration. Some deployments need extras (`huggingface_hub` for model inspection, `transformers` for tokenizer validation). Add these to a deployment-specific requirements file in the project, install with the env's Python, don't pin unless there's a reason. ## Common pitfalls **Mysterious `pip install` resolution errors** Almost always Python 3.13+ trying to install packages without wheels yet, or installing into a polluted system Python. Recreate at 3.12: delete `.venv` and re-run `python3 setup_env.py .venv 3.12` (the script recreates the env when the version doesn't match, so you can also just re-run it). **`pip install` succeeded but the script says "module not found"** You installed into a different interpreter than the one running the script. Always invoke Python explicitly: `.venv/bin/python -m pip install ...` and `.venv/bin/python deploy.py`. **Inline `python -c "..."` one-liners fail in PowerShell** PowerShell's quoting rules mangle nested/escaped quotes in inline Python. Don't debug the quoting — write the snippet to a small `.py` file and run that. (All bundled helpers are files for exactly this reason.) **boto3 call fails with "unknown parameter"** Your boto3 is older than the API surface. Upgrade with `.venv/bin/python -m pip install --upgrade boto3`. Don't downgrade the script to match an old version. **`sagemaker` (the SDK) installed but the bundled scripts fail** The bundled scripts don't use the SDK — they only need `boto3`/`awscli` from `requirements.txt`. Installing `sagemaker` alongside is harmless, but it doesn't replace the requirements install. --- ## 5. hf-cloud-sagemaker-deployment-planner - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-deployment-planner/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-deployment-planner/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-deployment-planner.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: hf-cloud-sagemaker-deployment-planner description: 'Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like "deploy a model", "host this LLM on AWS", "serve this embedding model", "deploy a reranker", "deploy a text-to-image / diffusion model", "host this for async inference", "create an endpoint", "serve my fine-tuned model", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. "I just want to get this running on AWS, you figure it out"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image / diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.' --- # SageMaker Deployment Planner You are helping a user deploy a model to Amazon SageMaker. Most users invoking this skill want the model deployed with reasonable defaults, in as few questions as possible. Ask only what you need, recommend a pathway honestly, and hand off to the specialized skills. ## Workflow phases 1. **Discovery** — what is being deployed and what are the constraints (this skill) 2. **Pathway selection** — real-time / serverless / async / batch / Bedrock CMI (this skill) 3. **Context preflight** — `hf-cloud-aws-context-discovery`, then `hf-cloud-python-env-setup` 4. **IAM preflight** — `hf-cloud-sagemaker-iam-preflight` 5. **Image selection** — `hf-cloud-serving-image-selection` 6. **Deployment** — `hf-cloud-sagemaker-production-defaults` Phases 1–2 are this skill's job. The others activate when their patterns match. ## Discovery: ask only what you need You will eventually need to know: - **What model**: HuggingFace ID, S3 path to artifacts, or model name. If the user is vague ("the model I fine-tuned"), ask for the artifact location. - **Model type**: text-generation LLM, embedding/reranker, or other (classifier, NER, etc.). This determines the serving stack — usually inferable from the model name (anything ending in `-embed-*`, starting with `BAAI/bge-`, `sentence-transformers/*` etc. is embeddings; chat/instruct models are LLMs). Only ask if it's genuinely ambiguous. - **Traffic shape**: roughly how often will this be called? - **Latency tolerance**: interactive, near-real-time, or async? - **Cost sensitivity**: ask only if the user signals it or the traffic pattern is ambiguous. Region comes from `hf-cloud-aws-context-discovery` — don't ask unless the user volunteers it. Do **not** front-load all of these. A common minimal set is just: *what model, and roughly how often will it be called?* The model name usually settles the model-type question. That alone is often enough to narrow the pathway to two candidates. If the user already told you something, don't ask again. ## Pathway selection | Pathway | When it fits | When it does not | |---|---|---| | **Real-time endpoint** | Steady traffic, sub-second to few-second latency, always-on | Very spiky or very sparse traffic (wastes money on idle) | | **Real-time, scale to zero** | Sparse or scheduled traffic, dev/test endpoints, and a client that tolerates a ~9 min first request after idle | Any interactive SLA: every request during the wake fails with a 400 | | **Serverless inference** | Spiky/intermittent, tolerates cold starts (~10s+), simpler models | LLMs above a few B params (memory/cold-start limits), strict SLAs | | **Async inference** | Long inference (>60s), large payloads, queue-friendly | Interactive synchronous calls | | **Batch transform** | Offline scoring over a dataset | Anything online or interactive | | **Bedrock Custom Model Import** | Wants Bedrock-compatible API, supported base family, weights only | Custom inference logic, unsupported architectures | For LLMs, **real-time endpoints are the default** unless traffic is explicitly spiky/sparse or inference is long-running. Serverless looks attractive for "low traffic" cases but most LLMs exceed its memory limits. For **embeddings**, real-time is again the default — but CPU instances are usually the right choice (much cheaper, fast enough for most embedding workloads). Don't reflexively recommend GPU instances for embedding models; ask `hf-cloud-serving-image-selection` to consider CPU variants if the model is small (<1B params) and traffic is moderate. For **text-to-image, video generation, or other long-inference workloads** (>30s per request) where traffic is also bursty: async inference is the right answer. It supports genuine scale-to-zero between batches and queues requests via S3, so you don't pay for idle GPU. `hf-cloud-sagemaker-production-defaults` has a dedicated `deploy_async.py` for this. Real-time, real-time scale-to-zero, and async are the three scripted pathways (`deploy.py`, `deploy_ic.py`, `deploy_async.py` in `hf-cloud-sagemaker-production-defaults`). Serverless, batch transform, and Bedrock Custom Model Import are not currently scripted — for those, hand the user off with a brief explanation rather than trying to deploy them through this workflow. **Scale to zero, real-time or async?** Both reach zero and both make the first request after idle slow. Pick async when one inference can exceed the 60s `InvokeEndpoint` limit, when payloads are large, or when the client can accept an S3 result instead of a synchronous response. Pick real-time scale-to-zero when the client needs a normal synchronous HTTP response and can retry through the wake. Real-time scale-to-zero needs inference components; the plain real-time pathway cannot go below one instance. If two pathways are both reasonable, say so in one sentence each and pick one. Don't bury the recommendation in options. ## Instance selection: check quota before recommending Endpoint quotas are per instance type, per region, and default to **0** for GPU types in many accounts. Recommending an instance the account can't launch wastes a full deploy cycle on `ResourceLimitExceeded`. Check first: ```bash aws service-quotas list-service-quotas --service-code sagemaker --region \ --query "Quotas[?contains(QuotaName, 'for endpoint usage') && Value > \`0\`].[QuotaName, Value]" \ --output table ``` If the type you want isn't in the result, recommend one that is — or tell the user to request an increase (hours to days) *before* creating anything. If the call itself is denied, say so once and continue. The quota check is an optimization, not a gate: the deployment surfaces the real limit as `ResourceLimitExceeded`. Never stop the workflow, and never ask the user to change IAM, for a preflight check. GPU family notes for the common 24 GB tier: - `ml.g5.*` (A10G) and `ml.g6.*` (L4) both work with current vLLM images when the gpu-3-1 AMI is set (see `hf-cloud-serving-image-selection`). g6 is the newer generation and slightly cheaper per hour; g5 has roughly double the memory bandwidth, which usually means better LLM token throughput. Pick whichever has quota; when both do, either is defensible — g5 for throughput, g6 for cost. - `ml.g6e.*` (L40S, 48 GB) when the model doesn't fit in 24 GB. Once you have enough to recommend, state it plainly: > Based on what you've told me, I'd recommend a real-time endpoint on `ml.g5.xlarge`. The model is small enough that this is cost-effective, and your traffic pattern is steady enough that you won't be paying for idle. Alternative: serverless would be cheaper if traffic dries up for hours at a time, but Qwen3-0.6B is at the edge of serverless memory limits and cold starts would be 15–30s. Want me to proceed with the real-time endpoint? Then wait for confirmation. The user should know what they're about to spend money on before you create anything. The plan lives in the conversation — don't generate `plan.yaml` or similar artifacts unless explicitly asked. ## Style - Users invoking this skill are deferring to the agent because they don't want to do AWS plumbing. Match that energy: efficient, not exhaustive. - One round of clarifying questions is usually enough. Three rounds is interrogation. - When you don't know something specific (current image URI, SDK API surface, quotas), check it rather than guess. Other skills handle the "how to check" details. - If the user pushes back on a recommendation, accept it. They know their constraints better than you do. --- ## 6. hf-cloud-sagemaker-iam-preflight - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-iam-preflight/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-iam-preflight/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-iam-preflight.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `references/minimum-permissions.json` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-iam-preflight/references/minimum-permissions.json - `references/trust-policy.json` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-iam-preflight/references/trust-policy.json - `scripts/check_role.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-iam-preflight/scripts/check_role.py - `scripts/create_role.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-iam-preflight/scripts/create_role.py --- name: hf-cloud-sagemaker-iam-preflight description: 'Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.' --- # SageMaker IAM Preflight Every SageMaker resource needs an **execution role** — the IAM role SageMaker assumes to read model artifacts from S3, pull serving containers from ECR, and write logs. Most deployments fail here because the script tried to create a new role without checking if a usable one already existed, then blew up because the caller is an SSO principal. This skill encodes the right order: discover, validate, only create if necessary. ## Running the helpers (cross-platform) The helpers are Python so they run identically on Windows, macOS, and Linux: ```bash python3 scripts/check_role.py # macOS / Linux python scripts/check_role.py # Windows (PowerShell / cmd) ``` **Run them from the shell where the AWS CLI already works** — i.e. wherever `aws sts get-caller-identity` succeeds. The script shells out to that same `aws` binary and inherits the shell's profile, region, SSO session, proxy, and credential chain. > **Windows / WSL / Git Bash caveat.** Do **not** invoke these through a Bash shim (WSL, Git Bash, MSYS) on Windows. Those Bash environments frequently do **not** share the Windows AWS config, credentials, SSO sessions, environment variables, or proxy settings — so `aws sts get-caller-identity` fails inside Bash even when it works natively in PowerShell. (This is exactly why the old `.sh` helpers failed on Windows and were replaced with Python.) If you're in PowerShell, run `python ...\check_role.py` directly in PowerShell. If the helper still can't see your identity, run the same discovery natively (see "Native AWS CLI equivalent" below) in the shell where `aws sts get-caller-identity` returns your ARN. ## Order of operations ### Step 1 — Did the user provide a role? Validate that one specifically: ```bash python3 scripts/check_role.py "" ``` On success it prints the ARN to stdout (exit 0). On failure it logs why on stderr. Don't try to silently fix a broken role — surface the problem. ### Step 2 — Discover existing roles ```bash python3 scripts/check_role.py ``` Lists roles matching common SageMaker patterns (`AmazonSageMaker-ExecutionRole-*`, `SageMakerExecutionRole*`, etc.), **ranks by last-used date** (most recent first), validates trust policy in that order, returns the first usable ARN. Most accounts that have used SageMaker before already have one. Why rank by last-used: in accounts with multiple roles (auto-generated 2021 role + manual project role + etc.), the alphabetically-first one is rarely the actively-maintained one. The most-recently-used role is more likely to have current policies — including cross-account ECR pull. The script prints the ranking so you can see which got picked. IAM frequently reports **no** `RoleLastUsed` at all (tracking only covers recent activity). When every candidate ties at "never used", the script falls back to **newest creation date** — a newer role is more likely to have current policies than a 2021 leftover. ### Step 3 — Create, only if discovery found nothing **If the user can create** (has IAM permissions): ```bash python3 scripts/create_role.py "" "" ``` Second arg scopes S3 access to a specific bucket. Omit if unknown; script warns and the user can update the policy later. **If the user cannot create** (SSO principal — `hf-cloud-aws-context-discovery` will have flagged this): Stop and surface this clearly. Don't retry alternative IAM operations hoping one works: > I can't find an existing SageMaker execution role, and you're authenticated via SSO so you can't create one directly. Please either: > - Ask your AWS admin for a SageMaker execution role ARN, or > - Have them grant your SSO permission set `iam:CreateRole`, `iam:AttachRolePolicy`, `iam:PutRolePolicy` Specific instructions get unblocked fast; vague "permission denied" messages don't. ## What "validated" means A role is usable when (1) it exists, (2) its trust policy allows `sagemaker.amazonaws.com` to `sts:AssumeRole` — see `references/trust-policy.json` for the canonical form. `check_role.py` verifies these two. It does **not** deep-check permissions because comprehensive analysis is expensive (`iam:SimulatePrincipalPolicy` per action) and most existing SageMaker roles are over-permissioned via `AmazonSageMakerFullAccess`. If you suspect a permissions issue at deploy time, the deployment error will tell you which action was denied — fix it then, not preemptively. ## Minimum permissions `references/minimum-permissions.json` covers what SageMaker actually needs: - `s3:GetObject` + `s3:ListBucket` on the model artifact bucket - ECR pull permissions - CloudWatch logs and metrics Layered on top of `AmazonSageMakerFullAccess` (attached by `create_role.py`). Replace `REPLACE_WITH_MODEL_BUCKET` in the template with the actual bucket name — `create_role.py` does this automatically when given a bucket as its second argument. ## Native AWS CLI equivalent (fallback) If the Python helper can't run or can't see your identity (rare — usually a broken PATH or running under a Bash shim that lacks AWS context), do the same preflight by hand in the shell where `aws sts get-caller-identity` works. The logic is just AWS CLI calls; the helper exists only to bundle and rank them. PowerShell: ```powershell # 1. List candidate SageMaker roles aws iam list-roles --query "Roles[?contains(RoleName,'SageMaker') || contains(RoleName,'sagemaker')]" --output json # 2. For each candidate, confirm the trust policy allows sagemaker.amazonaws.com aws iam get-role --role-name --query "Role.AssumeRolePolicyDocument" --output json # 3. Prefer the most-recently-used role with SageMaker-execution naming # (LastUsedDate is often None for every role — then prefer newest CreateDate) aws iam get-role --role-name --query "Role.[RoleLastUsed.LastUsedDate, CreateDate]" --output text ``` Pick the most-recently-used role whose trust policy contains `sagemaker.amazonaws.com`. Use the resulting ARN exactly as if `check_role.py` had returned it. Bash/macOS/Linux use the same commands. --- ## 7. hf-cloud-sagemaker-production-defaults - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-sagemaker-production-defaults/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-sagemaker-production-defaults.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `references/deployment-template.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/references/deployment-template.md - `scripts/_common.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/_common.py - `scripts/deploy_async.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/deploy_async.py - `scripts/deploy_ic.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/deploy_ic.py - `scripts/deploy.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/deploy.py - `scripts/invoke_endpoint.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/invoke_endpoint.py - `scripts/teardown.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-sagemaker-production-defaults/scripts/teardown.py --- name: hf-cloud-sagemaker-production-defaults description: 'Create a SageMaker endpoint (real-time, real-time scale-to-zero, or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints, deploy_ic.py for real-time endpoints that scale to zero instances via inference components, and deploy_async.py for async endpoints (also scale-to-zero). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.' --- # SageMaker Production Defaults The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras. By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment. ## What gets created For every endpoint, the skill creates these as a unit: 1. **SageMaker Model** — image + env vars + execution role + S3 artifacts 2. **Endpoint config** — instance type, initial count, optional data capture 3. **Endpoint** — the real-time endpoint serving inference 4. **Autoscaling target + policy** — target tracking on invocations per instance 5. **CloudWatch alarms** — latency, errors, platform overhead An inference-component deployment (`deploy_ic.py`) creates the same set with two changes: the endpoint config carries the execution role and `ManagedInstanceScaling`, and an **inference component** carries the model. Its autoscaling target is the component, not the variant. Data capture (logging requests/responses to S3) is **off by default** — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with `--enable-data-capture`. All resources get a consistent tag set including `CreatedBy=agentic-deploy-skills` for later cleanup. Defaults and reasoning in `references/deployment-template.md`. ## Running the deployment For a text-generation LLM (vLLM): ```bash python scripts/deploy.py \ --model-name qwen3-medical \ --image-uri "$IMAGE_URI" \ --inference-ami-version "$AMI" \ --role-arn "$ROLE_ARN" \ --instance-type ml.g5.xlarge \ --region "$REGION" \ --env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \ --env SM_VLLM_HOST=0.0.0.0 \ --env SM_VLLM_TRUST_REMOTE_CODE=true \ --env SM_VLLM_MAX_MODEL_LEN=4096 ``` For an embedding model (TEI, often on CPU): ```bash python scripts/deploy.py \ --model-name bge-large-embeddings \ --image-uri "$IMAGE_URI" \ --role-arn "$ROLE_ARN" \ --instance-type ml.c6i.2xlarge \ --region "$REGION" \ --env HF_MODEL_ID=BAAI/bge-large-en-v1.5 ``` Note: TEI deployments **do not** need `--inference-ami-version`. That flag is vLLM-specific. TEI env vars are also simpler (`HF_MODEL_ID` instead of `SM_VLLM_*`, no host or trust-remote-code to configure). Where each value comes from: | Parameter | Source | |---|---| | `--image-uri` | `hf-cloud-serving-image-selection` — agent reads from the AWS DLC catalog page | | `--inference-ami-version` | `hf-cloud-serving-image-selection` — required for vLLM tags containing cu130+ | | `--role-arn` | `hf-cloud-sagemaker-iam-preflight` (`check_role.py`) | | `--region` | `hf-cloud-aws-context-discovery` | | `--instance-type` | User input or planner recommendation | | `--env` | Model-specific; see `hf-cloud-serving-image-selection` for required `SM_VLLM_*` vars | | `--model-s3-uri` | Optional — S3 path to model artifacts; omit if loading from HF Hub | The script creates resources in order with error handling, waits for `InService` (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint/config/model names for downstream scripting. The scripts ship with this skill. If the installed copy is missing the `scripts/` directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description. **Cold-start expectation**: when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. `deploy.py` waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (`--model-s3-uri`) cuts this and removes the Hub dependency. ## InService is not success — smoke-test before declaring victory `InService` only means the container answered `/ping`. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always: 1. **One real invocation.** - Real-time: `invoke_endpoint.py` (below) with a minimal payload; require an HTTP 200 with a sane body. - Async: upload one input to S3, call `invoke-endpoint-async`, poll the output URI for a few minutes (see "Invoking async endpoints"). A result object = success; an object at the failure URI, or nothing appearing, = broken. 2. **Scan the endpoint logs for worker-crash markers** — catches the crash-loop case even when the smoke request merely times out: ```bash aws logs filter-log-events \ --log-group-name /aws/sagemaker/Endpoints/ \ --filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \ --region --max-items 5 ``` Inference-component deployments log to `/aws/sagemaker/InferenceComponents/` instead. `deploy_ic.py` scans that group automatically while it waits. **General rule for denied diagnostics**: when a read-only call the workflow uses for diagnosis is denied (a restricted role without `logs:FilterLogEvents`, `servicequotas:ListServiceQuotas`, and so on), say so in one line and carry on with the checks that do work. Never block a deployment on a permission needed only for diagnosis, and never read a denied call as evidence that nothing is wrong. Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status. ## Testing a real-time endpoint Once the endpoint is `InService`, test it with the bundled helper. It is cross-platform and **BOM-safe** — use it instead of hand-writing a payload file and calling `invoke-endpoint` directly: ```bash # macOS / Linux python3 scripts/invoke_endpoint.py \ --endpoint-name \ --payload '{"inputs": "Hello"}' \ --region "$REGION" ``` ```powershell # Windows (PowerShell) python scripts\invoke_endpoint.py ` --endpoint-name ` --payload-file payload.json ` --region $REGION ``` It accepts either `--payload ''` (inline) or `--payload-file `, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout. ### The UTF-8 BOM gotcha (Windows) If you write the request payload yourself on Windows, **do not** use `Set-Content -Encoding UTF8` — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 `ModelError`: ``` Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0) ``` This is **not** a model, endpoint-health, or image problem — only the file encoding of the request body. `invoke_endpoint.py` avoids it entirely (it even strips a BOM from a `--payload-file` that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8: ```powershell # BOM-free UTF-8 — use this [System.IO.File]::WriteAllText((Resolve-Path "payload.json"), $json, [System.Text.UTF8Encoding]::new($false)) aws sagemaker-runtime invoke-endpoint ` --endpoint-name ` --content-type application/json ` --body fileb://payload.json ` --region $REGION ` response.json ``` **Fallback:** if any invocation fails with `Unexpected UTF-8 BOM`, rewrite the payload as BOM-free UTF-8 (or re-run via `invoke_endpoint.py`) and retry once before treating the endpoint or model as broken. ### Invoking a generative reranker (vLLM) Generative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by `hf-cloud-serving-image-selection`) are causal LMs scored by their first generated token, not chat models. Use the **completions API with a raw `prompt`**, not the messages/chat API: chat templating does not reliably honor `chat_template_kwargs` such as `{"enable_thinking": false}`, and a wrong template silently returns near-identical scores for every query–document pair instead of erroring. Payload shape (Qwen3-Reranker's expected format — substitute `{query}` / `{document}`): ```json { "prompt": "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n: Given a web search query, retrieve relevant passages that answer the query\n: {query}\n: {document}<|im_end|>\n<|im_start|>assistant\n\n\n\n\n", "max_tokens": 1, "temperature": 0, "logprobs": 20 } ``` The trailing `<|im_start|>assistant\n\n\n\n\n` suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes/no judgment. Score from the returned logprobs: `P("yes") / (P("yes") + P("no"))`. Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect <0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken. The same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat. ### Picking the image URI The agent reads the image URI from AWS's [Deep Learning Containers catalog](https://aws.github.io/deep-learning-containers/reference/available_images/) — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute `` with the deployment region, and pass to `deploy.py --image-uri`. For vLLM images specifically (both `huggingface-vllm` and the AWS `vllm` fallback), also check the tag's CUDA version: ```bash # Example: HuggingFace vLLM 0.21.0 from the catalog IMAGE_URI="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04" # cu130 tag → must pass --inference-ami-version python deploy.py --image-uri "$IMAGE_URI" \ --inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 \ ... ``` For tags with `cu129` or lower, omit `--inference-ami-version`. See `hf-cloud-serving-image-selection` for the full vLLM AMI lookup table and the env-var requirements for each image family. ## Scale to zero for real-time endpoints A real-time endpoint reaches zero instances only when it hosts **inference components**. The variant-scoped target that `deploy.py` registers cannot go below one instance. `deploy_ic.py` builds the component-based shape instead. Use it when traffic is sparse or scheduled, and the client tolerates a multi-minute first request. Do not use it for interactive traffic with an SLA: the wake takes minutes, and every request during the wake fails. ```bash python scripts/deploy_ic.py \ --model-name qwen3-scale-to-zero \ --image-uri "$IMAGE_URI" \ --inference-ami-version "$AMI" \ --role-arn "$ROLE_ARN" \ --instance-type ml.g5.xlarge \ --region "$REGION" \ --env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \ --env SM_VLLM_HOST=0.0.0.0 \ --env SM_VLLM_TRUST_REMOTE_CODE=true \ --env SM_VLLM_MAX_MODEL_LEN=4096 ``` ### How it differs from deploy.py | Piece | Model-based (`deploy.py`) | Component-based (`deploy_ic.py`) | |---|---|---| | Execution role | on the Model | on the **endpoint config** (`ExecutionRoleArn`) | | Model reference | `ProductionVariants[].ModelName` | `InferenceComponent.Specification.ModelName`; the variant has no `ModelName` | | Instance floor | `InitialInstanceCount`, min 1 | `ManagedInstanceScaling {Status: ENABLED, MinInstanceCount: 0}` | | Scaling target | `endpoint//variant/AllTraffic`, `sagemaker:variant:DesiredInstanceCount` | `inference-component/`, `sagemaker:inference-component:DesiredCopyCount` | | Scaling metric | `SageMakerVariantInvocationsPerInstance` (20/min) | `SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution` (5 concurrent/copy) | | Wake from zero | not applicable | step policy + `NoCapacityInvocationFailures` alarm | | Invocation | endpoint name | endpoint name **plus** `InferenceComponentName` | `InferenceAmiVersion` still belongs on the variant, and it coexists with `ManagedInstanceScaling` (verified on `cu130` + `ml.g5.xlarge`). Four pieces make zero work, and all four are required. Target tracking cannot leave zero, because it cannot divide by zero copies. Drop the step policy or its alarm and the endpoint scales to zero once, then never answers again. `deploy_ic.py` wires all four. ### Measured behaviour `Qwen/Qwen3-0.6B` from the Hub, `ml.g5.xlarge`, `us-east-1`, July 2026: | Step | Time | |---|---| | Endpoint `InService` (it starts empty, no model loads) | 4 min | | Component `InService` (Hub download + vLLM boot + CUDA graphs) | +6 min | | Idle to 0 copies | 11 min after the last request | | 0 copies to 0 instances | +12 min | | First request at zero → HTTP 400 `has no capacity` | immediate | | `NoCapacityInvocationFailures` alarm → ALARM | +67 s | | Step policy raises desired copies and instances to 1 | +1 min 42 s | | HTTP 200 | **+9 min 24 s** | Scale-in is not tunable through this skill: Application Auto Scaling creates the AlarmLow itself with a 10 s period and 90 evaluation periods, so 15 minutes of idle datapoints are required before it fires. Pre-stage the weights and pass `--model-s3-uri` when wake time matters. The weights are downloaded again on **every** wake, so the Hub download sits on the critical path of the first request after each idle period. ### Sizing the component `ComputeResourceRequirements` is a scheduling reservation, not a cap. A component that requests 1024 MB runs vLLM with several GB of host memory without trouble. The schedulable pool is far smaller than the instance memory. On `ml.g5.xlarge` (16 GiB) the scheduler **accepts 1024 MB and rejects 4096 MB**. Over-asking gives an instant, confusing failure: ``` There is not enough hardware resources on the instances for this endpoint to create a copy of the inference component. ``` That message appears even when the endpoint has a healthy instance. Treat it as "the request is too large", not "add instances". Start at the 1024 MB default and raise it only when several components share one instance. `--accelerator-devices` must match `SM_VLLM_TENSOR_PARALLEL_SIZE` for multi-GPU models. ### Invoking and testing Pass the component name, and allow for the wake: ```bash python3 scripts/invoke_endpoint.py \ --endpoint-name \ --inference-component-name \ --payload '{"prompt": "hello", "max_tokens": 16}' \ --wait-for-capacity 900 --region "$REGION" ``` The 400 `has no capacity` error is the wake signal, not a fault: it publishes the metric that triggers the step policy. With `--wait-for-capacity` the helper retries every 30 s until a copy serves the request. Without it, a cold endpoint always looks broken. ### Teardown order `teardown.py` handles both shapes, but the order is load-bearing: 1. alarms (`-*` and `-*`) 2. scaling policies and the scalable target, on the component resource id 3. **inference components** 4. endpoint, endpoint config, model Two behaviours make this necessary: - **`delete-endpoint` does not delete the components.** They survive, keep reporting `InService`, and block a new component with the same name. Always delete components first. - **Component deletion is refused during transient states** — `CREATE_IN_PROGRESS` while the container boots, and `UPDATE_RC_IN_PROGRESS` while a scaling action changes the copy count. The script retries every 15 s for 15 min; a teardown right after a scaling event legitimately takes several minutes. The `TargetTracking-inference-component/-AlarmHigh|Low` alarms belong to Application Auto Scaling. Deleting the policy removes them, so the script does not touch them (verified: no alarms remain after teardown). ## Async inference deployments For long-running inferences (>60s), large payloads, or workloads that are bursty/sparse enough to benefit from scale-to-zero, use `deploy_async.py` instead of `deploy.py`. Async supports `MinCapacity=0` on the variant itself. Real-time endpoints also reach zero, but only through inference components — see "Scale to zero for real-time endpoints" below. Async remains the right choice when a single inference exceeds the 60s `InvokeEndpoint` response limit. ```bash python scripts/deploy_async.py \ --model-name flux-text-to-image \ --image-uri "$IMAGE_URI" \ --role-arn "$ROLE_ARN" \ --instance-type ml.g5.2xlarge \ --region "$REGION" \ --output-s3-uri s3://my-bucket/async-output/ \ --env HF_MODEL_ID=black-forest-labs/FLUX.1-dev ``` Required extras over `deploy.py`: - `--output-s3-uri` — where async results land (results are not returned synchronously) Optional async-specific flags: - `--failure-s3-uri` — separate path for failed invocations - `--success-sns-topic`, `--error-sns-topic` — get notified when async results are ready or fail - `--min-capacity 0` (the default) — scale to zero between batches - `--backlog-per-instance-target N` — target queue depth per instance (default 5) - `--max-concurrent-invocations-per-instance N` — default 4 ### How scale-to-zero works The async script registers **two** autoscaling policies on the variant: 1. **Target-tracking** on `ApproximateBacklogSizePerInstance` — handles ongoing scaling between min and max 2. **Step-scaling** triggered by a `HasBacklogWithoutCapacity` CloudWatch alarm — handles `0→1` wake-from-zero Both are needed. Target-tracking alone cannot transition from zero (it can't divide by zero instances), so without the step policy the endpoint comes up, scales to zero after the first batch, and never wakes again. The script wires this up automatically. ### Async alarms The script creates three CloudWatch alarms: - `ApproximateBacklogSize > 50` — queue is building faster than capacity can drain it - `InvocationsFailed > 5` — repeated processing failures - `HasBacklogWithoutCapacity` — drives the wake-from-zero policy (not a notification alarm; its action is the step-scaling policy, not the SNS topic) If you pass `--sns-alarm-topic `, the first two notify on that topic. The wake alarm always points at the step policy. ### Invoking async endpoints Async endpoints aren't called synchronously. You upload the input to S3, call `invoke-endpoint-async` with the S3 input location, and SageMaker writes the result to your `--output-s3-uri` when done: ```bash # Upload your input first aws s3 cp input.json s3://my-input-bucket/job1/input.json # Invoke aws sagemaker-runtime invoke-endpoint-async \ --endpoint-name \ --input-location s3://my-input-bucket/job1/input.json \ --content-type application/json \ --region # Poll for the result at your output URI aws s3 cp s3://my-bucket/async-output/.out result.json ``` The same UTF-8 BOM caveat applies to the `input.json` you upload (see "The UTF-8 BOM gotcha" above) — if you build it on Windows, write it as BOM-free UTF-8 or the container's JSON parser will reject it. Teardown works the same as real-time: `python3 scripts/teardown.py ` (the teardown script discovers policies and alarms by name prefix, so it handles both deployment modes). ## Defaults at a glance | Setting | Default | Override | |---|---|---| | Initial instance count | 1 | `--initial-instance-count` | | Autoscaling min / max | 1 / 4 | `--min-capacity`, `--max-capacity` | | Autoscaling target | 20 invocations/min/instance | `--target-invocations-per-instance` | | Data capture | disabled (opt-in) | `--enable-data-capture` | | CloudWatch alarms | 3 alarms | `--no-alarms` | | SNS notification | none (alarms created but won't notify) | `--sns-alarm-topic ` | | Environment tag | `dev` | `--environment` | | InferenceAmiVersion | none (SageMaker default) | `--inference-ami-version` (REQUIRED for vLLM CUDA 13+) | Not defaulted (user-specific input needed): VPC config, KMS key, multi-variant, async inference. ### Autoscaling target — tune by model type The default `--target-invocations-per-instance 20` is conservative and tuned for LLM workloads where each request takes 1–5 seconds. For embedding deployments (TEI), each request is much faster (typically <100ms on CPU, <20ms on GPU), so a single instance can handle far more throughput. **For embedding deployments, raise the target to 100–500** depending on instance and model size. The default of 20 will trigger autoscaling far too aggressively for embeddings and waste money. A rule of thumb: target value ≈ 60 / (typical request latency in seconds). LLM at 3s latency → target 20. Embedding at 100ms → target 600. Generative rerankers sit in between — they generate a single token per request, so ~40–100 is a reasonable target. ## Data capture + IAM gotcha If the user enables data capture, the execution role needs S3 write access to the capture prefix. The default URI (`s3://sagemaker--//data-capture/`) is typically a different bucket than the model artifact bucket. If `hf-cloud-sagemaker-iam-preflight` scoped the inline policy narrowly to just the model bucket, capture writes fail silently — endpoint keeps serving but no data appears. If the user reports "data capture isn't showing up", check the role's S3 access. Either widen the inline policy or pass `--data-capture-s3-uri` pointing to a bucket the role can write. ## Teardown ```bash python3 scripts/teardown.py # macOS / Linux python scripts\teardown.py # Windows ``` Deletes in safe order: alarms → autoscaling → endpoint (stops billing) → endpoint config → model. Idempotent. Does **not** delete: the IAM execution role (might be shared), data capture S3 objects (user might want to keep), SNS topic, original model artifacts. Always tell the user about the teardown command after the deployment summary. Users forget; endpoints accrue cost. ## When the deployment fails **`CannotStartContainerError` + no CloudWatch logs ever created** — the InferenceAmiVersion problem. If the image tag contains `cu130` or later and you didn't pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1`, this is the cause. See `hf-cloud-serving-image-selection`. Do NOT chase images, IAM roles, env vars, or instance types — the failure signature is identical for many other things but the cause here is the AMI. **"Failed to pass ping health check"** — the container *did* start and produced logs, but `/ping` isn't responding. Check CloudWatch at `/aws/sagemaker/Endpoints/`. Usually: wrong image for model architecture, missing HF token, or OOM. **"Container failed to start" (with logs present)** — entrypoint ran, then exited. Check CloudWatch. Common: missing required env vars (`SM_VLLM_MODEL`, `SM_VLLM_HOST`, `SM_VLLM_TRUST_REMOTE_CODE`), wrong `ModelDataUrl` format, unreadable model artifacts. **`ResourceLimitExceeded`** — no quota for the instance type in this region. Request increase or pick a different type (the planner should have checked quotas up front — see `hf-cloud-sagemaker-deployment-planner`). **`ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` in CloudWatch logs** — known packaging defect in `huggingface-pytorch-inference` GPU images (see "Known-broken images" in `hf-cloud-serving-image-selection`). Inside the container, so no env var, AMI, instance type, or sibling tag fixes it. Switch to DJL Inference. **InService, but invocations time out / async outputs never appear** — dead Python worker behind a live MMS front-end. Run the log scan from "InService is not success" above; the traceback in CloudWatch is the real error. **`403 Forbidden` downloading weights from HF Hub during startup** — the container's bundled `huggingface_hub` predates HF's XET CDN auth. Add `--env HF_HUB_ENABLE_HF_TRANSFER=0`, or pre-stage the weights in S3. Note: this can *mask* a deeper failure (the worker may still crash after the download succeeds) — re-check logs after fixing it. **Diagnostic rule**: when failures look identical across multiple configurations (different images, roles, instance types) and **no logs are ever produced**, the cause is almost always below the container — host AMI, networking, account-level — not the deployment config. Stop iterating on config; check the AMI version and account state. **Component stuck in `Creating`, no `FailureReason`** — the container is crash-looping and supervisord restarts it, so the status never changes. The component holds `Creating` until `ContainerStartupHealthCheckTimeoutInSeconds` expires, up to an hour. Read `/aws/sagemaker/InferenceComponents/` and look for `exited: app`, `not expected`, or `api_server.py: error:`. `deploy_ic.py` does this scan on every poll and aborts in about a minute. **`There is not enough hardware resources on the instances for this endpoint`** — the component's `ComputeResourceRequirements` exceed the schedulable pool, which is much smaller than the instance memory. Lower `--min-memory-mb` (1024 works on `ml.g5.xlarge`; 4096 is rejected there). Do not add instances: the message appears with a healthy instance present. **`Cannot delete inference component ... while it is in state CREATE_IN_PROGRESS` / `UPDATE_RC_IN_PROGRESS`** — normal, not an error. Retry; `teardown.py` retries for 15 min. `UPDATE_RC_IN_PROGRESS` means a scaling action is changing the copy count. **A component outlives its endpoint** — `delete-endpoint` leaves components behind, still reporting `InService`. They block reuse of the name. Delete components first, which is what `teardown.py` does. Don't retry blindly. The script prints the specific `FailureReason` from `describe-endpoint` — fix the root cause before retrying. --- ## 8. hf-cloud-serving-image-selection - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-cloud-serving-image-selection/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-serving-image-selection/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-cloud-serving-image-selection.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (2), referenced from this skill's directory: - `references/model-to-image.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-serving-image-selection/references/model-to-image.md - `scripts/mirror_image.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-cloud-serving-image-selection/scripts/mirror_image.py --- name: hf-cloud-serving-image-selection description: 'Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this fine-tuned model", "deploy this embedding model", "host a reranker", "serve a sentence-transformers model", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings/cross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.' --- # Serving Image Selection The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque `Failed to pass health check` error. ## Rule zero: HuggingFace images always win When both a HuggingFace-curated family (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-sglang`, `tei`, `huggingface-pytorch-inference`) and a generic family (`vllm`, `vllm-omni`, `sglang`, `djl-inference`) can serve the model, **the HuggingFace one is mandatory, not preferred**. The only valid reasons to use a generic image: 1. **Verified incompatibility** — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed). 2. **No HuggingFace tag exists in the target region** and mirroring is not an option. 3. **The HuggingFace image is in "Known-broken images"** below. A **newer version number on the generic repo is not a reason**. The AWS `vllm` repo often publishes a higher vLLM version than `huggingface-vllm`; an older-but-compatible `huggingface-vllm` tag still wins. "Latest vLLM" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied. ## Where image URIs come from **Primary source: AWS's official Deep Learning Containers catalog.** URL: https://aws.github.io/deep-learning-containers/reference/available_images/ This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2/ECS/EKS). When picking a URI for a deployment, **read it from this page directly** — copy the example URL, substitute `` with the user's region, and pass it to `deploy.py --image-uri`. The example URLs use `763104351884` as the account ID for most regions. A few regions use different accounts (e.g. `eu-south-1` uses `692866216735`). Check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) when in doubt. **Exception: none currently.** Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via `mirror_image.py` and pass the resulting URI directly. ## Quick decision | Model | Container family | How to get the URI | |---|---|---| | HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.) | **HuggingFace vLLM** | AWS catalog → "HuggingFace vLLM Inference" (ECR repo `huggingface-vllm`) | | Same as above, multimodal | **HuggingFace vLLM-Omni** | AWS catalog → "HuggingFace vLLM-Omni Inference" (ECR repo `huggingface-vllm-omni`) | | HuggingFace embeddings | TEI | AWS catalog → "HuggingFace Text Embeddings Inference" | | Encoder / cross-encoder rerankers (BERT-family `*ForSequenceClassification`) | TEI | Same as embeddings | | **Generative rerankers** (causal-LM, e.g. Qwen3-Reranker) | HuggingFace vLLM | Same as text-generation LLMs — **not TEI**, see "Rerankers: TEI or vLLM?" | | Text-to-image / diffusion (Stable Diffusion, FLUX) | DJL Inference | AWS catalog → "DJL Inference" — **not** HF Inference Toolkit, see "Known-broken images" | | HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU) | AWS catalog → "HuggingFace PyTorch Inference"; GPU tags currently broken — see "Known-broken images" | | User specifically wants SGLang | HuggingFace SGLang | AWS catalog → "HuggingFace SGLang Inference" | | No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see "Rule zero") | vLLM (AWS) | AWS catalog → "vLLM" section — fallback only, never for version freshness | | User specifically wants DJL-LMI | DJL Inference | AWS catalog → "DJL Inference" | | Amazon Nova | SageMaker JumpStart | Use JumpStart, not raw endpoint creation | | Custom inference code | BYOC | User provides URI | **HuggingFace-curated DLCs are mandatory when one is compatible (see "Rule zero").** `huggingface-vllm` is layered directly on the AWS vLLM DLC — **identical `SM_VLLM_*` env contract and the same cu130 AMI rule** — and adds current `transformers`, current `huggingface_hub` + `hf_xet` (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS `vllm` image is a compatibility escape hatch only; it usually shows a higher vLLM version than `huggingface-vllm`, and that is not a reason to pick it. **Do not use TGI.** Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960), June 2026, its `ModelBuilder` auto-routes `text-generation` to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.) Full reasoning for each family in `references/model-to-image.md`. ## Rerankers: TEI or vLLM? "Reranker" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model: - **Encoder cross-encoders** (BAAI/bge-reranker-*, mixedbread, most `sentence-transformers` rerankers) — BERT-family models with a classification head. `config.json` has `architectures: [..ForSequenceClassification]` on a TEI-supported encoder type. → **TEI**. - **Generative rerankers** (Qwen/Qwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes/no token. `config.json` has `architectures: [..ForCausalLM]`. → **HuggingFace vLLM**, deployed exactly like a text-generation LLM. TEI will load the architecture then reject the `classifier` model type (Qwen3 support in TEI is *embeddings-only*). Invocation pattern (raw completions API, `max_tokens=1`, logprobs scoring) is in `hf-cloud-sagemaker-production-defaults`. **Preflight before creating any resources** — one HTTP GET settles it: ```bash curl -s https://huggingface.co//raw/main/config.json # "architectures": ["Qwen3ForCausalLM"] → vLLM # "architectures": ["XLMRobertaForSequenceClassification"] → TEI ``` For TEI also confirm the *(architecture, task)* pair: an architecture appearing in TEI's supported list means embeddings support, not necessarily classification/reranking support. Heads-up: SageMaker SDK v3 (PR #5960) routes the `text-ranking` task to TEI **unconditionally** — correct for cross-encoders, wrong for generative rerankers. Don't treat the SDK's routing as evidence that TEI can serve a given reranker. ## Workflow For every family: **read the URI from the AWS catalog page**. 1. Open https://aws.github.io/deep-learning-containers/reference/available_images/ 2. Find the section for the right family (e.g. "HuggingFace vLLM Inference" for HuggingFace LLMs, "HuggingFace Text Embeddings Inference" for embeddings) 3. Pick the newest row marked `SageMaker` for the platform column — newest **within that family**. Do not switch to another family's section because it lists a higher engine version (see "Rule zero") 4. Substitute `` with the user's region (from `hf-cloud-aws-context-discovery`) 5. For vLLM: also check the AMI requirement (see "vLLM AMI requirement" below) 6. Pass the URI to `deploy.py --image-uri` (real-time) or `deploy_async.py --image-uri` (async) ### TEI: pick the right variant The TEI catalog row lists two URIs — GPU (`tei` repo) and CPU (`tei-cpu` repo). Pick based on the instance type: - `ml.g*`, `ml.p*`, `ml.inf*` → GPU variant - `ml.c*`, `ml.m*`, `ml.t*` → CPU variant Mixing them fails: CPU image on a GPU instance wastes hardware, GPU image on a CPU instance fails to start. **Note on the TEI account ID**: the catalog page shows `683313688378` as the example account, but TEI is published from a different account namespace than the main AWS DLCs and the per-region account IDs vary. If `683313688378.dkr.ecr..amazonaws.com/tei:...` returns an ECR pull error for a region other than us-east-1, check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) for the correct account ID for that region. ## vLLM AMI requirement vLLM DLC images with **CUDA 13 or higher** (current default: `cu130`) require setting `InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1` on the ProductionVariant. This applies equally to `huggingface-vllm` and `huggingface-vllm-omni` (layered on the same cu130 base) and to the AWS `vllm` repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths. Lookup table: | Tag contains | InferenceAmiVersion to pass | |---|---| | `cu130` (or higher) | `al2-ami-sagemaker-inference-gpu-3-1` | | `cu129` or lower | (omit the flag; default AMI works) | Rule of thumb: if the vLLM tag you picked contains `cu130` or later, pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1` to `deploy.py`. If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image. This is a vLLM-specific concern. TEI and HF Inference Toolkit images don't need an AMI override. ## Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM) Both images share the same contract: configuration as environment variables on the SageMaker model definition, `SM_VLLM_*` mapped to vLLM CLI flags. The `huggingface-vllm` entrypoint additionally auto-detects the model when `SM_VLLM_MODEL` is unset — from `/opt/ml/model` if artifacts are mounted, else from `HF_MODEL_ID` — but setting `SM_VLLM_MODEL` explicitly works on both and is what our examples use. ### Required for every HuggingFace LLM deployment | Env var | Purpose | Notes | |---|---|---| | `SM_VLLM_MODEL` | HF model ID (e.g. `Qwen/Qwen3-0.6B`) or `/opt/ml/model` if loading from S3 | — | | `SM_VLLM_HOST` | **Must be `0.0.0.0`** | Otherwise vLLM binds localhost only, ping fails, container dies before logs. Top cause of mystery failures with this image. | | `SM_VLLM_TRUST_REMOTE_CODE` | `true` for Qwen and several recent architectures | Set unconditionally — downside negligible, upside is the model loads. | | `HUGGING_FACE_HUB_TOKEN` | HF token | Required for gated models. | ### Tuning (optional) | Env var | Purpose | |---|---| | `SM_VLLM_MAX_MODEL_LEN` | Max sequence length — set this; defaults can be wrong for fine-tunes | | `SM_VLLM_GPU_MEMORY_UTILIZATION` | Float 0.0–1.0, ~0.9 reasonable | | `SM_VLLM_TENSOR_PARALLEL_SIZE` | GPU count for multi-GPU instances | | `SM_VLLM_DTYPE` | `auto`, `bfloat16`, `float16` | Any vLLM CLI flag works — uppercase, replace dashes with underscores, prepend `SM_VLLM_`. ## Configuring TEI Simpler env contract than vLLM: | Env var | Purpose | Required | |---|---|---| | `HF_MODEL_ID` | HF model ID (e.g. `BAAI/bge-large-en-v1.5`) or `/opt/ml/model` | Yes | | `HF_TOKEN` | HF auth token | Only for gated models | | `MAX_BATCH_TOKENS` | Max tokens per batch (default 16384) | No | | `MAX_CLIENT_BATCH_SIZE` | Max requests per client batch (default 32) | No | No host-binding to configure, no trust-remote-code flag. The architectures TEI supports (BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2/3, Gemma2/3, ModernBert) are baked into the image. ## CUDA / instance compatibility Critical and easy to get wrong: | CUDA in image tag | Default AMI | With `al2-ami-sagemaker-inference-gpu-3-1` | |---|---|---| | cu124 / cu128 | g5, g6, p5 all work | (not needed) | | cu129 | g6, p5; g5 fails (driver mismatch → CannotStartContainerError) | expected to fix g5 (unverified) | | cu130+ | fails everywhere — AMI flag is mandatory | g5, g6, p5 all work (cu130-on-g5 verified June 2026) | The driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes `ml.g5.*` viable for cu129+ images. ## VPC / NAT gateway problem SageMaker endpoints inside a VPC **without** a NAT gateway can't pull from `public.ecr.aws`. The deployment fails with an image-pull error that doesn't mention "VPC" or "egress". For images on AWS's regional ECR (everything in the catalog): SageMaker reaches them through built-in routing, no NAT needed. Use the regional URI pattern (`.dkr.ecr..amazonaws.com/...`), not the `public.ecr.aws/...` pattern. For images requiring `public.ecr.aws` access (less common): mirror to a private ECR repo in your account with `scripts/mirror_image.py` (cross-platform; needs Docker + the `aws` CLI). Run it from the shell where the AWS CLI works. ```bash # macOS / Linux PRIVATE_URI=$(python3 scripts/mirror_image.py \ public.ecr.aws/deep-learning-containers/vllm: \ vllm-mirror) ``` ```powershell # Windows (PowerShell) — capture stdout into a variable $PRIVATE_URI = python scripts\mirror_image.py ` public.ecr.aws/deep-learning-containers/vllm: vllm-mirror ``` ## When the catalog page won't render, is stale, or is wrong **The page won't render / fetch returns junk**: the catalog page is JavaScript-heavy and some fetch tools get an empty shell. Fallbacks, in order: 1. **The catalog's source data on GitHub** — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest: ```bash curl -s https://api.github.com/repos/aws/deep-learning-containers/contents/docs/src/data/huggingface-vllm curl -s https://raw.githubusercontent.com/aws/deep-learning-containers/main/docs/src/data/huggingface-vllm/0.21.0-gpu-sagemaker.yml ``` Directory names match ECR repos (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-tei`, `vllm`, `djl-inference`, ...). 2. **Query ECR directly** for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files): ```bash aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \ --region --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json ``` 3. [Release notes on the DLC GitHub repo](https://github.com/aws/deep-learning-containers/releases). **A tag was just released and isn't on the page yet**: rare; AWS updates the page on each release. Check the release notes above. **An architecture you need isn't supported by the listed image yet**: for TEI specifically, you can mirror the upstream image from GHCR (`ghcr.io/huggingface/text-embeddings-inference:`) into private ECR and pass the resulting URI directly to `deploy.py --image-uri`. Same `mirror_image.py` script. ## Known-broken images (last checked July 2026) | Image | Defect | Use instead | |---|---|---| | `huggingface-pytorch-inference` **GPU** tags — all recent ones tested (PT 2.3–2.6, cu121/cu124, transformers 4.48–5.5.3) | `ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` at `import torch`. The NCCL bundled in the image is older than what torch links against — a packaging defect *inside the container*, on g5 **and** g6, regardless of AMI, model, or inference code. The MMS Java front-end keeps answering `/ping`, so the endpoint can reach InService while the Python worker crash-loops and serves nothing. | DJL Inference (bundles its own complete CUDA/NCCL stack) or BYOC. CPU tags are unaffected. | Re-check when AWS publishes new `huggingface-pytorch-inference` GPU tags — remove the row once a fixed image is confirmed. **General fallback rule**: when an HF DLC fails with CUDA/NCCL linker errors, switch to DJL Inference rather than iterating over sibling tags — the defect class is per-repo, not per-tag (three different tags were tried for the case above; all broken). **Related HF Hub gotcha**: older DLCs can fail model download with `403 Forbidden` from HF's XET CDN (their bundled `huggingface_hub` predates XET auth). Set `HF_HUB_ENABLE_HF_TRANSFER=0` to force the standard download path, or pre-stage weights in S3. ## Hub download time at first boot Loading the model from HF Hub happens *inside the container after the endpoint starts* — expect **5–15+ minutes** before InService even for small models, longer for multi-GB ones. A slow first boot is not a failure; don't tear down or re-diagnose before the deploy script's 30-minute wait expires. For production or repeated deployments, pre-stage the weights in S3 and pass `--model-s3-uri` to `deploy.py` (the model then loads from `/opt/ml/model`) — faster, immune to Hub rate limits/outages, and no `HUGGING_FACE_HUB_TOKEN` needed at runtime. --- ## 9. hf-mem - **Source:** https://github.com/huggingface/skills/blob/main/skills/hf-mem/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/hf-mem/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/hf-mem.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: hf-mem description: Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub --- `hf_mem` estimates the required memory for inference, including model weights and an optional KV cache, for Safetensors and GGUF for models on the Hugging Face Hub using HTTP Range requests i.e., without downloading or loading any weights locally. ## When to use? - User asks how much VRAM or memory a model needs to run - User wants to know if a model fits on their GPU or a given instance - User references a Hugging Face model ID or URL and asks about inference requirements ## What are the requirements? - `uv` installed (for `uvx`) - `HF_TOKEN` env var or `--hf-token` flag (for gated or private models only) ## How to run? Run with `--model-id` pointing to the Hugging Face Hub repository which will check that it either contains Safetensors (via `model.safetensors`, `model.safetensors.index.json` if sharded, or `model_index.json` for Diffusers) or GGUF model weights within. ```bash uvx hf-mem --model-id --json-output ``` If the repository contains GGUF model weights in multiple precisions / quantizations, the estimations will be on a per-file basis, whereas for inference you won't load all of those but rather only a single precision. This being said, for GGUF you might as well need to provide `--gguf-file` to target the specific file (or path if sharded) you want to run. ```bash uvx hf-mem --model-id --gguf-file --json-output ``` Additionally, `hf-mem` comes with an `--experimental` flag that will also calculate the KV cache memory requirements too, useful for large-language models, meaning it applies to LLMs (`...ForCausalLM`), VLMs (`...ForConditionalGeneration`), and GGUF models. As per the context window, it will be read from the default or overridden with `--max-model-len` a la vLLM. And, same goes for the KV cache precision, which will default to the model precision unless manually set via `--kv-cache-dtype` a la vLLM too. For Safetensors use as: ```bash uvx hf-mem --model-id --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|bfloat16|fp8|fp8_ds_mla|fp8_e4m3|fp8_e5m2|fp8_inc] --json-output ``` And, for GGUF use as: ```bash uvx hf-mem --model-id --gguf-file --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|F32|F16|Q4_0|Q4_1|Q5_0|Q5_1|Q8_0|Q8_1|Q2_K|Q3_K|Q4_K|Q5_K|Q6_K|Q8_K|IQ2_XXS|IQ2_XS|IQ3_XXS|IQ1_S|IQ4_NL|IQ3_S|IQ2_S|IQ4_XS|I8|I16|I32|I64|F64|IQ1_M|BF16|TQ1_0|TQ2_0|MXFP4] --json-output ``` ## Examples For Transformers with Safetensors weights: ```bash uvx hf-mem --model-id MiniMaxAI/MiniMax-M2 --json-output ``` For Diffusers with Safetensors weights: ```bash uvx hf-mem --model-id Qwen/Qwen-Image --json-output ``` For Sentence Transformers with Safetensors weights: ```bash uvx hf-mem --model-id google/embeddinggemma-300m --json-output ``` With `--experimental` to include the KV cache estimation for LLMs and VLMs: ```bash uvx hf-mem --model-id mistralai/Mistral-7B-v0.1 --experimental --json-output ``` And, for LLMs or VLMs with GGUF weights: ```bash uvx hf-mem --model-id unsloth/Qwen3.5-397B-A17B-GGUF --gguf-file Q4_K_M --experimental --json-output ``` --- ## 10. huggingface-best - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-best/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-best/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-best.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: huggingface-best description: >- Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: "best model for X", "what model should I use for", "top models for [task]", "which model runs on my laptop/machine/device", "recommend a model for", "what LLM should I use for", "compare models for", "what's state of the art for", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks. --- # HuggingFace Best Model Finder Finds the best models for a task by querying official HF benchmark leaderboards, enriching results with model size data, filtering for what fits on the user's device, and returning a comparison table with benchmark scores. --- ## Step 1: Parse the request Extract from the user's message: - **Task**: what they want the model to do (coding, math/reasoning, chat, OCR, RAG/retrieval, speech recognition, image classification, multimodal, agents, etc.) - **Device**: hardware constraints (MacBook M-series 8/16/32/64GB unified memory, RTX GPU with VRAM amount, CPU-only, cloud/no constraint, etc.) If device is not mentioned, skip filtering entirely and return the highest-performing models regardless of size. If the task is genuinely ambiguous, ask one clarifying question. ### Device → max parameter budget When a device is specified, extract its available memory (unified RAM for Apple Silicon, VRAM for discrete GPUs) and apply: - **fp16 max params (B)** ≈ memory (GB) ÷ 2 - **Q4 max params (B)** ≈ memory (GB) × 2 Examples: 16GB → 8B fp16 / 32B Q4 — 24GB VRAM → 12B fp16 / 48B Q4 — 8GB → 4B fp16 / 16B Q4 --- ## Step 2: Find relevant benchmark datasets Fetch the full list of official HF benchmarks: ```bash curl -s -H "Authorization: Bearer $(cat ~/.cache/huggingface/token)" \ "https://huggingface.co/api/datasets?filter=benchmark:official&limit=500" | jq '[.[] | {id, tags, description}]' ``` Read the returned list and select the datasets most relevant to the user's task — match on dataset id, tags, and description. Use your judgment; don't limit yourself to 2-3. Aim for comprehensive coverage: if 5 benchmarks clearly cover the task, use all 5. --- ## Step 3: Fetch top models from leaderboards For each selected benchmark dataset: ```bash curl -s -H "Authorization: Bearer $(cat ~/.cache/huggingface/token)" \ "https://huggingface.co/api/datasets///leaderboard" | jq '[.[:15] | .[] | {rank, modelId, value, verified}]' ``` Collect model IDs and scores across all benchmarks. If a leaderboard returns an error (404, 401, etc.), skip it and note it in the output. --- ## Step 4: Enrich with model metadata For the top 10-15 candidate model IDs, get model infos. ```bash # REST API curl -s -H "Authorization: Bearer $(cat ~/.cache/huggingface/token)" \ "https://huggingface.co/api/models/org/model1" | jq '{safetensors, tags, cardData}' # CLI (hf-cli) hf models info org/model1 --json | jq '{safetensors, tags, cardData}' ``` Extract from each response: - **Parameters**: `safetensors.total` → convert to B (e.g., 7_241_748_480 → "7.2B") - **License**: from model card tags (look for `license:apache-2.0`, `license:mit`, etc.) - If `safetensors` is absent, parse size from the model name (look for "7b", "8b", "13b", "70b", "72b", etc.) --- ## Step 5: Filter and rank **If a device was specified:** 1. Remove models exceeding the fp16 parameter budget for the device 2. Flag models that fit only with Q4 quantization (multiply budget by ~4 for Q4 capacity) 3. If a highly-ranked model is slightly over budget, keep it with a "needs Q4" note — don't silently drop it **If no device was mentioned:** skip all size filtering — just rank by benchmark score. Then: rank by benchmark score (descending), keep top 5-8 models. Include proprietary models (GPT-4, Claude, Gemini) if they appear on leaderboards, but flag them as "API only / not self-hostable". If the user explicitly asked for local/open models only, exclude them. --- ## Step 6: Output ### Comparison table ```markdown | # | Model | Params | [Benchmark 1] | [Benchmark 2] | License | On device | |---|-------|--------|--------------|--------------|---------|-----------| | ⭐1 | [org/name](https://huggingface.co/org/name) | 7B | 85.2% | — | Apache 2.0 | Yes (fp16) | | 2 | [org/name](https://huggingface.co/org/name) | 13B | 83.1% | 71.5% | MIT | Q4 only | | 3 | [org/name](https://huggingface.co/org/name) | 70B | 90.0% | 81.0% | Llama | Too large | ``` - Link model names to `https://huggingface.co/` - Use `—` for benchmarks where the model wasn't evaluated - Star the top recommended pick with ⭐ - "On device" values: `Yes (fp16)`, `Q4 only`, `Too large`, `API only` ### Follow-up After presenting the table, ask the user: "Would you like to run **[top recommended model]**?" If they say yes, ask whether they'd prefer to: - **Run locally** — ask about their device if not already known, then give appropriate setup instructions - **Run on HF Jobs** — point them to the HF Jobs guide: https://huggingface.co/docs/huggingface_hub/en/guides/jobs --- ## Error handling - **Leaderboard not found**: skip, note "leaderboard unavailable" in output - **Model missing from hub_repo_details**: fall back to parsing size from model name - **No benchmarks found for task**: use the curated fallback table above, or try `hub_repo_search` with `filters=[""]` sorted by `trendingScore` - **All leaderboards fail**: fall back to `hub_repo_search` for popular models tagged with the task, note that results are by popularity rather than benchmark score --- ## 11. huggingface-community-evals - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-community-evals/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-community-evals.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (5), referenced from this skill's directory: - `examples/.env.example` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/examples/.env.example - `examples/USAGE_EXAMPLES.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/examples/USAGE_EXAMPLES.md - `scripts/inspect_eval_uv.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/scripts/inspect_eval_uv.py - `scripts/inspect_vllm_uv.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/scripts/inspect_vllm_uv.py - `scripts/lighteval_vllm_uv.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-community-evals/scripts/lighteval_vllm_uv.py --- name: huggingface-community-evals description: Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation. --- # Overview This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**. It covers: - `inspect-ai` with local inference - `lighteval` with local inference - choosing between `vllm`, Hugging Face Transformers, and `accelerate` - smoke tests, task selection, and backend fallback strategy It does **not** cover: - Hugging Face Jobs orchestration - model-card or `model-index` edits - README table extraction - Artificial Analysis imports - `.eval_results` generation or publishing - PR creation or community-evals automation If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill. If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`. > All paths below are relative to the directory containing this `SKILL.md`. # When To Use Which Script | Use case | Script | |---|---| | Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` | | Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts/inspect_vllm_uv.py` | | Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts/lighteval_vllm_uv.py` | | Extra command patterns | `examples/USAGE_EXAMPLES.md` | # Prerequisites - Prefer `uv run` for local execution. - Set `HF_TOKEN` for gated/private models. - For local GPU runs, verify GPU access before starting: ```bash uv --version printenv HF_TOKEN >/dev/null nvidia-smi ``` If `nvidia-smi` is unavailable, either: - use `scripts/inspect_eval_uv.py` for lighter provider-backed evaluation, or - hand off to the `hugging-face-jobs` skill if the user wants remote compute. # Core Workflow 1. Choose the evaluation framework. - Use `inspect-ai` when you want explicit task control and inspect-native flows. - Use `lighteval` when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks. 2. Choose the inference backend. - Prefer `vllm` for throughput on supported architectures. - Use Hugging Face Transformers (`--backend hf`) or `accelerate` as compatibility fallbacks. 3. Start with a smoke test. - `inspect-ai`: add `--limit 10` or similar. - `lighteval`: add `--max-samples 10`. 4. Scale up only after the smoke test passes. 5. If the user wants remote execution, hand off to `hugging-face-jobs` with the same script + args. # Quick Start ## Option A: inspect-ai with local inference providers path Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead. ```bash uv run scripts/inspect_eval_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu \ --limit 20 ``` Use this path when: - you want a quick local smoke test - you do not need direct GPU control - the task already exists in `inspect-evals` ## Option B: inspect-ai on Local GPU Best when you need to load the Hub model directly, use `vllm`, or fall back to Transformers for unsupported architectures. Local GPU: ```bash uv run scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --task gsm8k \ --limit 20 ``` Transformers fallback: ```bash uv run scripts/inspect_vllm_uv.py \ --model microsoft/phi-2 \ --task mmlu \ --backend hf \ --trust-remote-code \ --limit 20 ``` ## Option C: lighteval on Local GPU Best when the task is naturally expressed as a `lighteval` task string, especially Open LLM Leaderboard style benchmarks. Local GPU: ```bash uv run scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-3B-Instruct \ --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \ --max-samples 20 \ --use-chat-template ``` `accelerate` fallback: ```bash uv run scripts/lighteval_vllm_uv.py \ --model microsoft/phi-2 \ --tasks "leaderboard|mmlu|5" \ --backend accelerate \ --trust-remote-code \ --max-samples 20 ``` # Remote Execution Boundary This skill intentionally stops at **local execution and backend selection**. If the user wants to: - run these scripts on Hugging Face Jobs - pick remote hardware - pass secrets to remote jobs - schedule recurring runs - inspect / cancel / monitor jobs then switch to the **`hugging-face-jobs`** skill and pass it one of these scripts plus the chosen arguments. # Task Selection `inspect-ai` examples: - `mmlu` - `gsm8k` - `hellaswag` - `arc_challenge` - `truthfulqa` - `winogrande` - `humaneval` `lighteval` task strings use `suite|task|num_fewshot`: - `leaderboard|mmlu|5` - `leaderboard|gsm8k|5` - `leaderboard|arc_challenge|25` - `lighteval|hellaswag|0` Multiple `lighteval` tasks can be comma-separated in `--tasks`. # Backend Selection - Prefer `inspect_vllm_uv.py --backend vllm` for fast GPU inference on supported architectures. - Use `inspect_vllm_uv.py --backend hf` when `vllm` does not support the model. - Prefer `lighteval_vllm_uv.py --backend vllm` for throughput on supported models. - Use `lighteval_vllm_uv.py --backend accelerate` as the compatibility fallback. - Use `inspect_eval_uv.py` when Inference Providers already cover the model and you do not need direct GPU control. # Hardware Guidance | Model size | Suggested local hardware | |---|---| | `< 3B` | consumer GPU / Apple Silicon / small dev GPU | | `3B - 13B` | stronger local GPU | | `13B+` | high-memory local GPU or hand off to `hugging-face-jobs` | For smoke tests, prefer cheaper local runs plus `--limit` or `--max-samples`. # Troubleshooting - CUDA or vLLM OOM: - reduce `--batch-size` - reduce `--gpu-memory-utilization` - switch to a smaller model for the smoke test - if necessary, hand off to `hugging-face-jobs` - Model unsupported by `vllm`: - switch to `--backend hf` for `inspect-ai` - switch to `--backend accelerate` for `lighteval` - Gated/private repo access fails: - verify `HF_TOKEN` - Custom model code required: - add `--trust-remote-code` # Examples See: - `examples/USAGE_EXAMPLES.md` for local command patterns - `scripts/inspect_eval_uv.py` - `scripts/inspect_vllm_uv.py` - `scripts/lighteval_vllm_uv.py` --- ## 12. huggingface-datasets - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-datasets/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-datasets/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-datasets.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: huggingface-datasets description: Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics. --- # Hugging Face Dataset Viewer Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction. ## Core workflow 1. Optionally validate dataset availability with `/is-valid`. 2. Resolve `config` + `split` with `/splits`. 3. Preview with `/first-rows`. 4. Paginate content with `/rows` using `offset` and `length` (max 100). 5. Use `/search` for text matching and `/filter` for row predicates. 6. Retrieve parquet links via `/parquet` and totals/metadata via `/size` and `/statistics`. ## Defaults - Base URL: `https://datasets-server.huggingface.co` - Default API method: `GET` - Query params should be URL-encoded. - `offset` is 0-based. - `length` max is usually `100` for row-like endpoints. - Gated/private datasets require `Authorization: Bearer `. ## Dataset Viewer - `Validate dataset`: `/is-valid?dataset=` - `List subsets and splits`: `/splits?dataset=` - `Preview first rows`: `/first-rows?dataset=&config=&split=` - `Paginate rows`: `/rows?dataset=&config=&split=&offset=&length=` - `Search text`: `/search?dataset=&config=&split=&query=&offset=&length=` - `Filter with predicates`: `/filter?dataset=&config=&split=&where=&orderby=&offset=&length=` - `List parquet shards`: `/parquet?dataset=` - `Get size totals`: `/size?dataset=` - `Get column statistics`: `/statistics?dataset=&config=&split=` - `Get Croissant metadata (if available)`: `/croissant?dataset=` Pagination pattern: ```bash curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100" curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100" ``` When pagination is partial, use response fields such as `num_rows_total`, `num_rows_per_page`, and `partial` to drive continuation logic. Search/filter notes: - `/search` matches string columns (full-text style behavior is internal to the API). - `/filter` requires predicate syntax in `where` and optional sort in `orderby`. - Keep filtering and searches read-only and side-effect free. For CLI-based parquet URL discovery or SQL, use the `hf-cli` skill with `hf datasets parquet` and `hf datasets sql`. ## Creating and Uploading Datasets Use one of these flows depending on dependency constraints. Zero local dependencies (Hub UI): - Create dataset repo in browser: `https://huggingface.co/new-dataset` - Upload parquet files in the repo "Files and versions" page. - Verify shards appear in Dataset Viewer: ```bash curl -s "https://datasets-server.huggingface.co/parquet?dataset=/" ``` Low dependency CLI flow (`npx @huggingface/hub` / `hfjs`): - Set auth token: ```bash export HF_TOKEN= ``` - Upload parquet folder to a dataset repo (auto-creates repo if missing): ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data ``` - Upload as private repo on creation: ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data --private ``` After upload, call `/parquet` to discover `//` values for querying with `@~parquet`. ## Agent Traces The Hub supports raw agent session traces from Claude Code, Codex, and Pi Agent. Upload them to Hugging Face Datasets as original JSONL files and the Hub can auto-detect the trace format, tag the dataset as `Traces`, and enable the trace viewer for browsing sessions, turns, tool calls, and model responses. Common local session directories: - Claude Code: `~/.claude/projects` - Codex: `~/.codex/sessions` - Pi: `~/.pi/agent/sessions` Default to private dataset repos because traces can contain prompts, file paths, tool outputs, secrets, or PII. Preserve the raw `.jsonl` files and nest them by project/cwd instead of uploading every session at the dataset root. ```bash hf repos create / --type dataset --private --exist-ok hf upload / ~/.codex/sessions codex/ --type dataset ``` --- ## 13. huggingface-gradio - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-gradio/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-gradio/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-gradio.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (1), referenced from this skill's directory: - `examples.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-gradio/examples.md --- name: huggingface-gradio description: Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots. --- # Gradio Gradio is a Python library for building interactive web UIs and ML demos. This skill covers the core API, patterns, and examples. ## Guides Detailed guides on specific topics (read these when relevant): - [Quickstart](https://www.gradio.app/guides/quickstart) - [The Interface Class](https://www.gradio.app/guides/the-interface-class) - [Blocks and Event Listeners](https://www.gradio.app/guides/blocks-and-event-listeners) - [Controlling Layout](https://www.gradio.app/guides/controlling-layout) - [More Blocks Features](https://www.gradio.app/guides/more-blocks-features) - [Custom CSS and JS](https://www.gradio.app/guides/custom-CSS-and-JS) - [Streaming Outputs](https://www.gradio.app/guides/streaming-outputs) - [Streaming Inputs](https://www.gradio.app/guides/streaming-inputs) - [Sharing Your App](https://www.gradio.app/guides/sharing-your-app) - [Custom HTML Components](https://www.gradio.app/guides/custom-HTML-components) - [Getting Started with the Python Client](https://www.gradio.app/guides/getting-started-with-the-python-client) - [Getting Started with the JS Client](https://www.gradio.app/guides/getting-started-with-the-js-client) ## Core Patterns **Interface** (high-level): wraps a function with input/output components. ```python import gradio as gr def greet(name): return f"Hello {name}!" gr.Interface(fn=greet, inputs="text", outputs="text").launch() ``` **Blocks** (low-level): flexible layout with explicit event wiring. ```python import gradio as gr with gr.Blocks() as demo: name = gr.Textbox(label="Name") output = gr.Textbox(label="Greeting") btn = gr.Button("Greet") btn.click(fn=lambda n: f"Hello {n}!", inputs=name, outputs=output) demo.launch() ``` **ChatInterface**: high-level wrapper for chatbot UIs. ```python import gradio as gr def respond(message, history): return f"You said: {message}" gr.ChatInterface(fn=respond).launch() ``` ## Key Component Signatures ### `Textbox(value: str | I18nData | Callable | None = None, type: Literal['text', 'password', 'email'] = "text", lines: int = 1, max_lines: int | None = None, placeholder: str | I18nData | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, autofocus: bool = False, autoscroll: bool = True, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", text_align: Literal['left', 'right'] | None = None, rtl: bool = False, buttons: list[Literal['copy'] | Button] | None = None, max_length: int | None = None, submit_btn: str | bool | None = False, stop_btn: str | bool | None = False, html_attributes: InputHTMLAttributes | None = None)` Creates a textarea for user to enter string input or display string output.. ### `Number(value: float | Callable | None = None, label: str | I18nData | None = None, placeholder: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None, precision: int | None = None, minimum: float | None = None, maximum: float | None = None, step: float = 1)` Creates a numeric field for user to enter numbers as input or display numeric output.. ### `Slider(minimum: float = 0, maximum: float = 100, value: float | Callable | None = None, step: float | None = None, precision: int | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", randomize: bool = False, buttons: list[Literal['reset']] | None = None)` Creates a slider that ranges from {minimum} to {maximum} with a step size of {step}.. ### `Checkbox(value: bool | Callable = False, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)` Creates a checkbox that can be set to `True` or `False`. ### `Dropdown(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Sequence[str | int | float] | Callable | DefaultValue | None = DefaultValue(), type: Literal['value', 'index'] = "value", multiselect: bool | None = None, allow_custom_value: bool = False, max_choices: int | None = None, filterable: bool = True, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)` Creates a dropdown of choices from which a single entry or multiple entries can be selected (as an input component) or displayed (as an output component).. ### `Radio(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Callable | None = None, type: Literal['value', 'index'] = "value", label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", rtl: bool = False, buttons: list[Button] | None = None)` Creates a set of (string or numeric type) radio buttons of which only one can be selected.. ### `Image(value: str | PIL.Image.Image | np.ndarray | Callable | None = None, format: str = "webp", height: int | str | None = None, width: int | str | None = None, image_mode: Literal['1', 'L', 'P', 'RGB', 'RGBA', 'CMYK', 'YCbCr', 'LAB', 'HSV', 'I', 'F'] | None = "RGB", sources: list[Literal['upload', 'webcam', 'clipboard']] | Literal['upload', 'webcam', 'clipboard'] | None = None, type: Literal['numpy', 'pil', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, buttons: list[Literal['download', 'share', 'fullscreen'] | Button] | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, placeholder: str | None = None, watermark: WatermarkOptions | None = None)` Creates an image component that can be used to upload images (as an input) or display images (as an output).. ### `Audio(value: str | Path | tuple[int, np.ndarray] | Callable | None = None, sources: list[Literal['upload', 'microphone']] | Literal['upload', 'microphone'] | None = None, type: Literal['numpy', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", format: Literal['wav', 'mp3'] | None = None, autoplay: bool = False, editable: bool = True, buttons: list[Literal['download', 'share'] | Button] | None = None, waveform_options: WaveformOptions | dict | None = None, loop: bool = False, recording: bool = False, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)` Creates an audio component that can be used to upload/record audio (as an input) or display audio (as an output).. ### `Video(value: str | Path | Callable | None = None, format: str | None = None, sources: list[Literal['upload', 'webcam']] | Literal['upload', 'webcam'] | None = None, height: int | str | None = None, width: int | str | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, include_audio: bool | None = None, autoplay: bool = False, buttons: list[Literal['download', 'share'] | Button] | None = None, loop: bool = False, streaming: bool = False, watermark: WatermarkOptions | None = None, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)` Creates a video component that can be used to upload/record videos (as an input) or display videos (as an output). ### `File(value: str | list[str] | Callable | None = None, file_count: Literal['single', 'multiple', 'directory'] = "single", file_types: list[str] | None = None, type: Literal['filepath', 'binary'] = "filepath", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, height: int | str | float | None = None, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", allow_reordering: bool = False, buttons: list[Button] | None = None)` Creates a file component that allows uploading one or more generic files (when used as an input) or displaying generic files or URLs for download (as output). ### `Chatbot(value: list[MessageDict | Message] | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, autoscroll: bool = True, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", height: int | str | None = 400, resizable: bool = False, max_height: int | str | None = None, min_height: int | str | None = None, editable: Literal['user', 'all'] | None = None, latex_delimiters: list[dict[str, str | bool]] | None = None, rtl: bool = False, buttons: list[Literal['share', 'copy', 'copy_all'] | Button] | None = None, watermark: str | None = None, avatar_images: tuple[str | Path | None, str | Path | None] | None = None, sanitize_html: bool = True, render_markdown: bool = True, feedback_options: list[str] | tuple[str, ...] | None = ('Like', 'Dislike'), feedback_value: Sequence[str | None] | None = None, line_breaks: bool = True, layout: Literal['panel', 'bubble'] | None = None, placeholder: str | None = None, examples: list[ExampleMessage] | None = None, allow_file_downloads: = True, group_consecutive_messages: bool = True, allow_tags: list[str] | bool = True, reasoning_tags: list[tuple[str, str]] | None = None, like_user_message: bool = False)` Creates a chatbot that displays user-submitted messages and responses. ### `Button(value: str | I18nData | Callable = "Run", every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, variant: Literal['primary', 'secondary', 'stop', 'huggingface'] = "secondary", size: Literal['sm', 'md', 'lg'] = "lg", icon: str | Path | None = None, link: str | None = None, link_target: Literal['_self', '_blank', '_parent', '_top'] = "_self", visible: bool | Literal['hidden'] = True, interactive: bool = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", scale: int | None = None, min_width: int | None = None)` Creates a button that can be assigned arbitrary .click() events. ### `Markdown(value: str | I18nData | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, rtl: bool = False, latex_delimiters: list[dict[str, str | bool]] | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", sanitize_html: bool = True, line_breaks: bool = False, header_links: bool = False, height: int | str | None = None, max_height: int | str | None = None, min_height: int | str | None = None, buttons: list[Literal['copy']] | None = None, container: bool = False, padding: bool = False)` Used to render arbitrary Markdown output. ### `HTML(value: Any | Callable | None = None, label: str | I18nData | None = None, html_template: str = "${value}", css_template: str = "", js_on_load: str | None = "element.addEventListener('click', function() { trigger('click') });", apply_default_css: bool = True, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool = False, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", min_height: int | None = None, max_height: int | None = None, container: bool = False, padding: bool = False, autoscroll: bool = False, buttons: list[Button] | None = None, server_functions: list[Callable] | None = None, props: Any)` Creates a component with arbitrary HTML. ## Custom HTML Components If a task requires significant customization of an existing component or a component that doesn't exist in Gradio, you can create one with `gr.HTML`. It supports `html_template` (with `${}` JS expressions and `{{}}` Handlebars syntax), `css_template` for scoped styles, and `js_on_load` for interactivity — where `props.value` updates the component value and `trigger('event_name')` fires Gradio events. For reuse, subclass `gr.HTML` and define `api_info()` for API/MCP support. See the [full guide](https://www.gradio.app/guides/custom-HTML-components). Here's an example that shows how to create and use these kinds of components: ```python import gradio as gr class StarRating(gr.HTML): def __init__(self, label, value=0, **kwargs): html_template = """

${label} rating:

${Array.from({length: 5}, (_, i) => ``).join('')} """ css_template = """ img { height: 50px; display: inline-block; cursor: pointer; } .faded { filter: grayscale(100%); opacity: 0.3; } """ js_on_load = """ const imgs = element.querySelectorAll('img'); imgs.forEach((img, index) => { img.addEventListener('click', () => { props.value = index + 1; }); }); """ super().__init__(value=value, label=label, html_template=html_template, css_template=css_template, js_on_load=js_on_load, **kwargs) def api_info(self): return {"type": "integer", "minimum": 0, "maximum": 5} with gr.Blocks() as demo: gr.Markdown("# Restaurant Review") food_rating = StarRating(label="Food", value=3) service_rating = StarRating(label="Service", value=3) ambience_rating = StarRating(label="Ambience", value=3) average_btn = gr.Button("Calculate Average Rating") rating_output = StarRating(label="Average", value=3) def calculate_average(food, service, ambience): return round((food + service + ambience) / 3) average_btn.click( fn=calculate_average, inputs=[food_rating, service_rating, ambience_rating], outputs=rating_output ) demo.launch() ``` ## Event Listeners All event listeners share the same signature: ```python component.event_name( fn: Callable | None | Literal["decorator"] = "decorator", inputs: Component | Sequence[Component] | set[Component] | None = None, outputs: Component | Sequence[Component] | set[Component] | None = None, api_name: str | None = None, api_description: str | None | Literal[False] = None, scroll_to_output: bool = False, show_progress: Literal["full", "minimal", "hidden"] = "full", show_progress_on: Component | Sequence[Component] | None = None, queue: bool = True, batch: bool = False, max_batch_size: int = 4, preprocess: bool = True, postprocess: bool = True, cancels: dict[str, Any] | list[dict[str, Any]] | None = None, trigger_mode: Literal["once", "multiple", "always_last"] | None = None, js: str | Literal[True] | None = None, concurrency_limit: int | None | Literal["default"] = "default", concurrency_id: str | None = None, api_visibility: Literal["public", "private", "undocumented"] = "public", time_limit: int | None = None, stream_every: float = 0.5, key: int | str | tuple[int | str, ...] | None = None, validator: Callable | None = None, ) -> Dependency ``` Supported events per component: - **AnnotatedImage**: select - **Audio**: stream, change, clear, play, pause, stop, pause, start_recording, pause_recording, stop_recording, upload, input - **BarPlot**: select, double_click - **BrowserState**: change - **Button**: click - **Chatbot**: change, select, like, retry, undo, example_select, option_select, clear, copy, edit - **Checkbox**: change, input, select - **CheckboxGroup**: change, input, select - **ClearButton**: click - **Code**: change, input, focus, blur - **ColorPicker**: change, input, submit, focus, blur - **Dataframe**: change, input, select, edit - **Dataset**: click, select - **DateTime**: change, submit - **DeepLinkButton**: click - **Dialogue**: change, input, submit - **DownloadButton**: click - **Dropdown**: change, input, select, focus, blur, key_up - **DuplicateButton**: click - **File**: change, select, clear, upload, delete, download - **FileExplorer**: change, input, select - **Gallery**: select, upload, change, delete, preview_close, preview_open - **HTML**: change, input, click, double_click, submit, stop, edit, clear, play, pause, end, start_recording, pause_recording, stop_recording, focus, blur, upload, release, select, stream, like, example_select, option_select, load, key_up, apply, delete, tick, undo, retry, expand, collapse, download, copy - **HighlightedText**: change, select - **Image**: clear, change, stream, select, upload, input - **ImageEditor**: clear, change, input, select, upload, apply - **ImageSlider**: clear, change, stream, select, upload, input - **JSON**: change - **Label**: change, select - **LinePlot**: select, double_click - **LoginButton**: click - **Markdown**: change, copy - **Model3D**: change, upload, edit, clear - **MultimodalTextbox**: change, input, select, submit, focus, blur, stop - **Navbar**: change - **Number**: change, input, submit, focus, blur - **ParamViewer**: change, upload - **Plot**: change - **Radio**: select, change, input - **ScatterPlot**: select, double_click - **SimpleImage**: clear, change, upload - **Slider**: change, input, release - **State**: change - **Textbox**: change, input, select, submit, focus, blur, stop, copy - **Timer**: tick - **UploadButton**: click, upload - **Video**: change, clear, start_recording, stop_recording, stop, play, pause, end, upload, input ## Prediction CLI The `gradio` CLI includes `info` and `predict` commands for interacting with Gradio apps programmatically. These are especially useful for coding agents that need to use Spaces in their workflows. ### `gradio info` — Discover endpoints and parameters ```bash gradio info ``` Returns a JSON payload describing all endpoints, their parameters (with types and defaults), and return values. ```bash gradio info gradio/calculator # { # "/predict": { # "parameters": [ # {"name": "num1", "required": true, "default": null, "type": {"type": "number"}}, # {"name": "operation", "required": true, "default": null, "type": {"enum": ["add", "subtract", "multiply", "divide"], "type": "string"}}, # {"name": "num2", "required": true, "default": null, "type": {"type": "number"}} # ], # "returns": [{"name": "output", "type": {"type": "number"}}], # "description": "" # } # } ``` File-type parameters show `"type": "filepath"` with instructions to include `"meta": {"_type": "gradio.FileData"}` — this signals the file will be uploaded to the remote server. ### `gradio predict` — Send predictions ```bash gradio predict ``` Returns a JSON object with named output keys. ```bash # Simple numeric prediction gradio predict gradio/calculator /predict '{"num1": 5, "operation": "multiply", "num2": 3}' # {"output": 15} # Image generation gradio predict black-forest-labs/FLUX.2-dev /infer '{"prompt": "A majestic dragon"}' # {"Result": "/tmp/gradio/.../image.webp", "Seed": 1117868604} # File upload (must include meta key) gradio predict gradio/image_mod /predict '{"image": {"path": "/path/to/image.png", "meta": {"_type": "gradio.FileData"}}}' # {"output": "/tmp/gradio/.../output.png"} ``` Both commands accept `--token` for accessing private Spaces. ## Additional Reference - [End-to-End Examples](examples.md) — complete working apps --- ## 14. huggingface-llm-trainer - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-llm-trainer/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-llm-trainer.md - **Licence:** Complete terms in LICENSE.txt — https://github.com/huggingface/skills/blob/main/skills/huggingface-llm-trainer/SKILL.md Bundled files (18), referenced from this skill's directory: - `references/gguf_conversion.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/gguf_conversion.md - `references/hardware_guide.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/hardware_guide.md - `references/hub_saving.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/hub_saving.md - `references/local_training_macos.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/local_training_macos.md - `references/reliability_principles.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/reliability_principles.md - `references/trackio_guide.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/trackio_guide.md - `references/training_methods.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/training_methods.md - `references/training_patterns.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/training_patterns.md - `references/troubleshooting.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/troubleshooting.md - `references/unsloth.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/references/unsloth.md - `scripts/convert_to_gguf.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/convert_to_gguf.py - `scripts/dataset_inspector.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/dataset_inspector.py - `scripts/estimate_cost.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/estimate_cost.py - `scripts/hf_benchmarks.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/hf_benchmarks.py - `scripts/train_dpo_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/train_dpo_example.py - `scripts/train_grpo_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/train_grpo_example.py - `scripts/train_sft_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/train_sft_example.py - `scripts/unsloth_sft_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-llm-trainer/scripts/unsloth_sft_example.py --- name: huggingface-llm-trainer description: Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs package, UV scripts with PEP 723 format, dataset preparation and validation, hardware selection, cost estimation, Trackio monitoring, Hub authentication, model selection/leaderboards and model persistence. Use for tasks involving cloud GPU training, GGUF conversion, or when users mention training on Hugging Face Jobs without local GPU setup. license: Complete terms in LICENSE.txt --- # TRL Training on Hugging Face Jobs ## Overview Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required—models train on cloud GPUs and results are automatically saved to the Hugging Face Hub. **TRL provides multiple training methods:** - **SFT** (Supervised Fine-Tuning) - Standard instruction tuning - **DPO** (Direct Preference Optimization) - Alignment from preference data - **GRPO** (Group Relative Policy Optimization) - Online RL training - **Reward Modeling** - Train reward models for RLHF **For detailed TRL method documentation:** ```python hf_doc_search("your query", product="trl") hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer") # SFT hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer") # DPO # etc. ``` **See also:** `references/training_methods.md` for method overviews and selection guidance ## When to Use This Skill Use this skill when users want to: - Fine-tune language models on cloud GPUs without local infrastructure - Train with TRL methods (SFT, DPO, GRPO, etc.) - Run training jobs on Hugging Face Jobs infrastructure - Convert trained models to GGUF for local deployment (Ollama, LM Studio, llama.cpp) - Ensure trained models are permanently saved to the Hub - Use modern workflows with optimized defaults ### When to Use Unsloth Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when: - **Limited GPU memory** - Unsloth uses ~60% less VRAM - **Speed matters** - Unsloth is ~2x faster - Training **large models (>13B)** - memory efficiency is critical - Training **Vision-Language Models (VLMs)** - Unsloth has `FastVisionModel` support See `references/unsloth.md` for complete Unsloth documentation and `scripts/unsloth_sft_example.py` for a production-ready training script. ## Key Directives When assisting with training jobs: 1. **ALWAYS use `hf_jobs()` MCP tool** - Submit jobs using `hf_jobs("uv", {...})`, NOT bash `trl-jobs` commands. The `script` parameter accepts Python code directly. Do NOT save to local files unless the user explicitly requests it. Pass the script content as a string to `hf_jobs()`. If user asks to "train a model", "fine-tune", or similar requests, you MUST create the training script AND submit the job immediately using `hf_jobs()`. 2. **Always include Trackio** - Every training script should include Trackio for real-time monitoring. Use example scripts in `scripts/` as templates. 3. **Provide job details after submission** - After submitting, provide job ID, monitoring URL, estimated time, and note that the user can request status checks later. 4. **Use example scripts as templates** - Reference `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc. as starting points. ## Local Script Execution Repository scripts use PEP 723 inline dependencies. Run them with `uv run`: ```bash uv run scripts/estimate_cost.py --help uv run scripts/dataset_inspector.py --help ``` ## Prerequisites Checklist Before starting any training job, verify: ### ✅ **Account & Authentication** - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) - Authenticated login: Check with `hf_whoami()` - **HF_TOKEN for Hub Push** ⚠️ CRITICAL - Training environment is ephemeral, must push to Hub or ALL training results are lost - Token must have write permissions - **MUST pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config** to make token available (the `$HF_TOKEN` syntax references your actual token value) ### ✅ **Dataset Requirements** - Dataset must exist on Hub or be loadable via `datasets.load_dataset()` - Format must match training method (SFT: "messages"/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only) - **ALWAYS validate unknown datasets** before GPU training to prevent format failures (see Dataset Validation section below) - Size appropriate for hardware (Demo: 50-100 examples on t4-small; Production: 1K-10K+ on a10g-large/a100-large) ### ⚠️ **Critical Settings** - **Timeout must exceed expected training time** - Default 30min is TOO SHORT for most training. Minimum recommended: 1-2 hours. Job fails and loses all progress if timeout is exceeded. - **Hub push must be enabled** - Config: `push_to_hub=True`, `hub_model_id="username/model-name"`; Job: `secrets={"HF_TOKEN": "$HF_TOKEN"}` ## Asynchronous Job Guidelines **⚠️ IMPORTANT: Training jobs run asynchronously and can take hours** ### Action Required **When user requests training:** 1. **Create the training script** with Trackio included (use `scripts/train_sft_example.py` as template) 2. **Submit immediately** using `hf_jobs()` MCP tool with script content inline - don't save to file unless user requests 3. **Report submission** with job ID, monitoring URL, and estimated time 4. **Wait for user** to request status checks - don't poll automatically ### Ground Rules - **Jobs run in background** - Submission returns immediately; training continues independently - **Initial logs delayed** - Can take 30-60 seconds for logs to appear - **User checks status** - Wait for user to request status updates - **Avoid polling** - Check logs only on user request; provide monitoring links instead ### After Submission **Provide to user:** - ✅ Job ID and monitoring URL - ✅ Expected completion time - ✅ Trackio dashboard URL - ✅ Note that user can request status checks later **Example Response:** ``` ✅ Job submitted successfully! Job ID: abc123xyz Monitor: https://huggingface.co/jobs/username/abc123xyz Expected time: ~2 hours Estimated cost: ~$10 The job is running in the background. Ask me to check status/logs when ready! ``` ## Quick Start: Three Approaches **💡 Tip for Demos:** For quick demos on smaller GPUs (t4-small), omit `eval_dataset` and `eval_strategy` to save ~40% memory. You'll still see training loss and learning progress. ### Sequence Length Configuration **TRL config classes use `max_length` (not `max_seq_length`)** to control tokenized sequence length: ```python # ✅ CORRECT - If you need to set sequence length SFTConfig(max_length=512) # Truncate sequences to 512 tokens DPOConfig(max_length=2048) # Longer context (2048 tokens) # ❌ WRONG - This parameter doesn't exist SFTConfig(max_seq_length=512) # TypeError! ``` **Default behavior:** `max_length=1024` (truncates from right). This works well for most training. **When to override:** - **Longer context**: Set higher (e.g., `max_length=2048`) - **Memory constraints**: Set lower (e.g., `max_length=512`) - **Vision models**: Set `max_length=None` (prevents cutting image tokens) **Usually you don't need to set this parameter at all** - the examples below use the sensible default. ### Approach 1: UV Scripts (Recommended—Default Choice) UV scripts use PEP 723 inline dependencies for clean, self-contained training. **This is the primary approach for Claude Code.** ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"] # /// from datasets import load_dataset from peft import LoraConfig from trl import SFTTrainer, SFTConfig import trackio dataset = load_dataset("trl-lib/Capybara", split="train") # Create train/eval split for monitoring dataset_split = dataset.train_test_split(test_size=0.1, seed=42) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset_split["train"], eval_dataset=dataset_split["test"], peft_config=LoraConfig(r=16, lora_alpha=32), args=SFTConfig( output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model", num_train_epochs=3, eval_strategy="steps", eval_steps=50, report_to="trackio", project="meaningful_prject_name", # project name for the training name (trackio) run_name="meaningful_run_name", # descriptive name for the specific training run (trackio) ) ) trainer.train() trainer.push_to_hub() """, "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Benefits:** Direct MCP tool usage, clean code, dependencies declared inline (PEP 723), no file saving required, full control **When to use:** Default choice for all training tasks in Claude Code, custom training logic, any scenario requiring `hf_jobs()` #### Working with Scripts ⚠️ **Important:** The `script` parameter accepts either inline code (as shown above) OR a URL. **Local file paths do NOT work.** **Why local paths don't work:** Jobs run in isolated Docker containers without access to your local filesystem. Scripts must be: - Inline code (recommended for custom training) - Publicly accessible URLs - Private repo URLs (with HF_TOKEN) **Common mistakes:** ```python # ❌ These will all fail hf_jobs("uv", {"script": "train.py"}) hf_jobs("uv", {"script": "./scripts/train.py"}) hf_jobs("uv", {"script": "/path/to/train.py"}) ``` **Correct approaches:** ```python # ✅ Inline code (recommended) hf_jobs("uv", {"script": "# /// script\n# dependencies = [...]\n# ///\n\n"}) # ✅ From Hugging Face Hub hf_jobs("uv", {"script": "https://huggingface.co/user/repo/resolve/main/train.py"}) # ✅ From GitHub hf_jobs("uv", {"script": "https://raw.githubusercontent.com/user/repo/main/train.py"}) # ✅ From Gist hf_jobs("uv", {"script": "https://gist.githubusercontent.com/user/id/raw/train.py"}) ``` **To use local scripts:** Upload to HF Hub first: ```bash hf repos create my-training-scripts --type model hf upload my-training-scripts ./train.py train.py # Use: https://huggingface.co/USERNAME/my-training-scripts/resolve/main/train.py ``` ### Approach 2: TRL Maintained Scripts (Official Examples) TRL provides battle-tested scripts for all methods. Can be run from URLs: ```python hf_jobs("uv", { "script": "https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py", "script_args": [ "--model_name_or_path", "Qwen/Qwen2.5-0.5B", "--dataset_name", "trl-lib/Capybara", "--output_dir", "my-model", "--push_to_hub", "--hub_model_id", "username/my-model" ], "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Benefits:** No code to write, maintained by TRL team, production-tested **When to use:** Standard TRL training, quick experiments, don't need custom code **Available:** Scripts are available from https://github.com/huggingface/trl/tree/main/examples/scripts ### Finding More UV Scripts on Hub The `uv-scripts` organization provides ready-to-use UV scripts stored as datasets on Hugging Face Hub: ```python # Discover available UV script collections dataset_search({"author": "uv-scripts", "sort": "downloads", "limit": 20}) # Explore a specific collection hub_repo_details(["uv-scripts/classification"], repo_type="dataset", include_readme=True) ``` **Popular collections:** ocr, classification, synthetic-data, vllm, dataset-creation ### Approach 3: HF Jobs CLI (Direct Terminal Commands) When the `hf_jobs()` MCP tool is unavailable, use the `hf jobs` CLI directly. **⚠️ CRITICAL: CLI Syntax Rules** ```bash # ✅ CORRECT syntax - flags BEFORE script URL hf jobs uv run --flavor a10g-large --timeout 2h --secrets HF_TOKEN "https://example.com/train.py" # ❌ WRONG - "run uv" instead of "uv run" hf jobs run uv "https://example.com/train.py" --flavor a10g-large # ❌ WRONG - flags AFTER script URL (will be ignored!) hf jobs uv run "https://example.com/train.py" --flavor a10g-large # ❌ WRONG - "--secret" instead of "--secrets" (plural) hf jobs uv run --secret HF_TOKEN "https://example.com/train.py" ``` **Key syntax rules:** 1. Command order is `hf jobs uv run` (NOT `hf jobs run uv`) 2. All flags (`--flavor`, `--timeout`, `--secrets`) must come BEFORE the script URL 3. Use `--secrets` (plural), not `--secret` 4. Script URL must be the last positional argument **Complete CLI example:** ```bash hf jobs uv run \ --flavor a10g-large \ --timeout 2h \ --secrets HF_TOKEN \ "https://huggingface.co/user/repo/resolve/main/train.py" ``` **Check job status via CLI:** ```bash hf jobs ps # List all jobs hf jobs logs # View logs hf jobs inspect # Job details hf jobs cancel # Cancel a job ``` ### Approach 4: TRL Jobs Package (Simplified Training) The `trl-jobs` package provides optimized defaults and one-liner training. ```bash uvx trl-jobs sft \ --model_name Qwen/Qwen2.5-0.5B \ --dataset_name trl-lib/Capybara ``` **Benefits:** Pre-configured settings, automatic Trackio integration, automatic Hub push, one-line commands **When to use:** User working in terminal directly (not Claude Code context), quick local experimentation **Repository:** https://github.com/huggingface/trl-jobs ⚠️ **In Claude Code context, prefer using `hf_jobs()` MCP tool (Approach 1) when available.** ## Hardware Selection | Model Size | Recommended Hardware | Cost (approx/hr) | Use Case | |------------|---------------------|------------------|----------| | <1B params | `t4-small` | ~$0.75 | Demos, quick tests only without eval steps | | 1-3B params | `t4-medium`, `l4x1` | ~$1.50-2.50 | Development | | 3-7B params | `a10g-small`, `a10g-large` | ~$3.50-5.00 | Production training | | 7-13B params | `a10g-large`, `a100-large` | ~$5-10 | Large models (use LoRA) | | 13B+ params | `a100-large`, `a10g-largex2` | ~$10-20 | Very large (use LoRA) | **GPU Flavors:** cpu-basic/upgrade/performance/xl, t4-small/medium, l4x1/x4, a10g-small/large/largex2/largex4, a100-large, h100/h100x8 **Guidelines:** - Use **LoRA/PEFT** for models >7B to reduce memory - Multi-GPU automatically handled by TRL/Accelerate - Start with smaller hardware for testing **See:** `references/hardware_guide.md` for detailed specifications ## Critical: Saving Results to Hub **⚠️ EPHEMERAL ENVIRONMENT—MUST PUSH TO HUB** The Jobs environment is temporary. All files are deleted when the job ends. If the model isn't pushed to Hub, **ALL TRAINING IS LOST**. ### Required Configuration **In training script/config:** ```python SFTConfig( push_to_hub=True, hub_model_id="username/model-name", # MUST specify hub_strategy="every_save", # Optional: push checkpoints ) ``` **In job submission:** ```python { "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Enables authentication } ``` ### Verification Checklist Before submitting: - [ ] `push_to_hub=True` set in config - [ ] `hub_model_id` includes username/repo-name - [ ] `secrets` parameter includes HF_TOKEN - [ ] User has write access to target repo **See:** `references/hub_saving.md` for detailed troubleshooting ## Timeout Management **⚠️ DEFAULT: 30 MINUTES—TOO SHORT FOR TRAINING** ### Setting Timeouts ```python { "timeout": "2h" # 2 hours (formats: "90m", "2h", "1.5h", or seconds as integer) } ``` ### Timeout Guidelines | Scenario | Recommended | Notes | |----------|-------------|-------| | Quick demo (50-100 examples) | 10-30 min | Verify setup | | Development training | 1-2 hours | Small datasets | | Production (3-7B model) | 4-6 hours | Full datasets | | Large model with LoRA | 3-6 hours | Depends on dataset | **Always add 20-30% buffer** for model/dataset loading, checkpoint saving, Hub push operations, and network delays. **On timeout:** Job killed immediately, all unsaved progress lost, must restart from beginning ## Choose a Base Model (Model Selection) **Identify models to train based on task type or benchmark results.** Use `scripts/hf_benchmarks.py` to identify top-performing models for specific tasks. This helps the user select a model as the base for training, whilst keeping size and hardware constraints in mind. ```bash # Get help on the benchmarks command: uv run scripts/hf_benchmarks.py --help ``` ### Example -- choosing an OCR base model ```bash # Search for benchmarks containing whose name contains the text `ocr` uv run scripts/hf_benchmarks.py search --query ocr # Get the ranked leaderboard for the allenai/olmOCR-bench benchmark uv run scripts/hf_benchmarks.py leaderboard allenai/olmOCR-bench ``` ## Cost Estimation **Offer to estimate cost when planning jobs with known parameters.** Use `scripts/estimate_cost.py`: ```bash uv run scripts/estimate_cost.py \ --model meta-llama/Llama-2-7b-hf \ --dataset trl-lib/Capybara \ --hardware a10g-large \ --dataset-size 16000 \ --epochs 3 ``` Output includes estimated time, cost, recommended timeout (with buffer), and optimization suggestions. **When to offer:** User planning a job, asks about cost/time, choosing hardware, job will run >1 hour or cost >$5 ## Example Training Scripts **Production-ready templates with all best practices:** Load these scripts for correctly: - **`scripts/train_sft_example.py`** - Complete SFT training with Trackio, LoRA, checkpoints - **`scripts/train_dpo_example.py`** - DPO training for preference learning - **`scripts/train_grpo_example.py`** - GRPO training for online RL These scripts demonstrate proper Hub saving, Trackio integration, checkpoint management, and optimized parameters. Pass their content inline to `hf_jobs()` or use as templates for custom scripts. ## Monitoring and Tracking **Trackio** provides real-time metrics visualization. See `references/trackio_guide.md` for complete setup guide. **Key points:** - Add `trackio` to dependencies - Configure trainer with `report_to="trackio" and run_name="meaningful_name"` ### Trackio Configuration Defaults **Use sensible defaults unless user specifies otherwise.** When generating training scripts with Trackio: **Default Configuration:** - **Space ID**: `{username}/trackio` (use "trackio" as default space name) - **Run naming**: Unless otherwise specified, name the run in a way the user will recognize (e.g., descriptive of the task, model, or purpose) - **Config**: Keep minimal - only include hyperparameters and model/dataset info - **Project Name**: Use a Project Name to associate runs with a particular Project **User overrides:** If user requests specific trackio configuration (custom space, run naming, grouping, or additional config), apply their preferences instead of defaults. This is useful for managing multiple jobs with the same configuration or keeping training scripts portable. See `references/trackio_guide.md` for complete documentation including grouping runs for experiments. ### Check Job Status ```python # List all jobs hf_jobs("ps") # Inspect specific job hf_jobs("inspect", {"job_id": "your-job-id"}) # View logs hf_jobs("logs", {"job_id": "your-job-id"}) ``` **Remember:** Wait for user to request status checks. Avoid polling repeatedly. ## Dataset Validation **Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.** ### Why Validate - 50%+ of training failures are due to dataset format issues - DPO especially strict: requires exact column names (`prompt`, `chosen`, `rejected`) - Failed GPU jobs waste $1-10 and 30-60 minutes - Validation on CPU costs ~$0.01 and takes <1 minute ### When to Validate **ALWAYS validate for:** - Unknown or custom datasets - DPO training (CRITICAL - 90% of datasets need mapping) - Any dataset not explicitly TRL-compatible **Skip validation for known TRL datasets:** - `trl-lib/ultrachat_200k`, `trl-lib/Capybara`, `HuggingFaceH4/ultrachat_200k`, etc. ### Usage ```python hf_jobs("uv", { "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py", "script_args": ["--dataset", "username/dataset-name", "--split", "train"] }) ``` The script is fast, and will usually complete synchronously. ### Reading Results The output shows compatibility for each training method: - **`✓ READY`** - Dataset is compatible, use directly - **`✗ NEEDS MAPPING`** - Compatible but needs preprocessing (mapping code provided) - **`✗ INCOMPATIBLE`** - Cannot be used for this method When mapping is needed, the output includes a **"MAPPING CODE"** section with copy-paste ready Python code. ### Example Workflow ```python # 1. Inspect dataset (costs ~$0.01, <1 min on CPU) hf_jobs("uv", { "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py", "script_args": ["--dataset", "argilla/distilabel-math-preference-dpo", "--split", "train"] }) # 2. Check output markers: # ✓ READY → proceed with training # ✗ NEEDS MAPPING → apply mapping code below # ✗ INCOMPATIBLE → choose different method/dataset # 3. If mapping needed, apply before training: def format_for_dpo(example): return { 'prompt': example['instruction'], 'chosen': example['chosen_response'], 'rejected': example['rejected_response'], } dataset = dataset.map(format_for_dpo, remove_columns=dataset.column_names) # 4. Launch training job with confidence ``` ### Common Scenario: DPO Format Mismatch Most DPO datasets use non-standard column names. Example: ``` Dataset has: instruction, chosen_response, rejected_response DPO expects: prompt, chosen, rejected ``` The validator detects this and provides exact mapping code to fix it. ## Converting Models to GGUF After training, convert models to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools. **What is GGUF:** - Optimized for CPU/GPU inference with llama.cpp - Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size - Compatible with Ollama, LM Studio, Jan, GPT4All, llama.cpp - Typically 2-8GB for 7B models (vs 14GB unquantized) **When to convert:** - Running models locally with Ollama or LM Studio - Reducing model size with quantization - Deploying to edge devices - Sharing models for local-first use **See:** `references/gguf_conversion.md` for complete conversion guide, including production-ready conversion script, quantization options, hardware requirements, usage examples, and troubleshooting. **Quick conversion:** ```python hf_jobs("uv", { "script": "", "flavor": "a10g-large", "timeout": "45m", "secrets": {"HF_TOKEN": "$HF_TOKEN"}, "env": { "ADAPTER_MODEL": "username/my-finetuned-model", "BASE_MODEL": "Qwen/Qwen2.5-0.5B", "OUTPUT_REPO": "username/my-model-gguf" } }) ``` ## Common Training Patterns See `references/training_patterns.md` for detailed examples including: - Quick demo (5-10 minutes) - Production with checkpoints - Multi-GPU training - DPO training (preference learning) - GRPO training (online RL) ## Common Failure Modes ### Out of Memory (OOM) **Fix (try in order):** 1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128. 2. Enable: `gradient_checkpointing=True` 3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc. ### Dataset Misformatted **Fix:** 1. Validate first with dataset inspector: ```bash uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \ --dataset name --split train ``` 2. Check output for compatibility markers (✓ READY, ✗ NEEDS MAPPING, ✗ INCOMPATIBLE) 3. Apply mapping code from inspector output if needed ### Job Timeout **Fix:** 1. Check logs for actual runtime: `hf_jobs("logs", {"job_id": "..."})` 2. Increase timeout with buffer: `"timeout": "3h"` (add 30% to estimated time) 3. Or reduce training: lower `num_train_epochs`, use smaller dataset, enable `max_steps` 4. Save checkpoints: `save_strategy="steps"`, `save_steps=500`, `hub_strategy="every_save"` **Note:** Default 30min is insufficient for real training. Minimum 1-2 hours. ### Hub Push Failures **Fix:** 1. Add to job: `secrets={"HF_TOKEN": "$HF_TOKEN"}` 2. Add to config: `push_to_hub=True`, `hub_model_id="username/model-name"` 3. Verify auth: `mcp__huggingface__hf_whoami()` 4. Check token has write permissions and repo exists (or set `hub_private_repo=True`) ### Missing Dependencies **Fix:** Add to PEP 723 header: ```python # /// script # dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio", "missing-package"] # /// ``` ## Troubleshooting **Common issues:** - Job times out → Increase timeout, reduce epochs/dataset, use smaller model/LoRA - Model not saved to Hub → Check push_to_hub=True, hub_model_id, secrets=HF_TOKEN - Out of Memory (OOM) → Reduce batch size, increase gradient accumulation, enable LoRA, use larger GPU - Dataset format error → Validate with dataset inspector (see Dataset Validation section) - Import/module errors → Add PEP 723 header with dependencies, verify format - Authentication errors → Check `mcp__huggingface__hf_whoami()`, token permissions, secrets parameter **See:** `references/troubleshooting.md` for complete troubleshooting guide ## Resources ### References (In This Skill) - `references/training_methods.md` - Overview of SFT, DPO, GRPO, KTO, PPO, Reward Modeling - `references/training_patterns.md` - Common training patterns and examples - `references/unsloth.md` - Unsloth for fast VLM training (~2x speed, 60% less VRAM) - `references/gguf_conversion.md` - Complete GGUF conversion guide - `references/trackio_guide.md` - Trackio monitoring setup - `references/hardware_guide.md` - Hardware specs and selection - `references/hub_saving.md` - Hub authentication troubleshooting - `references/troubleshooting.md` - Common issues and solutions - `references/local_training_macos.md` - Local training on macOS ### Scripts (In This Skill) - `scripts/train_sft_example.py` - Production SFT template - `scripts/train_dpo_example.py` - Production DPO template - `scripts/train_grpo_example.py` - Production GRPO template - `scripts/unsloth_sft_example.py` - Unsloth text LLM training template (faster, less VRAM) - `scripts/estimate_cost.py` - Estimate time and cost (offer when appropriate) - `scripts/convert_to_gguf.py` - Complete GGUF conversion script - `scripts/hf_benchmarks.py` - Search for benchmark results and leaderboards by task, alias or free text. ### External Scripts - [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Validate dataset format before training (use via `uv run` or `hf_jobs`) ### External Links - [TRL Documentation](https://huggingface.co/docs/trl) - [TRL Jobs Training Guide](https://huggingface.co/docs/trl/en/jobs_training) - [TRL Jobs Package](https://github.com/huggingface/trl-jobs) - [HF Jobs Documentation](https://huggingface.co/docs/huggingface_hub/guides/jobs) - [TRL Example Scripts](https://github.com/huggingface/trl/tree/main/examples/scripts) - [UV Scripts Guide](https://docs.astral.sh/uv/guides/scripts/) - [UV Scripts Organization](https://huggingface.co/uv-scripts) ## Key Takeaways 1. **Submit scripts inline** - The `script` parameter accepts Python code directly; no file saving required unless user requests 2. **Jobs are asynchronous** - Don't wait/poll; let user check when ready 3. **Always set timeout** - Default 30 min is insufficient; minimum 1-2 hours recommended 4. **Always enable Hub push** - Environment is ephemeral; without push, all results lost 5. **Include Trackio** - Use example scripts as templates for real-time monitoring 6. **Offer cost estimation** - When parameters are known, use `scripts/estimate_cost.py` 7. **Use UV scripts (Approach 1)** - Default to `hf_jobs("uv", {...})` with inline scripts; TRL maintained scripts for standard training; avoid bash `trl-jobs` commands in Claude Code 8. **Use hf_doc_fetch/hf_doc_search** for latest TRL documentation 9. **Validate dataset format** before training with dataset inspector (see Dataset Validation section) 10. **Choose appropriate hardware** for model size; use LoRA for models >7B --- ## 15. huggingface-local-models - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-local-models/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-local-models/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-local-models.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (3), referenced from this skill's directory: - `references/hardware.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-local-models/references/hardware.md - `references/hub-discovery.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-local-models/references/hub-discovery.md - `references/quantization.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-local-models/references/quantization.md --- name: huggingface-local-models description: "Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving." --- # Hugging Face Local Models Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with `llama-cli` or `llama-server`. ## Default Workflow 1. Search the Hub with `apps=llama.cpp`. 2. Open `https://huggingface.co/?local-app=llama.cpp`. 3. Prefer the exact HF local-app snippet and quant recommendation when it is visible. 4. Confirm exact `.gguf` filenames with `https://huggingface.co/api/models//tree/main?recursive=true`. 5. Launch with `llama-cli -hf :` or `llama-server -hf :`. 6. Fall back to `--hf-repo` plus `--hf-file` when the repo uses custom file naming. 7. Convert from Transformers weights only if the repo does not already expose GGUF files. ## Quick Start ### Install llama.cpp ```bash brew install llama.cpp winget install llama.cpp ``` ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp make ``` ### Authenticate for gated repos ```bash hf auth login ``` ### Search the Hub ```text https://huggingface.co/models?apps=llama.cpp&sort=trending https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending ``` ### Run directly from the Hub ```bash llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ### Run an exact GGUF file ```bash llama-server \ --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \ --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ -c 4096 ``` ### Convert only when no GGUF is available ```bash hf download --local-dir ./model-src python convert_hf_to_gguf.py ./model-src \ --outfile model-f16.gguf \ --outtype f16 llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M ``` ### Smoke test a local server ```bash llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "messages": [ {"role": "user", "content": "Write a limerick about exception handling"} ] }' ``` ## Quant Choice - Prefer the exact quant that HF marks as compatible on the `?local-app=llama.cpp` page. - Keep repo-native labels such as `UD-Q4_K_M` instead of normalizing them. - Default to `Q4_K_M` unless the repo page or hardware profile suggests otherwise. - Prefer `Q5_K_M` or `Q6_K` for code or technical workloads when memory allows. - Consider `Q3_K_M`, `Q4_K_S`, or repo-specific `IQ` / `UD-*` variants for tighter RAM or VRAM budgets. - Treat `mmproj-*.gguf` files as projector weights, not the main checkpoint. ## Load References - Read [hub-discovery.md](references/hub-discovery.md) for URL-first workflows, model search, tree API extraction, and command reconstruction. - Read [quantization.md](references/quantization.md) for format tables, model scaling, quality tradeoffs, and `imatrix`. - Read [hardware.md](references/hardware.md) for Metal, CUDA, ROCm, or CPU build and acceleration details. ## Resources - llama.cpp: `https://github.com/ggml-org/llama.cpp` - Hugging Face GGUF + llama.cpp docs: `https://huggingface.co/docs/hub/gguf-llamacpp` - Hugging Face Local Apps docs: `https://huggingface.co/docs/hub/main/local-apps` - Hugging Face Local Agents docs: `https://huggingface.co/docs/hub/agents-local` - GGUF converter Space: `https://huggingface.co/spaces/ggml-org/gguf-my-repo` --- ## 16. huggingface-lora-space-builder - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-lora-space-builder/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-lora-space-builder.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `references/adapting-to-the-lora.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/adapting-to-the-lora.md - `references/base-models/krea-2.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/base-models/krea-2.md - `references/base-models/ltx.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/base-models/ltx.md - `references/base-models/qwen-image.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/base-models/qwen-image.md - `references/creative-mode.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/creative-mode.md - `references/tasks.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/tasks.md - `references/zerogpu-and-publishing.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-lora-space-builder/references/zerogpu-and-publishing.md --- name: huggingface-lora-space-builder description: Build and publish a Gradio demo on Hugging Face Spaces for a user-provided LoRA. Use when someone asks to create, generate, ship, or publish a Space, demo, Gradio app, or playground for a LoRA — including LoRAs for Qwen-Image, Qwen-Image-Edit, LTX-Video, Wan, FLUX, SDXL, or other diffusion base models. Also triggers when someone describes a LoRA they trained or hosts on the Hub and wants to share it. Covers picking the right base pipeline and `diffusers` inference recipe, designing a UI tailored to the LoRA's task and inputs (Union/multi-task control, edit, video, image, etc.), respecting model-card recommendations (trigger words, steps, guidance, LoRA scale, example inputs), and shipping to ZeroGPU hardware as a private Space by default. --- # Gradio LoRA Space Builder Build and publish a Gradio demo on Hugging Face Spaces that runs inference with a user-provided LoRA. Use whenever someone asks to create, generate, ship, or publish "a Space", "a demo", "a Gradio app", or "a playground" for a LoRA — whether the base model is Qwen-Image, Qwen-Image-Edit, LTX, or another diffusion model. Also use when someone describes a LoRA they trained or hosts on the Hub and wants to share it. The default target is ZeroGPU hardware and the default inference library is `diffusers` when the base model supports it. The output is a real, published Space (private by default) that the user can try in the browser, not a local script. ## What "good" looks like for these demos The demo should feel handcrafted for this specific LoRA, not a generic template with the LoRA bolted on. Two LoRAs that share a task can still need different demos: a pose-control video LoRA and an outpainting video LoRA both take video in and produce video out, but the inputs the user provides, the preprocessing, and the controls are completely different. Recognizing that is the central job here. Concretely, a good demo: - Loads fast and runs fast — minimal model loading, sensible step count, no wasted computation per call. - Has a UI with exactly the controls this LoRA needs and nothing else. Excess sliders are a cost, not a feature. - Shows the user what's happening — progress, intermediate outputs where useful, the seed used, a clear error when input is missing. - Honors the LoRA's own recommendations from its model card: trigger words, recommended step count, recommended guidance scale, recommended LoRA scale, example inputs. - Is creative where creativity helps — interactive canvases, before/after sliders, side-by-side previews of intermediate processing — and plain where plainness is right. ## Workflow Work through these phases in order. Information gathered in one phase decides the next. 1. Gather the LoRA info needed to pick a pipeline and design a UI. 2. Pick the base pipeline and inference recipe. 3. Design the UI for this specific LoRA's task and inputs. 4. Write `app.py`, `requirements.txt`, and `README.md` together; show all three to the user for one batched approval. 5. Publish the Space (private). Don't drip-feed questions across multiple turns. Batch them. --- ## Phase 1 — Gather LoRA info Required: a LoRA repo on the Hub (e.g. `username/my-lora`). **First, try to read the repo without a token.** If it succeeds, the repo is public — proceed. If it fails with 401/403, the repo is private/gated and you need an authenticated session to read it. **Don't immediately ask for a token.** Check first whether the user is already authenticated. ```python from huggingface_hub import HfApi, get_token cached_token = get_token() # picks up HF_TOKEN env var or cached CLI login if cached_token: try: info = HfApi().whoami(token=cached_token) username = info["name"] # info also has fine-grained token scope info if applicable except Exception: cached_token = None # token exists but is invalid/expired ``` Then: - If a valid cached token exists *and* it can read the repo, use it. No prompt needed. - If no cached token, or the cached token can't read this private repo, ask the user for a token — once, with the explanation below. When asking for a token (and only when you actually need to ask): > I need a Hugging Face access token with **write** scope (to read the LoRA if it's private/gated, and to publish the Space). Create one at https://huggingface.co/settings/tokens. Paste it here. The same token will be reused for publishing in the final phase, so this is a one-time ask. **Then read what's in the repo:** - List the repo files (`huggingface_hub.HfApi().list_repo_files(repo_id)`). Look for `.safetensors`, `README.md`, example images/videos, multiple checkpoints. - Fetch the model card (`huggingface_hub.ModelCard.load(repo_id)`). The `data` dict has structured fields; the `text` has the README body. - If multiple `.safetensors` files exist, pick the right one — see "Picking the LoRA weights file" in `references/zerogpu-and-publishing.md`. Briefly: README-recommended file wins, then `pytorch_lora_weights.safetensors`, then latest training checkpoint, otherwise ask. **From the model card, try to determine:** - **Base model** — the `base_model` field, or text mentions in the README. Usually present. Use it to pick the pipeline reference file (see Phase 2). - **Task** — `pipeline_tag` if set, otherwise inferred from the base model and README text. The five tasks this skill handles: `text-to-image`, `image-to-image`, `text-to-video`, `image-to-video`, `video-to-video`. - **Trigger words** — often called "trigger word", "instance prompt", "activation word"; sometimes embedded in example prompts. - **Recommended inference recipe** — step count, guidance scale, true CFG scale, LoRA scale, resolution. Many LoRA cards include a Python snippet; trust its *parameters* (steps, guidance, CFG, LoRA scale, dtype). For *loading mechanics*, see `adapting-to-the-lora.md` — prefer `pipe.load_lora_weights(...)` over whatever loading approach the snippet uses. - **Example prompts and example media** — use these as Gradio examples in the UI. - **Sub-task / specific use case** — for image edits and video LoRAs, "what does this LoRA actually do" matters as much as the task category. A relighting LoRA, a face-swap LoRA, and a style LoRA all might be image-to-image, but the UI for each is different. **When something can't be inferred, ask the user — once, in a single batched message.** Format the question to make answering trivial. For task category, list the five options as a numbered choice. For sub-task, give a one-line description ("what does this LoRA do? e.g. 'relight portraits', 'apply manga style', 'extend videos to wider aspect ratios'"). Don't ask if you can already infer it confidently from the base model or README. If the model card has nothing helpful at all — no base model, no task, no example — surface that clearly: "The model card has no usable info. I'll need you to tell me: (1) base model, (2) what this LoRA does, (3) recommended step count and guidance scale if you know them." --- ## Phase 2 — Pick the base pipeline Two things to decide here: which reference file to load, and which pipeline class to use. They're not the same question — a base-model family file (e.g. `qwen-image.md`) covers multiple variants, and variants in the same family don't always share a pipeline class. Get this wrong and the Space loads but produces wrong output, or fails at startup. **Step 1 — Load the reference file for this base model family.** - `references/base-models/qwen-image.md` — covers Qwen-Image and Qwen-Image-Edit family (text-to-image and image-to-image). - `references/base-models/ltx.md` — covers LTX family (text-to-video, image-to-video, video-to-video, including IC-LoRAs). - `references/base-models/krea-2.md` — covers Krea 2 (K2), text-to-image (train on RAW, run inference/LoRAs on the Turbo distilled checkpoint). If the base model isn't in one of these files, this skill doesn't have first-class support yet. Tell the user, and ask whether they want to proceed by analogy (use the closest model's recipe and adjust) or stop. Don't guess silently. **Step 2 — Verify the pipeline class against the base model's own card. This step is mandatory, not optional.** A new base model variant might use the same pipeline class with a different repo path, or a new pipeline class entirely. Don't trust the reference file's table alone — it's best-effort and can lag a recent release. Verify before committing: ```python from huggingface_hub import ModelCard base_card = ModelCard.load(base_model_id) # Read base_card.text — find the diffusers inference snippet, note the pipeline class it imports. ``` The class imported in the base model card's diffusers snippet is the source of truth. Real examples where this matters: - `Qwen-Image-Edit` uses `QwenImageEditPipeline`. `Qwen-Image-Edit-2509` and `Qwen-Image-Edit-2511` use `QwenImageEditPlusPipeline` — different class, different default parameters, takes a list of images instead of one. A LoRA targeting 2511 loaded onto `QwenImageEditPipeline` produces broken output. - LTX-Video uses `LTXPipeline`/`LTXImageToVideoPipeline`/`LTXConditionPipeline`. LTX-2 uses `LTX2Pipeline` from a different module path. LTX-2.3 sometimes needs a native pipeline outside diffusers. If the base model card has no diffusers snippet at all, fall back to the reference file's table — and tell the user you're falling back, in case they know something the table doesn't. The cost of this verification is one Hub fetch and a few seconds of reading. The cost of skipping it is the failure mode the previous bullet describes — a "working" Space that's quietly using the wrong class. **Step 3 — Diffusers vs native pipeline.** Default to `diffusers` when the base model has a diffusers pipeline class. That's the case for Qwen-Image and Qwen-Image-Edit and most of LTX. Some LTX variants (notably LTX-2.3 with certain IC-LoRAs) need a native pipeline; the LTX reference says when. Diffusers gives standard `load_lora_weights` / `set_adapters` semantics; the native path needs LoRA-specific glue. --- ## Phase 3 — Design the UI for this LoRA Don't reach for a template. Reason from the LoRA's task and inputs to a UI. Read `references/tasks.md` for the per-task baseline UI patterns (what the standard inputs/outputs look like for T2I, I2I, T2V, I2V, V2V). Then read `references/adapting-to-the-lora.md`, which is about *thinking through what this specific LoRA needs* — beyond the task category. That file is the most important one in this skill. The same task can need very different UIs: a pose-control LTX LoRA needs a video input and a pose-extraction preview; an outpaint LTX LoRA needs an aspect-ratio picker and a black-margin preview; a relighting Flux LoRA needs an image and a brush canvas for indicating where to add light. None of those reduce to "the V2V template" or "the I2I template". **Self-check before writing the UI.** Write one sentence describing what a user does with this Space in 10 seconds. If that sentence doesn't distinguish this LoRA from any other LoRA of the same task, the UI isn't shaped enough yet. Examples that pass the self-check: - "Upload a video, pick a target aspect ratio, click Generate; the model fills the empty margins." - "Draw colored brush strokes where you want light, pick an illumination style, click Generate; the model relights the photo." - "Upload a video of someone moving and an image of a different character; the model produces a video of the character doing the motion." Examples that fail: - "Type a prompt and click generate." (Generic T2I — say more.) - "Upload an image and an instruction." (Generic edit — what kind of edit?) **Gradio component freshness.** Gradio's component set evolves. Before defaulting to plain components, consider whether something newer fits better — for example `gr.ImageSlider` for before/after on edit LoRAs, `gr.BrowserState` for persistent prefs, `@gr.render` for UIs that change based on input. If you're unsure whether a component exists or what its signature is, web-fetch the current Gradio docs at https://www.gradio.app/docs rather than guessing. **When stock and Hub custom components aren't enough — creative mode.** If the LoRA's natural input is a shape no Gradio component (built-in or on the Hub) expresses well — point sets, strokes, trajectories, multi-region annotations with metadata, 3D rotation gizmos, timeline scrubbers, anything where the user manipulates a thing on top of media — drop down to custom HTML/JS via `gr.HTML`. See `references/creative-mode.md` for the Gradio primitives (`gr.HTML`, `head=` injection, `elem_id` addressing, the two JS↔Python state-sync approaches), the discipline around defining a JSON wire format, and the pitfalls. Don't reach for creative mode just because it would be cool — reach for it when the LoRA's input shape demands it. And don't skip the Hub custom components rung above (e.g. `gradio_image_annotation`) before going fully bespoke. **`gr.Examples` for media-input Spaces.** When no fitting example media is available from the model's own repo, pull from the shared input pools — split by modality so the HF dataset viewer can render proper thumbnails: images at [`linoyts/repo-to-space-example-inputs`](https://huggingface.co/datasets/linoyts/repo-to-space-example-inputs), videos at [`linoyts/repo-to-space-example-videos`](https://huggingface.co/datasets/linoyts/repo-to-space-example-videos). Both are CC0 with `categories` + natural-language `caption` metadata and the same filter/rank recipe in each dataset README. Pick 2–3 that fit the task, preprocess to the shapes the model expects, and bake the copies into the Space. Set `cache_examples=True, cache_mode="lazy"` so the first click caches without running examples at build time (see `references/zerogpu-and-publishing.md`). --- ## Phase 4 — Write the Space files Before writing, tell the user concretely what's about to happen — name the actual files. Not "I'll write the three files" but something like: > "Now I'll write the three files needed to publish a Space: **`app.py`** (the Gradio demo and inference code), **`requirements.txt`** (Python dependencies), and **`README.md`** (Space configuration including ZeroGPU hardware setting). Then I'll show all three for your review before publishing." This anchors the user in what's being produced. Don't say "three files" without naming them — it's vague and signals lack of commitment to the deliverable. The three files are tightly coupled: `requirements.txt` is determined by what `app.py` imports, and the `README.md` YAML frontmatter sets the SDK version, hardware, and Space title that have to match. Write them together, then show all three to the user for approval in **one batched message** before publishing. Read `references/zerogpu-and-publishing.md` for the ZeroGPU rules. The non-obvious ones: - Models go on `cuda` at module level (not lazy-loaded inside the GPU function). ZeroGPU has a CUDA emulation that makes this work pre-allocation, and module-level placement is significantly faster than deferred placement. - The function that runs inference is decorated with `@spaces.GPU(duration=...)`. Pick a duration appropriate for the task — short for image generation, longer for video. - Don't use `torch.compile` — it's incompatible with ZeroGPU's process model. ### `app.py` Compose from the pieces decided in Phases 1–3. Don't paste from a template. Each section should be there because it's needed: - Imports — `gradio as gr`, `torch`, `spaces`, the pipeline class, anything the preprocessing needs. - Constants — `LORA_REPO`, `BASE_MODEL`, recommended step count, guidance, LoRA scale, trigger word. - Module-level model load — pipeline `from_pretrained`, `.to("cuda")`, `load_lora_weights`. If the LoRA repo is private, pass `token=os.environ["HF_TOKEN"]`. - Preprocessing functions (if any) — pose extraction, padding, mask building, etc. CPU code can run at module level; GPU code needs to be inside a `@spaces.GPU` function. - The inference function — decorated with `@spaces.GPU(duration=...)`. Validates inputs, applies trigger word, builds the pipeline kwargs, returns outputs. - The Gradio Blocks — the UI from Phase 3, wired to the inference function. Common things to get right: - Return the actually-used seed alongside the result so the user can reproduce. - `gr.Progress(track_tqdm=True)` on the inference function surfaces diffusers' internal progress bar. - Validate inputs — raise `gr.Error("Please upload an image first.")` when a required input is missing, rather than letting the pipeline fail with a cryptic error. - On `gr.Examples`, use `cache_examples=True, cache_mode="lazy"` — plain `cache_examples=True` runs examples at build time and fails on ZeroGPU; lazy mode defers caching to the first user click. - When `gr.Examples` has `fn=`, clicking a row calls `fn` with **only** the `inputs=` values, positionally — so every `fn` parameter not in `inputs` needs a default, or the click raises `TypeError: missing N required positional arguments`. The run event wires all components, so this passes manual and smoke tests and only breaks on the example click. Fix: default the extra params (keep `inputs=[prompt]`), or list every input with full example rows, or drop `fn`/`outputs` so a click just fills the fields. ### `requirements.txt` Don't ship a fixed minimal list and hope for the best. The "minimal" list works for plain T2I LoRAs and breaks the moment the base model has a vision-language text encoder, video output, or any non-trivial preprocessing. **Derive `requirements.txt` from what the Space actually needs**, in this order: 1. **Every top-level non-stdlib import in `app.py`.** If `app.py` does `import cv2`, `requirements.txt` has `opencv-python`. If it does `from controlnet_aux import OpenposeDetector`, `requirements.txt` has `controlnet-aux`. Walk the imports mechanically. (Note the exclusions in the next paragraph — some imports are runtime built-ins and don't need to be listed.) 2. **What the base-model reference's "Required dependencies" subsection says.** Each base-model file lists the non-obvious extras the pipeline pulls in — `torchvision` for Qwen-Image (Qwen 2.5-VL text encoder), `imageio[ffmpeg]` for LTX (video export), etc. Include all of them. These are the deps that aren't picked up from imports because the pipeline's components import them transitively at load time. 3. **What the LoRA's own model card explicitly mentions installing.** If the LoRA README has its own `pip install` block, lift the deps from there. 4. **The diffusers/ML stack:** `diffusers`, `transformers`, `accelerate`, `peft`, `safetensors`. Default to plain (unpinned). Switch `diffusers` to `git+https://github.com/huggingface/diffusers` if the base-model reference says the model needs it (recent releases often do — Qwen-Image-Edit-2511 is a current example). **What *not* to list in `requirements.txt`:** - **`gradio`** — controlled by the `sdk_version:` field in `README.md`'s YAML frontmatter, not by `requirements.txt`. Listing it in requirements is at best ignored, at worst causes a version conflict with the SDK. Set the version in the README only. - **`torch`** — provided by the Space runtime. Only add if you need a specific version pinned (rare, and usually a sign something else is wrong). - **`spaces`** — provided by the Space runtime. Only add if you need a specific version pinned. - **`huggingface_hub`** — provided by the Space runtime. Only add if you need a specific version pinned. These four come pre-installed in the ZeroGPU container. Listing them anyway is the kind of "include rather than skip" instinct that's right for non-baseline deps but wrong for baseline ones, because pinning conflicts with the runtime's managed versions. **Bias for everything else: include rather than skip when uncertain.** A package the Space doesn't actually use causes a slightly slower build. A missing required package causes a startup-time crash that's much harder for the user to diagnose. These costs aren't symmetric — the test failure that prompted this rule was exactly the second kind. **But two specific deps are *not* safe to add reflexively** because they routinely cause more problems than they solve on ZeroGPU: - `xformers` — pinned to specific torch versions, frequent source of conflicts. The ZeroGPU runtime ships torch 2.8+, so any pinned `xformers` version must support that. Additional gotcha on Blackwell: xformers' FA3 dispatch mis-gates the hardware (FA3 kernels are Hopper-only at `sm_90a`, but the dispatcher gates on `device_capability >= (9, 0)`, which also matches Blackwell) and crashes at kernel launch with `CUDA invalid argument`. If a Space using xformers attention hits this, disable FA3 dispatch at module load: ```python try: from xformers.ops.fmha import _set_use_fa3 _set_use_fa3(False) except Exception: pass ``` Only include `xformers` if `app.py` actually uses it. - `flash-attn` — needs a build step, often fails to install. Same torch 2.8+ alignment caveat as `xformers`. Only include if `app.py` actually uses it. **Pin other versions only when you have a reason** (e.g. a known incompatibility, or matching a recipe from the model card). ### `README.md` Spaces are configured by the YAML frontmatter at the top of `README.md`. This frontmatter is what selects ZeroGPU. ``` --- title: emoji: 🎨 colorFrom: pink colorTo: purple sdk: gradio sdk_version: app_file: app.py pinned: false hardware: zero-a10g short_description: models: - - --- # A short description with links to the LoRA and base model. ``` Key fields: - `sdk: gradio` — required for ZeroGPU. - `sdk_version` — match the Gradio version you wrote against. Look up the current version (`pip index versions gradio`, or check https://www.gradio.app) rather than guessing. - `hardware: zero-a10g` — the legacy string for ZeroGPU. The actual hardware is NVIDIA RTX Pro 6000 Blackwell, but the identifier is `zero-a10g`. ZeroGPU is available to PRO, Team, and Enterprise accounts; if the user isn't subscribed, the Space will fall back to CPU. Mention this if you suspect they aren't on PRO. - `models:` — list base and LoRA repos. This enables Hub caching and discovery. - `short_description` — appears on the Space tile. **Keep it short (~60 characters or less).** The Hub's YAML validator rejects long values with a 400 from `https://huggingface.co/api/validate-yaml`, which surfaces as an `HfHubHTTPError` during `create_repo` or `upload_file`. The exact server-side limit isn't documented and may change, so target the visible-tile-length range rather than pushing right up to a cap. If you do hit the 400, the fix is almost always to shorten this field. One sentence describing what the Space does is plenty — the README body below the YAML is where you put longer prose. ### Single batched approval — order of operations matters The discipline here is **write all three files first, then show them all together in one message**. Not "write app.py → talk about it → write requirements → talk about it → write README → talk about it." That rhythm produces three approval moments even if you don't explicitly ask for approval, because the user is being asked to react after each file. Concretely: 1. **Write `app.py`, `requirements.txt`, and `README.md` in succession with no intervening prose.** No commentary between files. No "Now I'll write the next one." No description of what each file does as you produce it. Just the three files, back to back. 2. **Then, in a single message, ask for approval covering all three at once.** Something like: "Here's the Space — `app.py` (N lines), `requirements.txt`, and `README.md`. Review and confirm to publish, or tell me what to change." 3. The user responds once, covering whatever they want changed across any of the three files. What to avoid: - Walking through `app.py`'s structure or design choices after writing it but before writing the others. Save commentary for either the pre-writing announcement (Phase 4 opening) or the single approval message after all three exist. - Asking "ready for the next one?" or "want me to continue with requirements?" — those are implicit per-file approvals. - Showing one file inline and offering to "show the next when you're ready" — same trap. - Treating any of the three files as optional or as a follow-up. They are produced together as one deliverable. If the user interrupts after seeing the first file with feedback or a question, that's fine — engage with it — but the rule still applies: the next time you produce code, produce all remaining files together, not one at a time. --- ## Phase 5 — Publish the Space Use the authenticated session from Phase 1. Default to **private**, so the user can vet the Space before flipping it public. Confirm the target username with the user before creating: "I'll publish to `{username}/{space_name}` — confirm?" ```python from huggingface_hub import HfApi, SpaceHardware api = HfApi(token=hf_token) username = api.whoami()["name"] repo_id = f"{username}/{space_name}" api.create_repo( repo_id=repo_id, repo_type="space", space_sdk="gradio", space_hardware=SpaceHardware.ZERO_A10G, private=True, exist_ok=True, ) # Upload files for path in ["app.py", "requirements.txt", "README.md"]: api.upload_file(path_or_fileobj=path, path_in_repo=path, repo_id=repo_id, repo_type="space") ``` If the LoRA repo itself is private/gated, the Space needs the token at runtime to download the LoRA. Set it as a Space secret: ```python api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value=HF_TOKEN) ``` …and in `app.py`, load the LoRA with `token=os.environ["HF_TOKEN"]`. **After upload**, run the smoke-test below before sharing — the build runs asynchronously and silent failures (wrong `weight_name`, missing dep, wrong pipeline class) only surface at first inference. **Once the smoke-test passes**, share the Space URL (`https://huggingface.co/spaces/{repo_id}`) and tell the user the Space is private — they'll need to be logged in to view it. Note that the build takes a few minutes; the logs are at `https://huggingface.co/spaces/{repo_id}/logs/container` if anything fails. **Publish-time failures (before the build starts):** - **`HfHubHTTPError: 400 Bad Request` from `https://huggingface.co/api/validate-yaml`** during `create_repo` or `upload_file`. The README YAML failed server-side validation. By far the most common cause is a `short_description` that's too long; sometimes a stray field or malformed value. Fix: shorten `short_description` to ~60 characters and retry. If shortening doesn't fix it, look for typos in field names or invalid values (e.g. unsupported colors in `colorFrom`/`colorTo`, an invalid `hardware` string). - **403 on `create_repo`** with `space_hardware="zero-a10g"`: user isn't on PRO/Team/Enterprise, so they can't request ZeroGPU at creation time. Fix: retry `create_repo` without `space_hardware`, leave `hardware: zero-a10g` in the README YAML — the Space gets created on CPU. The user can then either upgrade to PRO (auto-promotes to ZeroGPU) or apply for a [community GPU grant](https://huggingface.co/docs/hub/spaces-gpus#community-gpu-grants) (request via the Space's hardware settings). - **401/403 on `upload_file`**: token doesn't have write scope. Fix: ask the user for a write-scoped token. **Common build failures (after the build starts):** - LoRA `weight_name` mismatch in `load_lora_weights` → check the actual filename via `list_repo_files`. - Base model is gated and the token wasn't set as a Space secret. - ZeroGPU not allocated (user not on PRO) → Space falls back to CPU and is unusably slow. - Diffusers version doesn't recognize the pipeline class → pin to git diffusers in `requirements.txt`. - Missing dependency at module load → see `requirements.txt` derivation rules above; the most common case is a transitive dep like `torchvision` for Qwen-Image's text encoder. If a build fails, offer to read the logs and propose a fix. --- ## Phase 6 — Smoke-test the Space Before declaring the Space done and handing the URL to the user, exercise it once end-to-end. Several failure modes (wrong `weight_name`, wrong pipeline class, missing transitive dep, gated-base-model token issue) build cleanly and only surface at first inference. The `gradio` Python package ships a CLI that does exactly this — `gradio info` returns the endpoint signature, `gradio predict` runs an actual inference. Both ship with the `gradio` pip dependency the Space already needs, so they're available in any environment where this skill ran. **Step 1 — Wait for the build.** `create_repo` returns immediately, but the container image is still building. Poll `HfApi().get_space_runtime(repo_id).stage` until it reaches `RUNNING`: ```python import time from huggingface_hub import HfApi api = HfApi(token=hf_token) while True: stage = api.get_space_runtime(repo_id).stage if stage == "RUNNING": break if stage in {"BUILD_ERROR", "RUNTIME_ERROR", "CONFIG_ERROR"}: raise RuntimeError(f"Build failed: {stage}. Logs: https://huggingface.co/spaces/{repo_id}/logs/container") time.sleep(15) ``` If the build fails, fetch the container logs (`https://huggingface.co/spaces/{repo_id}/logs/container`), read the traceback, and propose a fix. Don't run `gradio info` against a Space that isn't running — it'll hang or 503. **Step 2 — Verify the endpoint signature.** `gradio info {repo_id} --token {hf_token}` returns the exposed endpoints and their parameter types. Read the output and confirm: (a) the endpoint exists (default is `/predict`, but Blocks Spaces often have a custom name from the Python function name), (b) the parameters in order match what `app.py` declares, (c) file-typed params show `"type": "filepath"` as expected. If any of this is off, the user-facing UI may still appear correct but API calls will fail — fix and re-upload. **Step 3 — Run one real inference.** Pick the lightest viable input — the simplest example from the LoRA card, or one of the `gr.Examples` entries. Pass `--token` for private Spaces. For file inputs, the payload uses `{"path": "...", "meta": {"_type": "gradio.FileData"}}`. ```bash # Text-to-image: gradio predict {repo_id} /predict '{"prompt": "...", "aspect_ratio": "1:1", ...}' --token $HF_TOKEN # Image-to-image (file input): gradio predict {repo_id} /predict '{"input_image": {"path": "/tmp/sample.jpg", "meta": {"_type": "gradio.FileData"}}, "prompt": "..."}' --token $HF_TOKEN ``` If you don't have a local sample image for I2I, lift one from the LoRA repo (`hf_hub_download(repo_id, filename="example.png")`) or the base model card. **Caveat for creative-mode Spaces.** `gradio info` and `gradio predict` only exercise the Python endpoint — they tell you nothing about whether custom JS in a `gr.HTML` widget works. If the Space uses creative mode (see `references/creative-mode.md`), after the API smoke-test passes, **open the Space URL in a browser and verify the interaction once** before sharing. Server-side green plus broken JS is the most common failure mode for these. **Step 4 — Interpret the result.** - **Returns successfully and the output looks plausible** → done. Share the URL. - **HTTPError 503 / "Space is sleeping"** → the Space spun down between steps 1 and 3. Wake it (`api.restart_space(repo_id)`) and retry. - **Inference error mentioning `weight_name` / `safetensors`** → the LoRA filename in `app.py` doesn't match the actual file in the LoRA repo. Re-check `list_repo_files`, fix `weight_name=`, re-upload `app.py`. - **Inference error mentioning a missing pipeline class or attribute** → diffusers version too old. Switch `requirements.txt` to `git+https://github.com/huggingface/diffusers` and re-upload. - **`ImportError` at module load** → missing dep. Add it to `requirements.txt` and re-upload. The runtime logs (`/logs/run`) name the missing package. - **OOM** → reduce default resolution or step count, or pick a smaller base variant. - **Timeout / hangs** → bump `@spaces.GPU(duration=...)` and re-upload. The smoke-test exists to convert these from "user discovers it and reports back" to "you discover it and fix it before sharing." Don't skip it because the build went green — green-build-broken-inference is the most common failure mode for Spaces with a non-trivial pipeline. --- ## What to avoid - A generic "one demo for all LoRAs" template. The whole point of this skill is to tailor. - Lazy-loading the model inside the GPU function. Slow on ZeroGPU, and hides startup errors until first request. - `torch.compile`. Not supported on ZeroGPU. - `cache_examples=True` without `cache_mode="lazy"` on ZeroGPU. - `gr.Examples(fn=…)` whose `inputs` don't cover the inference function's required args — builds green, then crashes on the first example click with a missing-positional-argument error. - Uploading the LoRA weights into the Space repo. Pull from the LoRA's own Hub repo at runtime. - Asking for the HF token only at the end, then discovering the LoRA was private all along and you couldn't read the model card. - Exposing every diffusers knob. Pick the 1–3 controls that matter for this LoRA. - Long preambles in the chat reply once the Space is published. The Space URL is the deliverable; keep the wrap-up brief. <!-- chapter:end slug=huggingface-lora-space-builder --> --- <!-- chapter:begin slug=huggingface-paper-publisher position=17 --> ## 17. huggingface-paper-publisher - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-paper-publisher/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-paper-publisher.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `examples/example_usage.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/examples/example_usage.md - `references/quick_reference.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/references/quick_reference.md - `scripts/paper_manager.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/scripts/paper_manager.py - `templates/arxiv.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/templates/arxiv.md - `templates/ml-report.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/templates/ml-report.md - `templates/modern.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/templates/modern.md - `templates/standard.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-paper-publisher/templates/standard.md <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-paper-publisher description: Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles. --- # Overview This skill provides comprehensive tools for AI engineers and researchers to publish, manage, and link research papers on the Hugging Face Hub. It streamlines the workflow from paper creation to publication, including integration with arXiv, model/dataset linking, and authorship management. ## Integration with HF Ecosystem - **Paper Pages**: Index and discover papers on Hugging Face Hub - **arXiv Integration**: Automatic paper indexing from arXiv IDs - **Model/Dataset Linking**: Connect papers to relevant artifacts through metadata - **Authorship Verification**: Claim and verify paper authorship - **Research Article Template**: Generate professional, modern scientific papers # Version 1.0.0 # Dependencies The included script uses PEP 723 inline dependencies. Prefer `uv run` over manual environment setup. - huggingface_hub>=0.26.0 - pyyaml>=6.0.3 - requests>=2.32.5 - markdown>=3.5.0 - python-dotenv>=1.2.1 # Core Capabilities ## 1. Paper Page Management - **Index Papers**: Add papers to Hugging Face from arXiv - **Claim Authorship**: Verify and claim authorship on published papers - **Manage Visibility**: Control which papers appear on your profile - **Paper Discovery**: Find and explore papers in the HF ecosystem ## 2. Link Papers to Artifacts - **Model Cards**: Add paper citations to model metadata - **Dataset Cards**: Link papers to datasets via README - **Automatic Tagging**: Hub auto-generates arxiv:<PAPER_ID> tags - **Citation Management**: Maintain proper attribution and references ## 3. Research Article Creation - **Markdown Templates**: Generate professional paper formatting - **Modern Design**: Clean, readable research article layouts - **Dynamic TOC**: Automatic table of contents generation - **Section Structure**: Standard scientific paper organization - **LaTeX Math**: Support for equations and technical notation ## 4. Metadata Management - **YAML Frontmatter**: Proper model/dataset card metadata - **Citation Tracking**: Maintain paper references across repositories - **Version Control**: Track paper updates and revisions - **Multi-Paper Support**: Link multiple papers to single artifacts # Usage Instructions The skill includes Python scripts in `scripts/` for paper publishing operations. ### Prerequisites - Run scripts with `uv run` (dependencies are resolved from the script header) - Set `HF_TOKEN` environment variable with Write-access token > **All paths are relative to the directory containing this SKILL.md file.** > Before running any script, first `cd` to that directory or use the full path. ### Method 1: Index Paper from arXiv Add a paper to Hugging Face Paper Pages from arXiv. **Basic Usage:** ```bash uv run scripts/paper_manager.py index \ --arxiv-id "2301.12345" ``` **Check If Paper Exists:** ```bash uv run scripts/paper_manager.py check \ --arxiv-id "2301.12345" ``` **Direct URL Access:** You can also visit `https://huggingface.co/papers/{arxiv-id}` directly to index a paper. ### Method 2: Link Paper to Model/Dataset Add paper references to model or dataset README with proper YAML metadata. **Add to Model Card:** ```bash uv run scripts/paper_manager.py link \ --repo-id "username/model-name" \ --repo-type "model" \ --arxiv-id "2301.12345" ``` **Add to Dataset Card:** ```bash uv run scripts/paper_manager.py link \ --repo-id "username/dataset-name" \ --repo-type "dataset" \ --arxiv-id "2301.12345" ``` **Add Multiple Papers:** ```bash uv run scripts/paper_manager.py link \ --repo-id "username/model-name" \ --repo-type "model" \ --arxiv-ids "2301.12345,2302.67890,2303.11111" ``` **With Custom Citation:** ```bash uv run scripts/paper_manager.py link \ --repo-id "username/model-name" \ --repo-type "model" \ --arxiv-id "2301.12345" \ --citation "$(cat citation.txt)" ``` #### How Linking Works When you add an arXiv paper link to a model or dataset README: 1. The Hub extracts the arXiv ID from the link 2. A tag `arxiv:<PAPER_ID>` is automatically added to the repository 3. Users can click the tag to view the Paper Page 4. The Paper Page shows all models/datasets citing this paper 5. Papers are discoverable through filters and search ### Method 3: Claim Authorship Verify your authorship on papers published on Hugging Face. **Start Claim Process:** ```bash uv run scripts/paper_manager.py claim \ --arxiv-id "2301.12345" \ --email "your.email@institution.edu" ``` **Manual Process:** 1. Navigate to your paper's page: `https://huggingface.co/papers/{arxiv-id}` 2. Find your name in the author list 3. Click your name and select "Claim authorship" 4. Wait for admin team verification **Check Authorship Status:** ```bash uv run scripts/paper_manager.py check-authorship \ --arxiv-id "2301.12345" ``` ### Method 4: Manage Paper Visibility Control which verified papers appear on your public profile. **List Your Papers:** ```bash uv run scripts/paper_manager.py list-my-papers ``` **Toggle Visibility:** ```bash uv run scripts/paper_manager.py toggle-visibility \ --arxiv-id "2301.12345" \ --show true ``` **Manage in Settings:** Navigate to your account settings → Papers section to toggle "Show on profile" for each paper. ### Method 5: Create Research Article Generate a professional markdown-based research paper using modern templates. **Create from Template:** ```bash uv run scripts/paper_manager.py create \ --template "standard" \ --title "Your Paper Title" \ --output "paper.md" ``` **Available Templates:** - `standard` - Traditional scientific paper structure - `modern` - Clean, web-friendly format inspired by Distill - `arxiv` - arXiv-style formatting - `ml-report` - Machine learning experiment report **Generate Complete Paper:** ```bash uv run scripts/paper_manager.py create \ --template "modern" \ --title "Fine-Tuning Large Language Models with LoRA" \ --authors "Jane Doe, John Smith" \ --abstract "$(cat abstract.txt)" \ --output "paper.md" ``` **Convert to HTML:** ```bash uv run scripts/paper_manager.py convert \ --input "paper.md" \ --output "paper.html" \ --style "modern" ``` ### Paper Template Structure **Standard Research Paper Sections:** ```markdown --- title: Your Paper Title authors: Jane Doe, John Smith affiliations: University X, Lab Y date: 2025-01-15 arxiv: 2301.12345 tags: [machine-learning, nlp, fine-tuning] --- # Abstract Brief summary of the paper... # 1. Introduction Background and motivation... # 2. Related Work Previous research and context... # 3. Methodology Approach and implementation... # 4. Experiments Setup, datasets, and procedures... # 5. Results Findings and analysis... # 6. Discussion Interpretation and implications... # 7. Conclusion Summary and future work... # References ``` **Modern Template Features:** - Dynamic table of contents - Responsive design for web viewing - Code syntax highlighting - Interactive figures and charts - Math equation rendering (LaTeX) - Citation management - Author affiliation linking ### Commands Reference **Index Paper:** ```bash uv run scripts/paper_manager.py index --arxiv-id "2301.12345" ``` **Link to Repository:** ```bash uv run scripts/paper_manager.py link \ --repo-id "username/repo-name" \ --repo-type "model|dataset|space" \ --arxiv-id "2301.12345" \ [--citation "Full citation text"] \ [--create-pr] ``` **Claim Authorship:** ```bash uv run scripts/paper_manager.py claim \ --arxiv-id "2301.12345" \ --email "your.email@edu" ``` **Manage Visibility:** ```bash uv run scripts/paper_manager.py toggle-visibility \ --arxiv-id "2301.12345" \ --show true|false ``` **Create Research Article:** ```bash uv run scripts/paper_manager.py create \ --template "standard|modern|arxiv|ml-report" \ --title "Paper Title" \ [--authors "Author1, Author2"] \ [--abstract "Abstract text"] \ [--output "filename.md"] ``` **Convert Markdown to HTML:** ```bash uv run scripts/paper_manager.py convert \ --input "paper.md" \ --output "paper.html" \ [--style "modern|classic"] ``` **Check Paper Status:** ```bash uv run scripts/paper_manager.py check --arxiv-id "2301.12345" ``` **List Your Papers:** ```bash uv run scripts/paper_manager.py list-my-papers ``` **Search Papers:** ```bash uv run scripts/paper_manager.py search --query "transformer attention" ``` ### YAML Metadata Format When linking papers to models or datasets, proper YAML frontmatter is required: **Model Card Example:** ```yaml --- language: - en license: apache-2.0 tags: - text-generation - transformers - llm library_name: transformers --- # Model Name This model is based on the approach described in [Our Paper](https://arxiv.org/abs/2301.12345). ## Citation ```bibtex @article{doe2023paper, title={Your Paper Title}, author={Doe, Jane and Smith, John}, journal={arXiv preprint arXiv:2301.12345}, year={2023} } ``` ``` **Dataset Card Example:** ```yaml --- language: - en license: cc-by-4.0 task_categories: - text-generation - question-answering size_categories: - 10K<n<100K --- # Dataset Name Dataset introduced in [Our Paper](https://arxiv.org/abs/2301.12345). For more details, see the [paper page](https://huggingface.co/papers/2301.12345). ``` The Hub automatically extracts arXiv IDs from these links and creates `arxiv:2301.12345` tags. ### Integration Examples **Workflow 1: Publish New Research** ```bash # 1. Create research article uv run scripts/paper_manager.py create \ --template "modern" \ --title "Novel Fine-Tuning Approach" \ --output "paper.md" # 2. Edit paper.md with your content # 3. Submit to arXiv (external process) # Upload to arxiv.org, get arXiv ID # 4. Index on Hugging Face uv run scripts/paper_manager.py index --arxiv-id "2301.12345" # 5. Link to your model uv run scripts/paper_manager.py link \ --repo-id "your-username/your-model" \ --repo-type "model" \ --arxiv-id "2301.12345" # 6. Claim authorship uv run scripts/paper_manager.py claim \ --arxiv-id "2301.12345" \ --email "your.email@edu" ``` **Workflow 2: Link Existing Paper** ```bash # 1. Check if paper exists uv run scripts/paper_manager.py check --arxiv-id "2301.12345" # 2. Index if needed uv run scripts/paper_manager.py index --arxiv-id "2301.12345" # 3. Link to multiple repositories uv run scripts/paper_manager.py link \ --repo-id "username/model-v1" \ --repo-type "model" \ --arxiv-id "2301.12345" uv run scripts/paper_manager.py link \ --repo-id "username/training-data" \ --repo-type "dataset" \ --arxiv-id "2301.12345" uv run scripts/paper_manager.py link \ --repo-id "username/demo-space" \ --repo-type "space" \ --arxiv-id "2301.12345" ``` **Workflow 3: Update Model with Paper Reference** ```bash # 1. Get current README hf download username/model-name README.md # 2. Add paper link uv run scripts/paper_manager.py link \ --repo-id "username/model-name" \ --repo-type "model" \ --arxiv-id "2301.12345" \ --citation "Full citation for the paper" # The script will: # - Add YAML metadata if missing # - Insert arXiv link in README # - Add formatted citation # - Preserve existing content ``` ### Best Practices 1. **Paper Indexing** - Index papers as soon as they're published on arXiv - Include full citation information in model/dataset cards - Use consistent paper references across related repositories 2. **Metadata Management** - Add YAML frontmatter to all model/dataset cards - Include proper licensing information - Tag with relevant task categories and domains 3. **Authorship** - Claim authorship on papers where you're listed as author - Use institutional email addresses for verification - Keep paper visibility settings updated 4. **Repository Linking** - Link papers to all relevant models, datasets, and Spaces - Include paper context in README descriptions - Add BibTeX citations for easy reference 5. **Research Articles** - Use templates consistently within projects - Include code and data links in papers - Generate web-friendly HTML versions for sharing ### Advanced Usage **Batch Link Papers:** ```bash # Link multiple papers to one repository for arxiv_id in "2301.12345" "2302.67890" "2303.11111"; do uv run scripts/paper_manager.py link \ --repo-id "username/model-name" \ --repo-type "model" \ --arxiv-id "$arxiv_id" done ``` **Extract Paper Info:** ```bash # Get paper metadata from arXiv uv run scripts/paper_manager.py info \ --arxiv-id "2301.12345" \ --format "json" ``` **Generate Citation:** ```bash # Create BibTeX citation uv run scripts/paper_manager.py citation \ --arxiv-id "2301.12345" \ --format "bibtex" ``` **Validate Links:** ```bash # Check all paper links in a repository uv run scripts/paper_manager.py validate \ --repo-id "username/model-name" \ --repo-type "model" ``` ### Error Handling - **Paper Not Found**: arXiv ID doesn't exist or isn't indexed yet - **Permission Denied**: HF_TOKEN lacks write access to repository - **Invalid YAML**: Malformed metadata in README frontmatter - **Authorship Failed**: Email doesn't match paper author records - **Already Claimed**: Another user has claimed authorship - **Rate Limiting**: Too many API requests in short time ### Troubleshooting **Issue**: "Paper not found on Hugging Face" - **Solution**: Visit `hf.co/papers/{arxiv-id}` to trigger indexing **Issue**: "Authorship claim not verified" - **Solution**: Wait for admin review or contact HF support with proof **Issue**: "arXiv tag not appearing" - **Solution**: Ensure README includes proper arXiv URL format **Issue**: "Cannot link to repository" - **Solution**: Verify HF_TOKEN has write permissions **Issue**: "Template rendering errors" - **Solution**: Check markdown syntax and YAML frontmatter format ### Resources and References - **Hugging Face Paper Pages**: [hf.co/papers](https://huggingface.co/papers) - **Model Cards Guide**: [hf.co/docs/hub/model-cards](https://huggingface.co/docs/hub/en/model-cards) - **Dataset Cards Guide**: [hf.co/docs/hub/datasets-cards](https://huggingface.co/docs/hub/en/datasets-cards) - **Research Article Template**: [tfrere/research-article-template](https://huggingface.co/spaces/tfrere/research-article-template) - **arXiv Format Guide**: [arxiv.org/help/submit](https://arxiv.org/help/submit) ### Integration with tfrere's Research Template This skill complements [tfrere's research article template](https://huggingface.co/spaces/tfrere/research-article-template) by providing: - Automated paper indexing workflows - Repository linking capabilities - Metadata management tools - Citation generation utilities You can use tfrere's template for writing, then use this skill to publish and link the paper on Hugging Face Hub. ### Common Patterns **Pattern 1: New Paper Publication** ```bash # Write → Publish → Index → Link uv run scripts/paper_manager.py create --template modern --output paper.md # (Submit to arXiv) uv run scripts/paper_manager.py index --arxiv-id "2301.12345" uv run scripts/paper_manager.py link --repo-id "user/model" --arxiv-id "2301.12345" ``` **Pattern 2: Existing Paper Discovery** ```bash # Search → Check → Link uv run scripts/paper_manager.py search --query "transformers" uv run scripts/paper_manager.py check --arxiv-id "2301.12345" uv run scripts/paper_manager.py link --repo-id "user/model" --arxiv-id "2301.12345" ``` **Pattern 3: Author Portfolio Management** ```bash # Claim → Verify → Organize uv run scripts/paper_manager.py claim --arxiv-id "2301.12345" uv run scripts/paper_manager.py list-my-papers uv run scripts/paper_manager.py toggle-visibility --arxiv-id "2301.12345" --show true ``` ### API Integration **Python Script Example:** ```python from scripts.paper_manager import PaperManager pm = PaperManager(hf_token="your_token") # Index paper pm.index_paper("2301.12345") # Link to model pm.link_paper( repo_id="username/model", repo_type="model", arxiv_id="2301.12345", citation="Full citation text" ) # Check status status = pm.check_paper("2301.12345") print(status) ``` ### Future Enhancements Planned features for future versions: - Support for non-arXiv papers (conference proceedings, journals) - Automatic citation formatting from DOI - Paper comparison and versioning tools - Collaborative paper writing features - Integration with LaTeX workflows - Automated figure and table extraction - Paper metrics and impact tracking <!-- chapter:end slug=huggingface-paper-publisher --> --- <!-- chapter:begin slug=huggingface-papers position=18 --> ## 18. huggingface-papers - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-papers/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-papers/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-papers.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-papers description: Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. Use when the user shares a Hugging Face paper page URL, an arXiv URL or ID, or asks to summarize, explain, or analyze an AI research paper. --- # Hugging Face Paper Pages Hugging Face Paper pages (hf.co/papers) is a platform built on top of arXiv (arxiv.org), specifically for research papers in the field of artificial intelligence (AI) and computer science. Hugging Face users can submit their paper at hf.co/papers/submit, which features it on the Daily Papers feed (hf.co/papers). Each day, users can upvote papers and comment on papers. Each paper page allows authors to: - claim their paper (by clicking their name on the `authors` field). This makes the paper page appear on their Hugging Face profile. - link the associated model checkpoints, datasets and Spaces by including the HF paper or arXiv URL in the model card, dataset card or README of the Space - link the Github repository and/or project page URLs - link the HF organization. This also makes the paper page appear on the Hugging Face organization page. Whenever someone mentions a HF paper or arXiv abstract/PDF URL in a model card, dataset card or README of a Space repository, the paper will be automatically indexed. Note that not all papers indexed on Hugging Face are also submitted to daily papers. The latter is more a manner of promoting a research paper. Papers can only be submitted to daily papers up until 14 days after their publication date on arXiv. The Hugging Face team has built an easy-to-use API to interact with paper pages. Content of the papers can be fetched as markdown, or structured metadata can be returned such as author names, linked models/datasets/spaces, linked Github repo and project page. ## When to Use - User shares a Hugging Face paper page URL (e.g. `https://huggingface.co/papers/2602.08025`) - User shares a Hugging Face markdown paper page URL (e.g. `https://huggingface.co/papers/2602.08025.md`) - User shares an arXiv URL (e.g. `https://arxiv.org/abs/2602.08025` or `https://arxiv.org/pdf/2602.08025`) - User mentions a arXiv ID (e.g. `2602.08025`) - User asks you to summarize, explain, or analyze an AI research paper ## Parsing the paper ID It's recommended to parse the paper ID (arXiv ID) from whatever the user provides: | Input | Paper ID | | --- | --- | | `https://huggingface.co/papers/2602.08025` | `2602.08025` | | `https://huggingface.co/papers/2602.08025.md` | `2602.08025` | | `https://arxiv.org/abs/2602.08025` | `2602.08025` | | `https://arxiv.org/pdf/2602.08025` | `2602.08025` | | `2602.08025v1` | `2602.08025v1` | | `2602.08025` | `2602.08025` | This allows you to provide the paper ID into any of the hub API endpoints mentioned below. ### Fetch the paper page as markdown The content of a paper can be fetched as markdown like so: ```bash curl -s "https://huggingface.co/papers/{PAPER_ID}.md" ``` This should return the Hugging Face paper page as markdown. This relies on the HTML version of the paper at https://arxiv.org/html/{PAPER_ID}. There are 2 exceptions: - Not all arXiv papers have an HTML version. If the HTML version of the paper does not exist, then the content falls back to the HTML of the Hugging Face paper page. - If it results in a 404, it means the paper is not yet indexed on hf.co/papers. See [Error handling](#error-handling) for info. Alternatively, you can request markdown from the normal paper page URL, like so: ```bash curl -s -H "Accept: text/markdown" "https://huggingface.co/papers/{PAPER_ID}" ``` ### Paper Pages API Endpoints All endpoints use the base URL `https://huggingface.co`. #### Get structured metadata Fetch the paper metadata as JSON using the Hugging Face REST API: ```bash curl -s "https://huggingface.co/api/papers/{PAPER_ID}" ``` This returns structured metadata that can include: - authors (names and Hugging Face usernames, in case they have claimed the paper) - media URLs (uploaded when submitting the paper to Daily Papers) - summary (abstract) and AI-generated summary - project page and GitHub repository - organization and engagement metadata (number of upvotes) To find models linked to the paper, use: ```bash curl https://huggingface.co/api/models?filter=arxiv:{PAPER_ID} ``` To find datasets linked to the paper, use: ```bash curl https://huggingface.co/api/datasets?filter=arxiv:{PAPER_ID} ``` To find spaces linked to the paper, use: ```bash curl https://huggingface.co/api/spaces?filter=arxiv:{PAPER_ID} ``` #### Claim paper authorship Claim authorship of a paper for a Hugging Face user: ```bash curl "https://huggingface.co/api/settings/papers/claim" \ --request POST \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $HF_TOKEN" \ --data '{ "paperId": "{PAPER_ID}", "claimAuthorId": "{AUTHOR_ENTRY_ID}", "targetUserId": "{USER_ID}" }' ``` - Endpoint: `POST /api/settings/papers/claim` - Body: - `paperId` (string, required): arXiv paper identifier being claimed - `claimAuthorId` (string): author entry on the paper being claimed, 24-char hex ID - `targetUserId` (string): HF user who should receive the claim, 24-char hex ID - Response: paper authorship claim result, including the claimed paper ID #### Get daily papers Fetch the Daily Papers feed: ```bash curl -s -H "Authorization: Bearer $HF_TOKEN" \ "https://huggingface.co/api/daily_papers?p=0&limit=20&date=2017-07-21&sort=publishedAt" ``` - Endpoint: `GET /api/daily_papers` - Query parameters: - `p` (integer): page number - `limit` (integer): number of results, between 1 and 100 - `date` (string): RFC 3339 full-date, for example `2017-07-21` - `week` (string): ISO week, for example `2024-W03` - `month` (string): month value, for example `2024-01` - `submitter` (string): filter by submitter - `sort` (enum): `publishedAt` or `trending` - Response: list of daily papers #### List papers List arXiv papers sorted by published date: ```bash curl -s -H "Authorization: Bearer $HF_TOKEN" \ "https://huggingface.co/api/papers?cursor={CURSOR}&limit=20" ``` - Endpoint: `GET /api/papers` - Query parameters: - `cursor` (string): pagination cursor - `limit` (integer): number of results, between 1 and 100 - Response: list of papers #### Search papers Perform hybrid semantic and full-text search on papers: ```bash curl -s -H "Authorization: Bearer $HF_TOKEN" \ "https://huggingface.co/api/papers/search?q=vision+language&limit=20" ``` This searches over the paper title, authors, and content. - Endpoint: `GET /api/papers/search` - Query parameters: - `q` (string): search query, max length 250 - `limit` (integer): number of results, between 1 and 120 - Response: matching papers #### Index a paper Insert a paper from arXiv by ID. If the paper is already indexed, only its authors can re-index it: ```bash curl "https://huggingface.co/api/papers/index" \ --request POST \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $HF_TOKEN" \ --data '{ "arxivId": "{ARXIV_ID}" }' ``` - Endpoint: `POST /api/papers/index` - Body: - `arxivId` (string, required): arXiv ID to index, for example `2301.00001` - Pattern: `^\d{4}\.\d{4,5}$` - Response: empty JSON object on success #### Update paper links Update the project page, GitHub repository, or submitting organization for a paper. The requester must be the paper author, the Daily Papers submitter, or a papers admin: ```bash curl "https://huggingface.co/api/papers/{PAPER_OBJECT_ID}/links" \ --request POST \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $HF_TOKEN" \ --data '{ "projectPage": "https://example.com", "githubRepo": "https://github.com/org/repo", "organizationId": "{ORGANIZATION_ID}" }' ``` - Endpoint: `POST /api/papers/{paperId}/links` - Path parameters: - `paperId` (string, required): Hugging Face paper object ID - Body: - `githubRepo` (string, nullable): GitHub repository URL - `organizationId` (string, nullable): organization ID, 24-char hex ID - `projectPage` (string, nullable): project page URL - Response: empty JSON object on success ## Error Handling - **404 on `https://huggingface.co/papers/{PAPER_ID}` or `md` endpoint**: the paper is not indexed on Hugging Face paper pages yet. - **404 on `/api/papers/{PAPER_ID}`**: the paper may not be indexed on Hugging Face paper pages yet. - **Paper ID not found**: verify the extracted arXiv ID, including any version suffix ### Fallbacks If the Hugging Face paper page does not contain enough detail for the user's question: - Check the regular paper page at `https://huggingface.co/papers/{PAPER_ID}` - Fall back to the arXiv page or PDF for the original source: - `https://arxiv.org/abs/{PAPER_ID}` - `https://arxiv.org/pdf/{PAPER_ID}` ## Notes - No authentication is required for public paper pages. - Write endpoints such as claim authorship, index paper, and update paper links require `Authorization: Bearer $HF_TOKEN`. - Prefer the `.md` endpoint for reliable machine-readable output. - Prefer `/api/papers/{PAPER_ID}` when you need structured JSON fields instead of page markdown. <!-- chapter:end slug=huggingface-papers --> --- <!-- chapter:begin slug=huggingface-spaces position=19 --> ## 19. huggingface-spaces - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-spaces/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-spaces.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (14), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/README.md - `references/3d-cuda-extensions.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/3d-cuda-extensions.md - `references/3d-generation.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/3d-generation.md - `references/3d-gsplat.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/3d-gsplat.md - `references/3d-models.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/3d-models.md - `references/3d-outputs.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/3d-outputs.md - `references/buckets.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/buckets.md - `references/debugging.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/debugging.md - `references/gradio.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/gradio.md - `references/grants.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/grants.md - `references/inference-providers.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/inference-providers.md - `references/known-errors.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/known-errors.md - `references/requirements.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/requirements.md - `references/zerogpu.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-spaces/references/zerogpu.md <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-spaces description: Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buckets, inference providers, community grants. Use whenever the user asks to create or host an app on Hugging Face, port code onto ZeroGPU, fix a Space that won't build or run, or otherwise work with `hf spaces …`, `@spaces.GPU`, Space README frontmatter, or the `spaces` Python package. --- # Hugging Face Spaces Hugging Face Spaces host machine-learning applications. There are 1M+ today; each Space is a git repo. This skill covers creating, building, debugging, and maintaining them. ## 0. Getting ready Before anything else: 1. Check the `hf` CLI is installed: `which hf`. If not, `pip install -U huggingface_hub`. 2. Check the user is logged in: `hf auth whoami`. If not, run `hf auth login` — it prints a URL and a one-time code; ask the user to open the URL and enter the code, then login completes automatically (OAuth, no token needed). Alternatively, pass a write-scoped token from https://huggingface.co/settings/tokens with `--token`. 3. Note `whoami`'s `canPay` and `isPro` flags — they gate hardware choices below. A free (`isPro=False`) account can only host Static Spaces and up to 2 ZeroGPU Spaces. The `hf-cli` skill teaches an agent every `hf` command and is the recommended companion to this one. Install it with `hf skills add hf-cli` (add `--claude --global` to install for Claude Code as well, user-level). ## 1. What a Space is A Space is a git repo with three possible SDKs: - **Gradio** — most Spaces. Python, fast iteration, supports ZeroGPU. - **Docker** — arbitrary container. Use when you need a non-Python stack or a pre-built template (Streamlit, Argilla, Shiny, etc. — full list at https://huggingface.co/docs/hub/spaces-sdks-docker). Does **not** support ZeroGPU. - **Static** — plain HTML, or a React/Svelte/Vue project built at deploy time. Use for in-browser ML (transformers.js / WebGPU / WebAssembly / onnxruntime-web), project pages, interactive reports, or Spaces that orchestrate other Spaces. No hardware needed. ### Hardware tiers Static Spaces are free for everyone and need no hardware. **Gradio and Docker Spaces run on compute and require a paid plan to create** — PRO for personal accounts, Team or Enterprise for organizations — with one exception: **free personal accounts in good standing (verified email, account older than 30 days) can host up to 2 ZeroGPU Spaces.** So on a free account ZeroGPU is the *only* way to host a Gradio Space. `cpu-basic` is not the safe fallback it used to be — it is gated too. **ZeroGPU (`zero-a10g`)** — dynamic, per-request GPU allocation on NVIDIA RTX PRO 6000 Blackwell (sm_120). Two sizes: `large` (half MIG, 48 GB, 1× quota) and `xlarge` (full, 96 GB, 2× quota). Free for the Space creator; Space visitors consume their own daily quota (~5 min free / 40 min Pro / 60 min Enterprise). **Gradio-only**, **PyTorch-first**. Hosting caps per account: **2** free personal, **10** PRO, **50** Team / Enterprise org. **`cpu-basic`** — 2 vCPU / 16 GB, no hourly cost but needs a paid plan. For data viz, API-proxy Spaces, small CPU-bound models. **Dedicated GPU** (T4, L4, A10G, L40S, A100, H200) — billed to the Space creator by the hour. List + pricing: `hf spaces hardware`. Only the creator can attach these, and only if `canPay=True`. Use when ZeroGPU genuinely doesn't fit — non-PyTorch main model with heavy init, very-large-model long-context inference, etc. If the user needs hardware they can't pay for — a dedicated GPU, or a Gradio Space beyond the free 2-ZeroGPU cap — they can still create a **Static** Space (free for everyone), push the app there, and request a community grant. See [`references/grants.md`](references/grants.md). For the authoritative reference: https://huggingface.co/docs/hub/spaces-overview ## 2. Look for an existing demo first Before deciding how to build anything, search for prior art: ```bash hf spaces search "<model name or task>" --sdk gradio --limit 10 ``` If someone has built a similar Space, read its `app.py` and `requirements.txt` — that gives you the working pattern. Saves a lot of blind iteration. Mention to the user what you found before committing to an approach. ## 3. Decide SDK and hardware Follow the user's explicit request first. If they were vague: - **Default for a public ML demo**: Gradio + ZeroGPU. Use this unless something below applies. - **The model's only inference path is non-PyTorch** (ONNX / TF / JAX / vLLM as the MAIN model, with heavy init): dedicated GPU. - But: marginal non-torch tools (a small ONNX preprocessor, a TF utility) inside a torch-main pipeline are fine on ZeroGPU. The hijack only patches torch; init the non-torch lib inside `@spaces.GPU` and pay the short per-call init cost. - **Tiny / CPU-bound model, or API-proxy Space**: `cpu-basic` — but it needs a paid plan. On a free account, put it on `zero-a10g` with a no-op decorated function (ZeroGPU requires at least one) and keep the real work outside it — nothing ever requests a GPU, so no quota is burned. See [`references/inference-providers.md`](references/inference-providers.md). - **Browser-side ML or project page**: Static. - **Container with non-Python stack**: Docker. ### Sourcing the model - **GitHub repo** — clone locally to read structure. If it already has a Gradio demo, the minimal viable path is to adapt it onto ZeroGPU (see [`references/zerogpu.md`](references/zerogpu.md)). Otherwise: read the README + inference code, prefer the PyTorch path, estimate VRAM (bf16 ≈ `params_B × 2` GB; 48 GB fits ≤24B params at bf16, or much larger with quantization — see [`references/zerogpu.md`](references/zerogpu.md) for quantization on ZeroGPU). - **HF model repo** — read its README, follow any linked GitHub. - **Paper / blog post** — look for an official or unofficial implementation. Don't reimplement unless trivial or the user explicitly asks. - **Vague request** — search Spaces first; surface results. If the model genuinely won't fit, check **Inference Providers** as an alternative: see [`references/inference-providers.md`](references/inference-providers.md). This avoids hosting the model at all. ## 4. Create the Space ```bash hf repos create <namespace>/<name> --type space --space-sdk <gradio|docker|static> \ [--flavor zero-a10g|cpu-basic|<paid-flavor>] \ [--secrets KEY=val] [--env KEY=val] \ --public|--private|--protected \ --exist-ok ``` - `--space-sdk` is required. - `--flavor` selects hardware. `zero-a10g` is the (legacy) identifier for ZeroGPU. Omitting it gives `cpu-basic` — which is itself gated behind a paid plan, so on a free account pass `--flavor zero-a10g` explicitly. Run `hf spaces hardware` for the full paid list and pricing. - Visibility: `--public` (anyone can view), `--private` (only you), `--protected` (app is reachable but git repo / Files tab is private). - `--secrets KEY=val` becomes an environment variable inside the Space and is **not** visible to visitors. Use for API keys, gated-repo tokens (`HF_TOKEN=hf_…`), etc. Can also be set later via `hf spaces secrets set <id> KEY=val`. - `--env KEY=val` is **visible to visitors** — use only for non-sensitive config (`GRADIO_SSR_MODE=false`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, etc.). > Note: `hardware:` in the README YAML is silently ignored — hardware is only set via `--flavor` at creation, or later via `hf spaces settings <id> --hardware <name>`. ## 5. Build the app The Space now exists at `https://huggingface.co/spaces/<namespace>/<name>` but is empty. ### README.md frontmatter Always required: ```yaml --- title: ... emoji: 🚀 # pick something representative colorFrom: blue # red|yellow|green|blue|indigo|purple|pink|gray (only these) colorTo: indigo sdk: gradio # gradio | docker | static sdk_version: 6.15.1 # latest stable unless you have a reason* app_file: app.py # gradio only (docker / static use Dockerfile / index.html) short_description: ... # ≤ 60 chars (server rejects longer) python_version: "3.12" # ZeroGPU officially supports 3.10.13 and 3.12.12 startup_duration_timeout: 30m # default; bump to 1h for big LLMs / heavy downloads --- ``` \* Default to the current latest stable, and **look up what that is** (`pip index versions gradio`, or the version a freshly-created Space defaults to) — the number above is a placeholder that goes stale, don't reuse it. Only pin older when the latest genuinely doesn't work for this Space: a custom component pins it, or you're adapting an existing demo and don't want to rewrite for 5.x→6.x breaking changes. If you need a 5.x, pick `5.50.0` (latest of the series; still supports custom components). All frontmatter options: https://huggingface.co/docs/hub/spaces-config-reference ### Minimal ZeroGPU Gradio app ```python import spaces # MUST come before torch / diffusers / transformers import torch import gradio as gr from diffusers import DiffusionPipeline pipe = DiffusionPipeline.from_pretrained("<repo>", torch_dtype=torch.bfloat16).to("cuda") @spaces.GPU(duration=60) def generate(prompt: str): """Generate an image from a text prompt.""" # docstring → API / MCP tool description return pipe(prompt).images[0] gr.Interface(fn=generate, inputs=gr.Text(), outputs=gr.Image()).launch(mcp_server=True) ``` Three rules — full treatment in [`references/zerogpu.md`](references/zerogpu.md): 1. **`import spaces` before torch / any CUDA-touching import.** It monkey-patches `torch.cuda.*`; once CUDA is initialized in the main process, it's too late. 2. **Load the model at module scope, `.to("cuda")` eagerly.** ZeroGPU intercepts the call, packs weights to disk, and streams them into VRAM on the first `@spaces.GPU` entry. Lazy loading inside the decorator costs every user. 3. **Decorate the function Gradio binds.** Estimate `duration` to the realistic worst case (smaller = higher queue priority and tighter quota check). For input-dependent runtime, pass a callable. ### Examples, docstrings, and MCP - **Add `gr.Examples` whenever it makes sense** (the app takes input and representative inputs exist) — prefer the model/repo's own official examples. Keep example rows to the few inputs a user actually varies (prompt, image) and give the handler defaults for the rest (steps, seed, guidance) so a row is `["a prompt"]`, not a wall of knobs. Use `cache_examples=True, cache_mode="lazy"`. See [`references/gradio.md`](references/gradio.md). - **Give every API-triggered function a docstring and type hints.** Each Gradio event handler is exposed over the API; the docstring + signature are what a caller — and the MCP tool schema — sees. - **Launch with `demo.launch(mcp_server=True)`** (Gradio 5+) so the Space doubles as an MCP server: each API function becomes an MCP tool described by its docstring and hints. ### requirements.txt Short version: - **Do NOT list**: `gradio`, `spaces`, `huggingface_hub` (preinstalled and platform-managed; pinning them causes resolution failures or silently breaks the ZeroGPU runtime). - **Do list if you use them**: `torchvision`, `torchaudio` (not preinstalled), plus everything else (`diffusers`, `transformers`, `accelerate`, `sentencepiece`, …). - ZeroGPU only accepts torch `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default to leaving torch unpinned (the runtime preinstalls the latest). Only pin when a dep forces it. - For prebuilt CUDA-extension wheels (`flash_attn`, `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`): use the prebuilt Blackwell wheels at `https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/tree/main/wheels`. Full mapping + caveats in [`references/requirements.md`](references/requirements.md). ### Per-SDK depth - **Gradio patterns** (themes, `gr.Examples`, streaming, custom HTML components, `gr.Server`): [`references/gradio.md`](references/gradio.md). - **Docker**: https://huggingface.co/docs/hub/spaces-sdks-docker. Examples: `hf spaces list --filter docker`. - **Static**: https://huggingface.co/docs/hub/spaces-sdks-static. For built SPAs, set `app_build_command: npm run build` and `app_file: dist/index.html` in frontmatter. - **ZeroGPU specifics** (decorator semantics, sizing, AoTI, generators, concurrency, pickle / `gr.State` across the worker boundary): [`references/zerogpu.md`](references/zerogpu.md) — read this whenever the Space targets ZeroGPU. ## 6. Iterate on the Space, not locally Try to build a release candidate from the user quest locally and push it — then use the live URL as your test loop. The Space environment is the only one that matters; do not try to test locally. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing. Push files with `hf upload <namespace>/<name> . --repo-type space`. **`--repo-type space` is required** — `hf upload` defaults to a *model* repo and will otherwise upload to (and silently create) a model repo of the same name. Add `--exclude "**/__pycache__/**"` so local bytecode caches aren't committed into the Space. Once pushed, pick the cheapest update mechanism for each change — hot-reload for pure Python edits, `hf upload` for code-only files hot-reload can't touch, full rebuild only when `requirements.txt` / `Dockerfile` / README frontmatter actually changed. Full ladder + footguns (hot-reload poisoning factory reboot, runtime.sha lag, etc.) in [`references/debugging.md`](references/debugging.md). ## 7. Verify Don't trust `RUNNING` alone — the app can be running but broken. Four steps, in order: **A. Alive?** Stage + hardware: ```bash hf spaces info <ns>/<name> --expand runtime ``` **B. Logs clean post-boot?** Read the run log to confirm startup finished without warnings or silent fallbacks: ```bash hf spaces logs <ns>/<name> --tail 200 ``` Look for model-load completion, no import warnings, no "falling back to CPU" / dtype downgrade messages, no `RUNNING` masking a half-broken app. **C. API actually responds.** With logs still tailing in another terminal (`hf spaces logs <ns>/<name> --follow`), call the endpoint: ```python from gradio_client import Client, handle_file import os c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"], httpx_kwargs={"timeout": 600}) print(c.view_api()) # discover endpoints — don't guess result = c.predict(..., api_name="/generate") ``` **D. Sniff output AND logs.** HTTP 200 ≠ correct output. Check both: ```python head = open(result, "rb").read(16) # glTF / \x89PNG / RIFF…WEBP / RIFF…WAVE / [4:8]==b"ftyp" → png/jpg/webp/wav/mp4 ``` And look at the run log emitted during the call — silent fallbacks (model snapping to a different size, missing optional dep, dtype downgrade) only show up there. Full smoke-test patterns (streaming endpoints, OAuth-gated Spaces, `gr.Server` custom routes): [`references/debugging.md`](references/debugging.md). ## 8. Permanent storage (buckets) Spaces are stateless — `/data` is wiped on restart. If the Space needs to persist user uploads, generations, logs, or interact with a long-lived store, mount a **bucket**: ```bash hf buckets create <ns>/<bucket-name> # --private optional hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data # read-write at /data ``` Buckets are paid storage; check `canPay` and confirm with the user. Full patterns (read-fast / write-durable, public bucket URLs, model-cache anti-pattern): [`references/buckets.md`](references/buckets.md). ## 9. When things break Order of operations: 1. Read the logs: `hf spaces logs <id> --build --follow` (build error) or `hf spaces logs <id> --follow` (runtime error). Find the **first** error, not the last. 2. Grep [`references/known-errors.md`](references/known-errors.md) for the error string. Check if this is a known issue before trying your own fix — most common ZeroGPU / Gradio / dependency errors have a 1–2 line fix there. 3. Iterate using the cheapest rung from [`references/debugging.md`](references/debugging.md). The vast majority of issues resolve with log-reading + smoke-test loops; interactive dev mode + SSH is a heavy-hammer last resort. If you solve an error that wasn't in the known-errors list, suggest the user PR it back to this skill so future runs benefit. --- ## Reference index | When to read | File | |---|---| | **How ZeroGPU works** + correct patterns (decorator, sizing, pickle, generators, real-time, AoTI) | [`references/zerogpu.md`](references/zerogpu.md) | | **Iterate + debug**: logs, rung ladder, smoke testing (and dev mode + SSH as a last resort) | [`references/debugging.md`](references/debugging.md) | | **Error-string lookup** — the single place for all error symptoms (Spaces, ZeroGPU, Gradio, deps) | [`references/known-errors.md`](references/known-errors.md) | | Pinning deps, picking wheels, torch-family alignment | [`references/requirements.md`](references/requirements.md) | | `gr.Examples` (add when it makes sense), themes, custom HTML components, `gr.Server`, MCP server (`mcp_server=True`) | [`references/gradio.md`](references/gradio.md) | | Persistent storage, public bucket URLs | [`references/buckets.md`](references/buckets.md) | | Community grant requests (hardware the user can't pay for) | [`references/grants.md`](references/grants.md) | | Provider proxy (zero-VRAM big LLM via Cerebras / Fireworks / Together / etc.) | [`references/inference-providers.md`](references/inference-providers.md) | | **3D Spaces: generation, CUDA extensions, output formats, and model recipes (incl. gaussian splatting)** | [`references/3d-generation.md`](references/3d-generation.md) | <!-- chapter:end slug=huggingface-spaces --> --- <!-- chapter:begin slug=huggingface-tool-builder position=20 --> ## 20. huggingface-tool-builder - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-tool-builder/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-tool-builder.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `references/baseline_hf_api.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/baseline_hf_api.py - `references/baseline_hf_api.sh` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/baseline_hf_api.sh - `references/baseline_hf_api.tsx` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/baseline_hf_api.tsx - `references/find_models_by_paper.sh` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/find_models_by_paper.sh - `references/hf_enrich_models.sh` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/hf_enrich_models.sh - `references/hf_model_card_frontmatter.sh` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/hf_model_card_frontmatter.sh - `references/hf_model_papers_auth.sh` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-tool-builder/references/hf_model_papers_auth.sh <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-tool-builder description: Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. This is especially useful when chaining or combining API calls or the task will be repeated/automated. This Skill creates a reusable script to fetch, enrich or process data. --- # Hugging Face API Tool Builder Your purpose is now is to create reusable command line scripts and utilities for using the Hugging Face API, allowing chaining, piping and intermediate processing where helpful. You can access the API directly, as well as use the `hf` command line tool. Model and Dataset cards can be accessed from repositories directly. ## Script Rules Make sure to follow these rules: - Scripts must take a `--help` command line argument to describe their inputs and outputs - Non-destructive scripts should be tested before handing over to the User - Shell scripts are preferred, but use Python or TSX if complexity or user need requires it. - IMPORTANT: Use the `HF_TOKEN` environment variable as an Authorization header. For example: `curl -H "Authorization: Bearer ${HF_TOKEN}" https://huggingface.co/api/`. This provides higher rate limits and appropriate authorization for data access. - Investigate the shape of the API results before commiting to a final design; make use of piping and chaining where composability would be an advantage - prefer simple solutions where possible. - Share usage examples once complete. Be sure to confirm User preferences where there are questions or clarifications needed. ## Sample Scripts Paths below are relative to this skill directory. Reference examples: - `references/hf_model_papers_auth.sh` — uses `HF_TOKEN` automatically and chains trending → model metadata → model card parsing with fallbacks; it demonstrates multi-step API usage plus auth hygiene for gated/private content. - `references/find_models_by_paper.sh` — optional `HF_TOKEN` usage via `--token`, consistent authenticated search, and a retry path when arXiv-prefixed searches are too narrow; it shows resilient query strategy and clear user-facing help. - `references/hf_model_card_frontmatter.sh` — uses the `hf` CLI to download model cards, extracts YAML frontmatter, and emits NDJSON summaries (license, pipeline tag, tags, gated prompt flag) for easy filtering. Baseline examples (ultra-simple, minimal logic, raw JSON output with `HF_TOKEN` header): - `references/baseline_hf_api.sh` — bash - `references/baseline_hf_api.py` — python - `references/baseline_hf_api.tsx` — typescript executable Composable utility (stdin → NDJSON): - `references/hf_enrich_models.sh` — reads model IDs from stdin, fetches metadata per ID, emits one JSON object per line for streaming pipelines. Composability through piping (shell-friendly JSON output): - `references/baseline_hf_api.sh 25 | jq -r '.[].id' | references/hf_enrich_models.sh | jq -s 'sort_by(.downloads) | reverse | .[:10]'` - `references/baseline_hf_api.sh 50 | jq '[.[] | {id, downloads}] | sort_by(.downloads) | reverse | .[:10]'` - `printf '%s\n' openai/gpt-oss-120b meta-llama/Meta-Llama-3.1-8B | references/hf_model_card_frontmatter.sh | jq -s 'map({id, license, has_extra_gated_prompt})'` ## High Level Endpoints The following are the main API endpoints available at `https://huggingface.co` ``` /api/datasets /api/models /api/spaces /api/collections /api/daily_papers /api/notifications /api/settings /api/whoami-v2 /api/trending /oauth/userinfo ``` ## Accessing the API The API is documented with the OpenAPI standard at `https://huggingface.co/.well-known/openapi.json`. **IMPORTANT:** DO NOT ATTEMPT to read `https://huggingface.co/.well-known/openapi.json` directly as it is too large to process. **IMPORTANT** Use `jq` to query and extract relevant parts. For example, Command to Get All 160 Endpoints ```bash curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths | keys | sort' ``` Model Search Endpoint Details ```bash curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths["/api/models"]' ``` You can also query endpoints to see the shape of the data. When doing so constrain results to low numbers to make them easy to process, yet representative. ## Using the HF command line tool The `hf` command line tool gives you further access to Hugging Face repository content and infrastructure. ```bash ❯ hf --help Usage: hf [OPTIONS] COMMAND [ARGS]... Hugging Face Hub CLI Options: --help Show this message and exit. Commands: auth Manage authentication (login, logout, etc.). buckets Commands to interact with buckets. cache Manage local cache directory. collections Interact with collections on the Hub. datasets Interact with datasets on the Hub. discussions Manage discussions and pull requests on the Hub. download Download files from the Hub. endpoints Manage Hugging Face Inference Endpoints. env Print information about the environment. extensions Manage hf CLI extensions. jobs Run and manage Jobs on the Hub. models Interact with models on the Hub. papers Interact with papers on the Hub. repos Manage repos on the Hub. skills Manage skills for AI assistants. spaces Interact with spaces on the Hub. sync Sync files between local directory and a bucket. upload Upload a file or a folder to the Hub. upload-large-folder Upload a large folder to the Hub. version Print information about the hf version. webhooks Manage webhooks on the Hub. ``` The `hf` CLI command has replaced the now deprecated `huggingface-cli` command. <!-- chapter:end slug=huggingface-tool-builder --> --- <!-- chapter:begin slug=huggingface-trackio position=21 --> ## 21. huggingface-trackio - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-trackio/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-trackio/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-trackio.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (3), referenced from this skill's directory: - `references/alerts.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-trackio/references/alerts.md - `references/logging_metrics.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-trackio/references/logging_metrics.md - `references/retrieving_metrics.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-trackio/references/retrieving_metrics.md <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-trackio description: Track and visualize ML training experiments with Trackio. Use when logging metrics during training (Python API), firing alerts for training diagnostics, or retrieving/analyzing logged metrics (CLI). Supports real-time dashboard visualization, alerts with webhooks, HF Space syncing, and JSON output for automation. --- # Trackio - Experiment Tracking for ML Training Trackio is an experiment tracking library for logging and visualizing ML training metrics. It syncs to Hugging Face Spaces for real-time monitoring dashboards. ## Three Interfaces | Task | Interface | Reference | |------|-----------|-----------| | **Logging metrics** during training | Python API | [references/logging_metrics.md](references/logging_metrics.md) | | **Firing alerts** for training diagnostics | Python API | [references/alerts.md](references/alerts.md) | | **Retrieving metrics & alerts** after/during training | CLI | [references/retrieving_metrics.md](references/retrieving_metrics.md) | ## When to Use Each ### Python API → Logging Use `import trackio` in your training scripts to log metrics: - Initialize tracking with `trackio.init()` - Log metrics with `trackio.log()` or use TRL's `report_to="trackio"` - Finalize with `trackio.finish()` **Key concept**: For remote/cloud training, pass `space_id` — metrics sync to a Space dashboard so they persist after the instance terminates. Auto-created Spaces are **public by default** — pass `private=True` if the metrics should not be public. → See [references/logging_metrics.md](references/logging_metrics.md) for setup, TRL integration, and configuration options. ### Python API → Alerts Insert `trackio.alert()` calls in training code to flag important events — like inserting print statements for debugging, but structured and queryable: - `trackio.alert(title="...", level=trackio.AlertLevel.WARN)` — fire an alert - Three severity levels: `INFO`, `WARN`, `ERROR` - Alerts are printed to terminal, stored in the database, shown in the dashboard, and optionally sent to webhooks (Slack/Discord) **Key concept for LLM agents**: Alerts are the primary mechanism for autonomous experiment iteration. An agent should insert alerts into training code for diagnostic conditions (loss spikes, NaN gradients, low accuracy, training stalls). Since alerts are printed to the terminal, an agent that is watching the training script's output will see them automatically. For background or detached runs, the agent can poll via CLI instead. → See [references/alerts.md](references/alerts.md) for the full alerts API, webhook setup, and autonomous agent workflows. ### CLI → Retrieving Use the `trackio` command to query logged metrics and alerts: - `trackio list projects/runs/metrics` — discover what's available - `trackio get project/run/metric` — retrieve summaries and values - `trackio list alerts --project <name> --json` — retrieve alerts - `trackio show` — launch the dashboard - `trackio sync` — sync to HF Space **Key concept**: Add `--json` for programmatic output suitable for automation and LLM agents. → See [references/retrieving_metrics.md](references/retrieving_metrics.md) for all commands, workflows, and JSON output formats. ## Minimal Logging Setup ```python import trackio # Spaces are PUBLIC by default (good for shareable dashboards); # pass private=True if the metrics should not be public trackio.init(project="my-project", space_id="username/trackio", private=True) trackio.log({"loss": 0.1, "accuracy": 0.9}) trackio.log({"loss": 0.09, "accuracy": 0.91}) trackio.finish() ``` ### Minimal Retrieval ```bash trackio list projects --json trackio get metric --project my-project --run my-run --metric loss --json ``` ## Autonomous ML Experiment Workflow When running experiments autonomously as an LLM agent, the recommended workflow is: 1. **Set up training with alerts** — insert `trackio.alert()` calls for diagnostic conditions 2. **Launch training** — run the script in the background 3. **Poll for alerts** — use `trackio list alerts --project <name> --json --since <timestamp>` to check for new alerts 4. **Read metrics** — use `trackio get metric ...` to inspect specific values 5. **Iterate** — based on alerts and metrics, stop the run, adjust hyperparameters, and launch a new run ```python import trackio trackio.init(project="my-project", config={"lr": 1e-4}) for step in range(num_steps): loss = train_step() trackio.log({"loss": loss, "step": step}) if step > 100 and loss > 5.0: trackio.alert( title="Loss divergence", text=f"Loss {loss:.4f} still high after {step} steps", level=trackio.AlertLevel.ERROR, ) if step > 0 and abs(loss) < 1e-8: trackio.alert( title="Vanishing loss", text="Loss near zero — possible gradient collapse", level=trackio.AlertLevel.WARN, ) trackio.finish() ``` Then poll from a separate terminal/process: ```bash trackio list alerts --project my-project --json --since "2025-01-01T00:00:00" ``` <!-- chapter:end slug=huggingface-trackio --> --- <!-- chapter:begin slug=huggingface-vision-trainer position=22 --> ## 22. huggingface-vision-trainer - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-vision-trainer/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-vision-trainer.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (11), referenced from this skill's directory: - `references/finetune_sam2_trainer.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/finetune_sam2_trainer.md - `references/hub_saving.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/hub_saving.md - `references/image_classification_training_notebook.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/image_classification_training_notebook.md - `references/object_detection_training_notebook.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/object_detection_training_notebook.md - `references/reliability_principles.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/reliability_principles.md - `references/timm_trainer.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/references/timm_trainer.md - `scripts/dataset_inspector.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/scripts/dataset_inspector.py - `scripts/estimate_cost.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/scripts/estimate_cost.py - `scripts/image_classification_training.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/scripts/image_classification_training.py - `scripts/object_detection_training.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/scripts/object_detection_training.py - `scripts/sam_segmentation_training.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-vision-trainer/scripts/sam_segmentation_training.py <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-vision-trainer description: Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3 — plus any Transformers classifier), and SAM/SAM2 segmentation using Hugging Face Transformers on Hugging Face Jobs cloud GPUs. Covers COCO-format dataset preparation, Albumentations augmentation, mAP/mAR evaluation, accuracy metrics, SAM segmentation with bbox/point prompts, DiceCE loss, hardware selection, cost estimation, Trackio monitoring, and Hub persistence. Use when users mention training object detection, image classification, SAM, SAM2, segmentation, image matting, DETR, D-FINE, RT-DETR, ViT, timm, MobileNet, ResNet, bounding box models, or fine-tuning vision models on Hugging Face Jobs. --- # Vision Model Training on Hugging Face Jobs Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required—results are automatically saved to the Hugging Face Hub. ## When to Use This Skill Use this skill when users want to: - Fine-tune object detection models (D-FINE, RT-DETR v2, DETR, YOLOS) on cloud GPUs or local - Fine-tune image classification models (timm: MobileNetV3, MobileViT, ResNet, ViT/DINOv3, or any Transformers classifier) on cloud GPUs or local - Fine-tune SAM or SAM2 models for segmentation / image matting using bbox or point prompts - Train bounding-box detectors on custom datasets - Train image classifiers on custom datasets - Train segmentation models on custom mask datasets with prompts - Run vision training jobs on Hugging Face Jobs infrastructure - Ensure trained vision models are permanently saved to the Hub ## Related Skills - **`hugging-face-jobs`** — General HF Jobs infrastructure: token authentication, hardware flavors, timeout management, cost estimation, secrets, environment variables, scheduled jobs, and result persistence. **Refer to the Jobs skill for any non-training-specific Jobs questions** (e.g., "how do secrets work?", "what hardware is available?", "how do I pass tokens?"). - **`hugging-face-model-trainer`** — TRL-based language model training (SFT, DPO, GRPO). Use that skill for text/language model fine-tuning. ## Local Script Execution Helper scripts use PEP 723 inline dependencies. Run them with `uv run`: ```bash uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train uv run scripts/estimate_cost.py --help ``` ## Prerequisites Checklist Before starting any training job, verify: ### Account & Authentication - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) - Authenticated login: Check with `hf_whoami()` (tool) or `hf auth whoami` (terminal) - Token has **write** permissions - **MUST pass token in job secrets** — see directive #3 below for syntax (MCP tool vs Python API) ### Dataset Requirements — Object Detection - Dataset must exist on Hub - Annotations must use the `objects` column with `bbox`, `category` (and optionally `area`) sub-fields - Bboxes can be in **xywh (COCO)** or **xyxy (Pascal VOC)** format — auto-detected and converted - Categories can be **integers or strings** — strings are auto-remapped to integer IDs - `image_id` column is **optional** — generated automatically if missing - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Dataset Requirements — Image Classification - Dataset must exist on Hub - Must have an **`image` column** (PIL images) and a **`label` column** (integer class IDs or strings) - The label column can be `ClassLabel` type (with names) or plain integers/strings — strings are auto-remapped - Common column names auto-detected: `label`, `labels`, `class`, `fine_label` - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Dataset Requirements — SAM/SAM2 Segmentation - Dataset must exist on Hub - Must have an **`image` column** (PIL images) and a **`mask` column** (binary ground-truth segmentation mask) - Must have a **prompt** — either: - A **`prompt` column** with JSON containing `{"bbox": [x0,y0,x1,y1]}` or `{"point": [x,y]}` - OR a dedicated **`bbox`** column with `[x0,y0,x1,y1]` values - OR a dedicated **`point`** column with `[x,y]` or `[[x,y],...]` values - Bboxes should be in **xyxy** format (absolute pixel coordinates) - Example dataset: `merve/MicroMat-mini` (image matting with bbox prompts) - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Critical Settings - **Timeout must exceed expected training time** — Default 30min is TOO SHORT. See directive #6 for recommended values. - **Hub push must be enabled** — `push_to_hub=True`, `hub_model_id="username/model-name"`, token in `secrets` ## Dataset Validation **Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.** **ALWAYS validate for** unknown/custom datasets or any dataset you haven't trained with before. **Skip for** `cppe-5` (the default in the training script). ### Running the Inspector **Option 1: Via HF Jobs (recommended — avoids local SSL/dependency issues):** ```python hf_jobs("uv", { "script": "path/to/dataset_inspector.py", "script_args": ["--dataset", "username/dataset-name", "--split", "train"] }) ``` **Option 2: Locally:** ```bash uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train ``` **Option 3: Via `HfApi().run_uv_job()` (if hf_jobs MCP unavailable):** ```python from huggingface_hub import HfApi api = HfApi() api.run_uv_job( script="scripts/dataset_inspector.py", script_args=["--dataset", "username/dataset-name", "--split", "train"], flavor="cpu-basic", timeout=300, ) ``` ### Reading Results - **`✓ READY`** — Dataset is compatible, use directly - **`✗ NEEDS FORMATTING`** — Needs preprocessing (mapping code provided in output) ## Automatic Bbox Preprocessing The object detection training script (`scripts/object_detection_training.py`) automatically handles bbox format detection (xyxy→xywh conversion), bbox sanitization, `image_id` generation, string category→integer remapping, and dataset truncation. **No manual preprocessing needed** — just ensure the dataset has `objects.bbox` and `objects.category` columns. ## Training workflow Copy this checklist and track progress: ``` Training Progress: - [ ] Step 1: Verify prerequisites (account, token, dataset) - [ ] Step 2: Validate dataset format (run dataset_inspector.py) - [ ] Step 3: Ask user about dataset size and validation split - [ ] Step 4: Prepare training script (OD: scripts/object_detection_training.py, IC: scripts/image_classification_training.py, SAM: scripts/sam_segmentation_training.py) - [ ] Step 5: Save script locally, submit job, and report details ``` **Step 1: Verify prerequisites** Follow the Prerequisites Checklist above. **Step 2: Validate dataset** Run the dataset inspector BEFORE spending GPU time. See "Dataset Validation" section above. **Step 3: Ask user preferences** ALWAYS use the AskUserQuestion tool with option-style format: ```python AskUserQuestion({ "questions": [ { "question": "Do you want to run a quick test with a subset of the data first?", "header": "Dataset Size", "options": [ {"label": "Quick test run (10% of data)", "description": "Faster, cheaper (~30-60 min, ~$2-5) to validate setup"}, {"label": "Full dataset (Recommended)", "description": "Complete training for best model quality"} ], "multiSelect": false }, { "question": "Do you want to create a validation split from the training data?", "header": "Split data", "options": [ {"label": "Yes (Recommended)", "description": "Automatically split 15% of training data for validation"}, {"label": "No", "description": "Use existing validation split from dataset"} ], "multiSelect": false }, { "question": "Which GPU hardware do you want to use?", "header": "Hardware Flavor", "options": [ {"label": "t4-small ($0.40/hr)", "description": "1x T4, 16 GB VRAM — sufficient for all OD models under 100M params"}, {"label": "l4x1 ($0.80/hr)", "description": "1x L4, 24 GB VRAM — more headroom for large images or batch sizes"}, {"label": "a10g-large ($1.50/hr)", "description": "1x A10G, 24 GB VRAM — faster training, more CPU/RAM"}, {"label": "a100-large ($2.50/hr)", "description": "1x A100, 80 GB VRAM — fastest, for very large datasets or image sizes"} ], "multiSelect": false } ] }) ``` **Step 4: Prepare training script** For object detection, use [scripts/object_detection_training.py](scripts/object_detection_training.py) as the production-ready template. For image classification, use [scripts/image_classification_training.py](scripts/image_classification_training.py). For SAM/SAM2 segmentation, use [scripts/sam_segmentation_training.py](scripts/sam_segmentation_training.py). All scripts use `HfArgumentParser` — all configuration is passed via CLI arguments in `script_args`, NOT by editing Python variables. For timm model details, see [references/timm_trainer.md](references/timm_trainer.md). For SAM2 training details, see [references/finetune_sam2_trainer.md](references/finetune_sam2_trainer.md). **Step 5: Save script, submit job, and report** 1. **Save the script locally** to `submitted_jobs/` in the workspace root (create if needed) with a descriptive name like `training_<dataset>_<YYYYMMDD_HHMMSS>.py`. Tell the user the path. 2. **Submit** using `hf_jobs` MCP tool (preferred) or `HfApi().run_uv_job()` — see directive #1 for both methods. Pass all config via `script_args`. 3. **Report** the job ID (from `.id` attribute), monitoring URL, Trackio dashboard (`https://huggingface.co/spaces/{username}/trackio`), expected time, and estimated cost. 4. **Wait for user** to request status checks — don't poll automatically. Training jobs run asynchronously and can take hours. ## Critical directives These rules prevent common failures. Follow them exactly. ### 1. Job submission: `hf_jobs` MCP tool vs Python API **`hf_jobs()` is an MCP tool, NOT a Python function.** Do NOT try to import it from `huggingface_hub`. Call it as a tool: ``` hf_jobs("uv", {"script": training_script_content, "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"}}) ``` **If `hf_jobs` MCP tool is unavailable**, use the Python API directly: ```python from huggingface_hub import HfApi, get_token api = HfApi() job_info = api.run_uv_job( script="path/to/training_script.py", # file PATH, NOT content script_args=["--dataset_name", "cppe-5", ...], flavor="a10g-large", timeout=14400, # seconds (4 hours) env={"PYTHONUNBUFFERED": "1"}, secrets={"HF_TOKEN": get_token()}, # MUST use get_token(), NOT "$HF_TOKEN" ) print(f"Job ID: {job_info.id}") ``` **Critical differences between the two methods:** | | `hf_jobs` MCP tool | `HfApi().run_uv_job()` | |---|---|---| | `script` param | Python code string or URL (NOT local paths) | File path to `.py` file (NOT content) | | Token in secrets | `"$HF_TOKEN"` (auto-replaced) | `get_token()` (actual token value) | | Timeout format | String (`"4h"`) | Seconds (`14400`) | **Rules for both methods:** - The training script MUST include PEP 723 inline metadata with dependencies - Do NOT use `image` or `command` parameters (those belong to `run_job()`, not `run_uv_job()`) ### 2. Authentication via job secrets + explicit hub_token injection **Job config** MUST include the token in secrets — syntax depends on submission method (see table above). **Training script requirement:** The Transformers `Trainer` calls `create_repo(token=self.args.hub_token)` during `__init__()` when `push_to_hub=True`. The training script MUST inject `HF_TOKEN` into `training_args.hub_token` AFTER parsing args but BEFORE creating the `Trainer`. The template `scripts/object_detection_training.py` already includes this: ```python hf_token = os.environ.get("HF_TOKEN") if training_args.push_to_hub and not training_args.hub_token: if hf_token: training_args.hub_token = hf_token ``` If you write a custom script, you MUST include this token injection before the `Trainer(...)` call. - Do NOT call `login()` in custom scripts unless replicating the full pattern from `scripts/object_detection_training.py` - Do NOT rely on implicit token resolution (`hub_token=None`) — unreliable in Jobs - See the `hugging-face-jobs` skill → *Token Usage Guide* for full details ### 3. JobInfo attribute Access the job identifier using `.id` (NOT `.job_id` or `.name` — these don't exist): ```python job_info = api.run_uv_job(...) # or hf_jobs("uv", {...}) job_id = job_info.id # Correct -- returns string like "687fb701029421ae5549d998" ``` ### 4. Required training flags and HfArgumentParser boolean syntax `scripts/object_detection_training.py` uses `HfArgumentParser` — all config is passed via `script_args`. Boolean arguments have two syntaxes: - **`bool` fields** (e.g., `push_to_hub`, `do_train`): Use as bare flags (`--push_to_hub`) or negate with `--no_` prefix (`--no_remove_unused_columns`) - **`Optional[bool]` fields** (e.g., `greater_is_better`): MUST pass explicit value (`--greater_is_better True`). Bare `--greater_is_better` causes `error: expected one argument` Required flags for object detection: ``` --no_remove_unused_columns # MUST: preserves image column for pixel_values --no_eval_do_concat_batches # MUST: images have different numbers of target boxes --push_to_hub # MUST: environment is ephemeral --hub_model_id username/model-name --metric_for_best_model eval_map --greater_is_better True # MUST pass "True" explicitly (Optional[bool]) --do_train --do_eval ``` Required flags for image classification: ``` --no_remove_unused_columns # MUST: preserves image column for pixel_values --push_to_hub # MUST: environment is ephemeral --hub_model_id username/model-name --metric_for_best_model eval_accuracy --greater_is_better True # MUST pass "True" explicitly (Optional[bool]) --do_train --do_eval ``` Required flags for SAM/SAM2 segmentation: ``` --remove_unused_columns False # MUST: preserves input_boxes/input_points --push_to_hub # MUST: environment is ephemeral --hub_model_id username/model-name --do_train --prompt_type bbox # or "point" --dataloader_pin_memory False # MUST: avoids pin_memory issues with custom collator ``` ### 5. Timeout management Default 30 min is TOO SHORT for object detection. Set minimum 2-4 hours. Add 30% buffer for model loading, preprocessing, and Hub push. | Scenario | Timeout | |----------|---------| | Quick test (100-200 images, 5-10 epochs) | 1h | | Development (500-1K images, 15-20 epochs) | 2-3h | | Production (1K-5K images, 30 epochs) | 4-6h | | Large dataset (5K+ images) | 6-12h | ### 6. Trackio monitoring Trackio is **always enabled** in the object detection training script — it calls `trackio.init()` and `trackio.finish()` automatically. No need to pass `--report_to trackio`. The project name is taken from `--output_dir` and the run name from `--run_name`. For image classification, pass `--report_to trackio` in `TrainingArguments`. Dashboard at: `https://huggingface.co/spaces/{username}/trackio` ## Model & hardware selection ### Recommended object detection models | Model | Params | Use case | |-------|--------|----------| | `ustc-community/dfine-small-coco` | 10.4M | Best starting point — fast, cheap, SOTA quality | | `PekingU/rtdetr_v2_r18vd` | 20.2M | Lightweight real-time detector | | `ustc-community/dfine-large-coco` | 31.4M | Higher accuracy, still efficient | | `PekingU/rtdetr_v2_r50vd` | 43M | Strong real-time baseline | | `ustc-community/dfine-xlarge-obj365` | 63.5M | Best accuracy (pretrained on Objects365) | | `PekingU/rtdetr_v2_r101vd` | 76M | Largest RT-DETR v2 variant | Start with `ustc-community/dfine-small-coco` for fast iteration. Move to D-FINE Large or RT-DETR v2 R50 for better accuracy. ### Recommended image classification models All `timm/` models work out of the box via `AutoModelForImageClassification` (loaded as `TimmWrapperForImageClassification`). See [references/timm_trainer.md](references/timm_trainer.md) for details. | Model | Params | Use case | |-------|--------|----------| | `timm/mobilenetv3_small_100.lamb_in1k` | 2.5M | Ultra-lightweight — mobile/edge, fastest training | | `timm/mobilevit_s.cvnets_in1k` | 5.6M | Mobile transformer — good accuracy/speed trade-off | | `timm/resnet50.a1_in1k` | 25.6M | Strong CNN baseline — reliable, well-studied | | `timm/vit_base_patch16_dinov3.lvd1689m` | 86.6M | Best accuracy — DINOv3 self-supervised ViT | Start with `timm/mobilenetv3_small_100.lamb_in1k` for fast iteration. Move to `timm/resnet50.a1_in1k` or `timm/vit_base_patch16_dinov3.lvd1689m` for better accuracy. ### Recommended SAM/SAM2 segmentation models | Model | Params | Use case | |-------|--------|----------| | `facebook/sam2.1-hiera-tiny` | 38.9M | Fastest SAM2 — good for quick experiments | | `facebook/sam2.1-hiera-small` | 46.0M | Best starting point — good quality/speed balance | | `facebook/sam2.1-hiera-base-plus` | 80.8M | Higher capacity for complex segmentation | | `facebook/sam2.1-hiera-large` | 224.4M | Best SAM2 accuracy — requires more VRAM | | `facebook/sam-vit-base` | 93.7M | Original SAM — ViT-B backbone | | `facebook/sam-vit-large` | 312.3M | Original SAM — ViT-L backbone | | `facebook/sam-vit-huge` | 641.1M | Original SAM — ViT-H, best SAM v1 accuracy | Start with `facebook/sam2.1-hiera-small` for fast iteration. SAM2 models are generally more efficient than SAM v1 at similar quality. Only the mask decoder is trained by default (vision and prompt encoders are frozen). ### Hardware recommendation All recommended OD and IC models are under 100M params — **`t4-small` (16 GB VRAM, $0.40/hr) is sufficient for all of them.** Image classification models are generally smaller and faster than object detection models — `t4-small` handles even ViT-Base comfortably. For SAM2 models up to `hiera-base-plus`, `t4-small` is sufficient since only the mask decoder is trained. For `sam2.1-hiera-large` or SAM v1 models, use `l4x1` or `a10g-large`. Only upgrade if you hit OOM from large batch sizes — reduce batch size first before switching hardware. Common upgrade path: `t4-small` → `l4x1` ($0.80/hr, 24 GB) → `a10g-large` ($1.50/hr, 24 GB). For full hardware flavor list: refer to the `hugging-face-jobs` skill. For cost estimation: run `scripts/estimate_cost.py`. ## Quick start — Object Detection The `script_args` below are the same for both submission methods. See directive #1 for the critical differences between them. ```python OD_SCRIPT_ARGS = [ "--model_name_or_path", "ustc-community/dfine-small-coco", "--dataset_name", "cppe-5", "--image_square_size", "640", "--output_dir", "dfine_finetuned", "--num_train_epochs", "30", "--per_device_train_batch_size", "8", "--learning_rate", "5e-5", "--eval_strategy", "epoch", "--save_strategy", "epoch", "--save_total_limit", "2", "--load_best_model_at_end", "--metric_for_best_model", "eval_map", "--greater_is_better", "True", "--no_remove_unused_columns", "--no_eval_do_concat_batches", "--push_to_hub", "--hub_model_id", "username/model-name", "--do_train", "--do_eval", ] ``` ```python from huggingface_hub import HfApi, get_token api = HfApi() job_info = api.run_uv_job( script="scripts/object_detection_training.py", script_args=OD_SCRIPT_ARGS, flavor="t4-small", timeout=14400, env={"PYTHONUNBUFFERED": "1"}, secrets={"HF_TOKEN": get_token()}, ) print(f"Job ID: {job_info.id}") ``` ### Key OD `script_args` - `--model_name_or_path` — recommended: `"ustc-community/dfine-small-coco"` (see model table above) - `--dataset_name` — the Hub dataset ID - `--image_square_size` — 480 (fast iteration) or 800 (better accuracy) - `--hub_model_id` — `"username/model-name"` for Hub persistence - `--num_train_epochs` — 30 typical for convergence - `--train_val_split` — fraction to split for validation (default 0.15), set if dataset lacks a validation split - `--max_train_samples` — truncate training set (useful for quick test runs, e.g. `"785"` for ~10% of a 7.8K dataset) - `--max_eval_samples` — truncate evaluation set ## Quick start — Image Classification ```python IC_SCRIPT_ARGS = [ "--model_name_or_path", "timm/mobilenetv3_small_100.lamb_in1k", "--dataset_name", "ethz/food101", "--output_dir", "food101_classifier", "--num_train_epochs", "5", "--per_device_train_batch_size", "32", "--per_device_eval_batch_size", "32", "--learning_rate", "5e-5", "--eval_strategy", "epoch", "--save_strategy", "epoch", "--save_total_limit", "2", "--load_best_model_at_end", "--metric_for_best_model", "eval_accuracy", "--greater_is_better", "True", "--no_remove_unused_columns", "--push_to_hub", "--hub_model_id", "username/food101-classifier", "--do_train", "--do_eval", ] ``` ```python from huggingface_hub import HfApi, get_token api = HfApi() job_info = api.run_uv_job( script="scripts/image_classification_training.py", script_args=IC_SCRIPT_ARGS, flavor="t4-small", timeout=7200, env={"PYTHONUNBUFFERED": "1"}, secrets={"HF_TOKEN": get_token()}, ) print(f"Job ID: {job_info.id}") ``` ### Key IC `script_args` - `--model_name_or_path` — any `timm/` model or Transformers classification model (see model table above) - `--dataset_name` — the Hub dataset ID - `--image_column_name` — column containing PIL images (default: `"image"`) - `--label_column_name` — column containing class labels (default: `"label"`) - `--hub_model_id` — `"username/model-name"` for Hub persistence - `--num_train_epochs` — 3-5 typical for classification (fewer than OD) - `--per_device_train_batch_size` — 16-64 (classification models use less memory than OD) - `--train_val_split` — fraction to split for validation (default 0.15), set if dataset lacks a validation split - `--max_train_samples` / `--max_eval_samples` — truncate for quick tests ## Quick start — SAM/SAM2 Segmentation ```python SAM_SCRIPT_ARGS = [ "--model_name_or_path", "facebook/sam2.1-hiera-small", "--dataset_name", "merve/MicroMat-mini", "--prompt_type", "bbox", "--prompt_column_name", "prompt", "--output_dir", "sam2-finetuned", "--num_train_epochs", "30", "--per_device_train_batch_size", "4", "--learning_rate", "1e-5", "--logging_steps", "1", "--save_strategy", "epoch", "--save_total_limit", "2", "--remove_unused_columns", "False", "--dataloader_pin_memory", "False", "--push_to_hub", "--hub_model_id", "username/sam2-finetuned", "--do_train", "--report_to", "trackio", ] ``` ```python from huggingface_hub import HfApi, get_token api = HfApi() job_info = api.run_uv_job( script="scripts/sam_segmentation_training.py", script_args=SAM_SCRIPT_ARGS, flavor="t4-small", timeout=7200, env={"PYTHONUNBUFFERED": "1"}, secrets={"HF_TOKEN": get_token()}, ) print(f"Job ID: {job_info.id}") ``` ### Key SAM `script_args` - `--model_name_or_path` — SAM or SAM2 model (see model table above); auto-detects SAM vs SAM2 - `--dataset_name` — the Hub dataset ID (e.g., `"merve/MicroMat-mini"`) - `--prompt_type` — `"bbox"` or `"point"` — type of prompt in the dataset - `--prompt_column_name` — column with JSON-encoded prompts (default: `"prompt"`) - `--bbox_column_name` — dedicated bbox column (alternative to JSON prompt column) - `--point_column_name` — dedicated point column (alternative to JSON prompt column) - `--mask_column_name` — column with ground-truth masks (default: `"mask"`) - `--hub_model_id` — `"username/model-name"` for Hub persistence - `--num_train_epochs` — 20-30 typical for SAM fine-tuning - `--per_device_train_batch_size` — 2-4 (SAM models use significant memory) - `--freeze_vision_encoder` / `--freeze_prompt_encoder` — freeze encoder weights (default: both frozen, only mask decoder trains) - `--train_val_split` — fraction to split for validation (default 0.1) ## Checking job status **MCP tool (if available):** ``` hf_jobs("ps") # List all jobs hf_jobs("logs", {"job_id": "your-job-id"}) # View logs hf_jobs("inspect", {"job_id": "your-job-id"}) # Job details ``` **Python API fallback:** ```python from huggingface_hub import HfApi api = HfApi() api.list_jobs() # List all jobs api.get_job_logs(job_id="your-job-id") # View logs api.get_job(job_id="your-job-id") # Job details ``` ## Common failure modes ### OOM (CUDA out of memory) Reduce `per_device_train_batch_size` (try 4, then 2), reduce `IMAGE_SIZE`, or upgrade hardware. ### Dataset format errors Run `scripts/dataset_inspector.py` first. The training script auto-detects xyxy vs xywh, converts string categories to integer IDs, and adds `image_id` if missing. Ensure `objects.bbox` contains 4-value coordinate lists in absolute pixels and `objects.category` contains either integer IDs or string labels. ### Hub push failures (401) Verify: (1) job secrets include token (see directive #2), (2) script sets `training_args.hub_token` BEFORE creating the `Trainer`, (3) `push_to_hub=True` is set, (4) correct `hub_model_id`, (5) token has write permissions. ### Job timeout Increase timeout (see directive #5 table), reduce epochs/dataset, or use checkpoint strategy with `hub_strategy="every_save"`. ### KeyError: 'test' (missing test split) The object detection training script handles this gracefully — it falls back to the `validation` split. Ensure you're using the latest `scripts/object_detection_training.py`. ### Single-class dataset: "iteration over a 0-d tensor" `torchmetrics.MeanAveragePrecision` returns scalar (0-d) tensors for per-class metrics when there's only one class. The template `scripts/object_detection_training.py` handles this by calling `.unsqueeze(0)` on these tensors. Ensure you're using the latest template. ### Poor detection performance (mAP < 0.15) Increase epochs (30-50), ensure 500+ images, check per-class mAP for imbalanced classes, try different learning rates (1e-5 to 1e-4), increase image size. For comprehensive troubleshooting: see [references/reliability_principles.md](references/reliability_principles.md) ## Reference files - [scripts/object_detection_training.py](scripts/object_detection_training.py) — Production-ready object detection training script - [scripts/image_classification_training.py](scripts/image_classification_training.py) — Production-ready image classification training script (supports timm models) - [scripts/sam_segmentation_training.py](scripts/sam_segmentation_training.py) — Production-ready SAM/SAM2 segmentation training script (bbox & point prompts) - [scripts/dataset_inspector.py](scripts/dataset_inspector.py) — Validate dataset format for OD, classification, and SAM segmentation - [scripts/estimate_cost.py](scripts/estimate_cost.py) — Estimate training costs for any vision model (includes SAM/SAM2) - [references/object_detection_training_notebook.md](references/object_detection_training_notebook.md) — Object detection training workflow, augmentation strategies, and training patterns - [references/image_classification_training_notebook.md](references/image_classification_training_notebook.md) — Image classification training workflow with ViT, preprocessing, and evaluation - [references/finetune_sam2_trainer.md](references/finetune_sam2_trainer.md) — SAM2 fine-tuning walkthrough with MicroMat dataset, DiceCE loss, and Trainer integration - [references/timm_trainer.md](references/timm_trainer.md) — Using timm models with HF Trainer (TimmWrapper, transforms, full example) - [references/hub_saving.md](references/hub_saving.md) — Detailed Hub persistence guide and verification checklist - [references/reliability_principles.md](references/reliability_principles.md) — Failure prevention principles from production experience ## External links - [Transformers Object Detection Guide](https://huggingface.co/docs/transformers/tasks/object_detection) - [Transformers Image Classification Guide](https://huggingface.co/docs/transformers/tasks/image_classification) - [DETR Model Documentation](https://huggingface.co/docs/transformers/model_doc/detr) - [ViT Model Documentation](https://huggingface.co/docs/transformers/model_doc/vit) - [HF Jobs Guide](https://huggingface.co/docs/huggingface_hub/guides/jobs) — Main Jobs documentation - [HF Jobs Configuration](https://huggingface.co/docs/hub/en/jobs-configuration) — Hardware, secrets, timeouts, namespaces - [HF Jobs CLI Reference](https://huggingface.co/docs/huggingface_hub/guides/cli#hf-jobs) — Command line interface - [Object Detection Models](https://huggingface.co/models?pipeline_tag=object-detection) - [Image Classification Models](https://huggingface.co/models?pipeline_tag=image-classification) - [SAM2 Model Documentation](https://huggingface.co/docs/transformers/model_doc/sam2) - [SAM Model Documentation](https://huggingface.co/docs/transformers/model_doc/sam) - [Object Detection Datasets](https://huggingface.co/datasets?task_categories=task_categories:object-detection) - [Image Classification Datasets](https://huggingface.co/datasets?task_categories=task_categories:image-classification) <!-- chapter:end slug=huggingface-vision-trainer --> --- <!-- chapter:begin slug=huggingface-zerogpu position=23 --> ## 23. huggingface-zerogpu - **Source:** https://github.com/huggingface/skills/blob/main/skills/huggingface-zerogpu/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-zerogpu/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/huggingface-zerogpu.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `references/concurrency.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-zerogpu/references/concurrency.md - `references/cuda-and-deps.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-zerogpu/references/cuda-and-deps.md - `references/how-quota-works.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-zerogpu/references/how-quota-works.md - `references/how-zerogpu-works.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/huggingface-zerogpu/references/how-zerogpu-works.md <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: huggingface-zerogpu description: AI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. Use when writing or reviewing code that uses `@spaces.GPU`, configuring `python_version` or `requirements.txt` for a ZeroGPU Space, or handling ZeroGPU-specific code constraints — pickle-based process isolation, `gr.State` semantics across the worker boundary, no `torch.compile` (use AoTI instead), CUDA wheel-only builds (no `nvcc` at build or runtime), large vs xlarge sizing, and dynamic duration callables. Make sure to use this skill whenever the user mentions ZeroGPU, `@spaces.GPU`, or the `spaces` Python package, or hits ZeroGPU-specific code errors like `PicklingError` across the worker boundary, `illegal duration`, or `flash-attn` wheel-build failures — even when the user does not explicitly ask for ZeroGPU coding guidance. Trigger on `import spaces` or `@spaces.GPU` in code. --- # Hugging Face ZeroGPU Rules and patterns for ML demos on Hugging Face Spaces with **ZeroGPU** hardware. Covers `@spaces.GPU`, duration and quota tuning, process isolation, the CUDA availability model, concurrency safety, and CUDA build constraints. ## Scope This skill is for **Gradio SDK Spaces using ZeroGPU hardware**. Docker and Static Spaces cannot schedule onto ZeroGPU, and Streamlit apps now run as Docker Spaces — so this skill applies only to Gradio. For general Gradio coding (components, layouts, event listeners), see the `huggingface-gradio` skill in this repo. The authoritative ZeroGPU docs live at https://huggingface.co/docs/hub/spaces-zerogpu — refer to them for the current backing GPU, runtime version lists, and tier thresholds, all of which change over time. ## Reference Files | Reference | When to read | |-----------|--------------| | `references/concurrency.md` | Always read alongside SKILL.md when writing ZeroGPU code — handlers run in parallel by default | | `references/how-zerogpu-works.md` | When reasoning about cold-starts, worker reuse, why module-scope warmup does not carry to requests, or why returning CUDA tensors hangs | | `references/how-quota-works.md` | When choosing `duration` values, debugging `illegal duration` vs `quota exceeded` errors, or explaining why default 60s blocks short tasks | | `references/cuda-and-deps.md` | When installing CUDA-dependent packages (e.g. `flash-attn`), pinning torch side-cars, or reading wheel filename tags | ## Hardware ZeroGPU exposes two GPU sizes that map to a fraction of the backing card: | `size` | Slice of backing GPU | Quota cost | |--------|----------------------|------------| | `large` *(default)* | Half | 1x | | `xlarge` | Full | 2x | Default `large` gives half a physical GPU, so memory bandwidth and compute are significantly lower than the full card's specs. Use `xlarge` only when the workload genuinely needs the extra memory or compute. > **Backing GPU changes without notice.** ZeroGPU has already migrated across GPU generations several times; older write-ups may name A100 or H200, but those are outdated. For the current backing GPU and exact per-size VRAM, always check the [ZeroGPU docs](https://huggingface.co/docs/hub/spaces-zerogpu) before sizing workloads. ## Basic Pattern ```python import spaces import torch from transformers import pipeline pipe = pipeline("text-generation", model="...", device="cuda") @spaces.GPU def generate(prompt: str) -> str: return pipe(prompt, max_new_tokens=100)[0]["generated_text"] ``` Key rules: 1. **Instantiate models at module scope** and call `.to("cuda")` eagerly. ZeroGPU handles the actual device mapping transparently (see CUDA availability model below). 2. **Decorate GPU functions with `@spaces.GPU`**. The decorator is a no-op outside ZeroGPU, so it is safe to keep in all environments. 3. **Set `duration` to match the realistic worst-case workload** (default 60s). The platform pre-checks `requested duration` against the user's `remaining quota` — not against the actual run time — so a 10-second task left at the 60s default fails with `quota exceeded` as soon as the user's remaining quota drops below 60s. Smaller declared `duration` also ranks higher in the node-level queue. See "Duration and Quota" below. 4. **`torch.compile` is NOT supported.** Use PyTorch [ahead-of-time compilation (AoTI)](https://huggingface.co/blog/zerogpu-aoti) (torch 2.8+) instead. 5. **Use `size="xlarge"` sparingly.** It allocates the full backing GPU, but costs 2x quota and tends to queue longer. ```python @spaces.GPU(duration=120) def generate_image(prompt: str): return pipe(prompt).images[0] ``` ## CUDA Availability Model Real GPU access is **only** available inside `@spaces.GPU`-decorated functions. Outside those functions, the GPU is not attached to the process. However, `import spaces` **monkey-patches `torch`** so that: - `torch.cuda.is_available()` returns `True` globally. - `.to("cuda")` / `device="cuda"` calls at module scope succeed without error. This is intentional. Module-scope `model.to("cuda")` calls register tensors with the ZeroGPU backend, which writes them to a disk offload directory at a startup "pack" step and frees the corresponding RAM. When a `@spaces.GPU` call lands, a forked GPU worker process streams those weights from disk into VRAM via a pinned-memory pipeline. Warm workers (reused across requests on the same GPU slot) keep weights resident on the GPU and skip the disk → VRAM step. The user-facing rule: write `device="cuda"` at module scope and it works — see `references/how-zerogpu-works.md` for the full lifecycle. | Action | Where | Why | |--------|-------|-----| | `model.to("cuda")` / `pipe(..., device="cuda")` | **Module scope** | ZeroGPU registers the tensor and manages device migration | | Actual CUDA computation (inference, etc.) | **Inside `@spaces.GPU`** | Real GPU is only attached during the decorated call | | Branching on `torch.cuda.is_available()` | Avoid relying on it | Always returns `True` due to the monkey-patch | Do not run inference or CUDA kernels at module scope — the real GPU is not attached, so operations either silently run on CPU or fail. ### Device selection idiom still works The standard idiom remains correct under ZeroGPU: ```python device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = AutoModel.from_pretrained("...").to(device) ``` - **ZeroGPU** — `is_available()` is `True` (monkey-patched), so the model is registered for automatic device migration. - **Dedicated GPU Spaces / local GPU** — `is_available()` is genuinely `True`. - **CPU Spaces / local CPU** — resolves to `"cpu"`. Do not hardcode `device="cuda"` — it breaks on CPU-only environments. ### Eager loading is the right default Load models at module scope, not lazily on first request. The Space process starts before any user arrives, so cold-start cost is paid once. Lazy loading (`global model; if model is None: ...`, `@lru_cache` wrappers, factory functions instantiating on first call) just pushes that cost onto the first user. ## Local Development: Just Install `spaces` Do **not** wrap `import spaces` in `try/except` and redefine `spaces.GPU` as a no-op fallback for local runs. Off-ZeroGPU, the `spaces` package is already a true no-op: - Heavyweight behavior (CUDA monkey-patching, client init, startup hooks) is gated on the `SPACES_ZERO_GPU` env var, set only on ZeroGPU. - `@spaces.GPU` returns the undecorated function unchanged off-ZeroGPU. - Top-level `import spaces` performs only lightweight imports. The Gradio SDK base image installs `spaces` on every hardware tier. So even after duplicating a Space onto a dedicated GPU (T4, L4, A10G, etc.) or CPU basic, no code changes are needed — `import spaces` still succeeds and `@spaces.GPU` becomes a transparent passthrough. ### Anti-pattern ```python try: import spaces except ImportError: class spaces: # type: ignore @staticmethod def GPU(func=None, **kwargs): return func if func else (lambda f: f) ``` Problems: 1. The fallback must mimic every `@spaces.GPU` call shape — bare decorator, `duration=...`, `size=...`, generators, `aoti_*` helpers — and drifts as the `spaces` API grows. 2. It hides `spaces` from `requirements.txt`, even though the Space needs it at deploy time. 3. It solves a non-problem: the real package is already a no-op locally. ### Do this instead Add `spaces` to dependencies and import it unconditionally: ```python import spaces @spaces.GPU def generate(prompt: str) -> str: ... ``` ## Duration and Quota Three things happen when you declare `@spaces.GPU(duration=N)`: 1. **Tier-max check** — each visitor tier has a per-call `duration` cap. Declaring `duration` larger than the cap fails immediately with `ZeroGPU illegal duration`, regardless of remaining quota. (Tier numbers change over time — see the [ZeroGPU docs](https://huggingface.co/docs/hub/spaces-zerogpu).) 2. **Quota pre-check** — the platform compares `requested duration` against the user's `remaining quota`. If `remaining < requested`, the call fails with `ZeroGPU quota exceeded` — even if the actual work would have fit. The error message shows the explicit numbers, e.g. `"60s requested vs. 30s left"`. A 10-second task left at the default 60s therefore blocks the user once their remaining quota drops below 60s. 3. **Queue priority** — the queue is node-level (requests from all Spaces on the same node compete for GPU slots), and shorter declared `duration` ranks higher. All three favor declaring the smallest realistic `duration` — including for short tasks. Explicit `@spaces.GPU(duration=15)` on a 10-second task avoids premature `quota exceeded` rejections and ranks higher in the queue. > **`xlarge` doubles the request.** `requested = N * 2` when `size="xlarge"`, both for the tier-max check and the quota pre-check. So `@spaces.GPU(duration=60, size="xlarge")` is internally a 120s request. ### Dynamic duration for variable workloads For workloads whose runtime depends on inputs, pass a callable that estimates per request. A static high `duration` locks out low-tier users (whose tier cap may be smaller than the static value) and unnecessarily reserves quota for light inputs. ```python def estimate_duration(prompt, steps): return int(steps * 3.5) @spaces.GPU(duration=estimate_duration) def generate(prompt, steps): return pipe(prompt, num_inference_steps=steps).images[0] ``` For the full distinction between `illegal duration` vs `quota exceeded`, runs-per-day limits, the 24h quota window, and pay-as-you-go billing, see `references/how-quota-works.md`. ## Process Isolation and Pickle `@spaces.GPU`-decorated functions run in a **separate process** managed by the ZeroGPU scheduler. Arguments and return values cross the process boundary via **pickle serialization**. Consequences: - **Only picklable objects** can be passed in or returned. Open file handles, database connections, locks, lambdas, and closures over unpicklable state will raise `PicklingError`. - **Do NOT return CUDA tensors directly.** Unpickling a CUDA tensor in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks. Convert to CPU first: return `tensor.cpu()` or `tensor.cpu().numpy()`. - CPU tensors, numpy arrays, PIL Images, and plain Python objects work fine. - Large objects incur serialization overhead. Prefer lightweight returns (tensors, arrays, file paths, base64 strings) over complex object graphs. ### `gr.State` semantics across the boundary Because handlers run in a separate process, `gr.State` values are **pickled on every yield** — they are NOT shared by reference. - The generator receives a **copy** of the state (`id()` differs from the caller's). - In-place mutations inside the generator are **invisible** to other handlers until the mutated state is explicitly yielded back. - Yielding `gr.update()` for a `gr.State` slot **skips the update** — other handlers continue to see the pre-yield value. - Each yield that returns the state object creates a **new copy** via pickle. Practical guidance: - **Do NOT assume reference semantics for `gr.State`** on ZeroGPU. Code that mutates state in a generator and expects another handler to see those mutations will silently use stale data. - **Every yield including a `gr.State` value triggers a full pickle round-trip.** For large state (model sessions, frame buffers), minimize how often you yield it — ideally once at the end. Use `gr.update()` for the state slot on intermediate yields. - **CUDA tensors inside state must be moved to CPU before yielding** — same `torch.cuda._lazy_init()` issue as above. ## Concurrency Handlers run **concurrently by default** on ZeroGPU. This is not opt-in. Code that worked in single-user testing can silently corrupt or leak data in production. Three rules. Full treatment with examples in `references/concurrency.md`. 1. **No mutable global state.** Concurrent requests overwrite each other. 2. **No fixed file paths for outputs.** Concurrent requests clobber the same file. Use `tempfile` for unique paths. 3. **Read-only globals are safe.** Model objects, tokenizers, configs loaded once at startup and only read during requests are safe and encouraged. ## Call Granularity Each entry into a `@spaces.GPU` function carries non-trivial cost — pickle round-trip across the process boundary, worker warm-up, CUDA re-attach, and a fresh pass through the node-level queue. Calling a decorated function from inside a hot loop multiplies these costs and adds a new failure mode: a later iteration may fail to acquire a GPU slot, stalling the whole job mid-way. Decorate the outer function that owns the loop, not the per-iteration worker: ```python # Avoid — N GPU entries for N frames def process_video(frames): return [process_frame(f) for f in frames] @spaces.GPU(duration=...) def process_frame(frame): ... # Prefer — one GPU entry for the whole video @spaces.GPU(duration=...) def process_video(frames): return [process_frame(f) for f in frames] def process_frame(frame): ... ``` If the loop mixes heavy CPU work with GPU work, wrapping the whole loop charges that CPU time against the user's quota. When that cost is material, batching the GPU work so CPU pre/post-processing stays outside the decorator is a situational optimization — not the default. ## CUDA Build Constraints HF Spaces builds Docker images in a CPU-only environment. **On ZeroGPU, the build phase has no `nvcc`** because the base image is `python:3.13` (dedicated-GPU Spaces use `nvidia/cuda:*-devel-*` and have `nvcc` at build time). A CUDA-dependent package whose only distribution is sdist — e.g. bare `flash-attn` — therefore cannot be installed via `requirements.txt` on ZeroGPU. Only pre-built wheels work. ZeroGPU **runtime** does have `nvcc` available, mounted from a CUDA devel image at `/cuda-image` since 2025-07 (originally added for AoTI support). This is what makes `torch.export` / AoTI workflows possible inside `@spaces.GPU` calls. **Bottom line**: install every CUDA-dependent package from a pre-built wheel. If no wheel is available on PyPI, build one externally (e.g. host on HF Hub) and pin the URL. For `flash-attn`, the upstream releases page ships a fairly complete wheel matrix covering most Python × CUDA × torch combinations. For wheel-tag reading (cxx11 ABI, `cu12torch2.X`, `cp3XX`), torch-family side-car drift, and the kernels-community fallback, see `references/cuda-and-deps.md`. ## Example Caching `gr.Examples` behavior is environment-dependent. On ZeroGPU specifically: - `cache_examples` defaults to `True` (Spaces sets `GRADIO_CACHE_EXAMPLES=true`). - `cache_mode` defaults to `"lazy"` (Spaces sets `GRADIO_CACHE_MODE=lazy` only on ZeroGPU). ZeroGPU defaults to `lazy` because eager caching pre-runs every example at app startup, but ZeroGPU has **no GPU attached at startup** — only during request handling. Eager caching of GPU-bound examples would fail there. When `cache_examples=True`, the `run_on_click` / `run_examples_on_click` parameter is silently ignored. If your app relies on click-populates-only behavior, set `cache_examples=False` explicitly to preserve it. To reproduce ZeroGPU example-caching behavior locally: ```bash GRADIO_CACHE_EXAMPLES=true GRADIO_CACHE_MODE=lazy python app.py ``` ## Dependency Management ### `python_version` pin in README frontmatter Pinning `python_version` is **effectively required** for ZeroGPU. The runtime default is currently Python 3.10, so a local environment using 3.11+ will fail to install on the Space without an explicit pin. Pin to a ZeroGPU-supported version (3.12 is a reasonable default); the authoritative supported list lives in the [ZeroGPU docs](https://huggingface.co/docs/hub/spaces-zerogpu) — do not hardcode the full list, refer to the docs. ```yaml # README.md frontmatter python_version: "3.12" ``` Both `"3.12"` and `"3.12.12"` forms are accepted. ### Do not pin `spaces` in `requirements.txt` The Space platform pins its own `spaces` version. A conflicting pin in `requirements.txt` causes pip resolution to fail at build time. > **Rule**: Do not include `spaces` in `requirements.txt`. How to achieve this depends on your tooling: - **Hand-written `requirements.txt`**: simply omit `spaces`. - **uv** (`pyproject.toml`-managed): declare `spaces` in `pyproject.toml` so uv co-resolves transitive constraints (notably `psutil`, which `spaces` pins), then exclude it from the export: ```bash uv export --no-hashes --no-dev --no-emit-package spaces -o requirements.txt ``` Without `spaces` in `pyproject.toml`, uv cannot see its transitive constraints and may resolve incompatible versions at build time. - **pip-tools** (`pip-compile`) / **Poetry**: use the equivalent exclude mechanism. ### Pin `torch` to match wheel tags If you install a CUDA-dependent wheel via direct URL, the wheel filename encodes the `torch` major.minor it was built against (e.g. `cu12torch2.8`). Pin `torch==X.Y.Z` in `requirements.txt` to match — otherwise pip may resolve `torch` to a different version and the Space fails on first import. Details and the kernels-community alternative are in `references/cuda-and-deps.md`. <!-- chapter:end slug=huggingface-zerogpu --> --- <!-- chapter:begin slug=train-sentence-transformers position=24 --> ## 24. train-sentence-transformers - **Source:** https://github.com/huggingface/skills/blob/main/skills/train-sentence-transformers/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/train-sentence-transformers.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (27), referenced from this skill's directory: - `references/base_model_selection.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/base_model_selection.md - `references/dataset_formats.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/dataset_formats.md - `references/evaluators_cross_encoder.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/evaluators_cross_encoder.md - `references/evaluators_sentence_transformer.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/evaluators_sentence_transformer.md - `references/evaluators_sparse_encoder.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/evaluators_sparse_encoder.md - `references/hardware_guide.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/hardware_guide.md - `references/hf_jobs_execution.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/hf_jobs_execution.md - `references/losses_cross_encoder.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/losses_cross_encoder.md - `references/losses_sentence_transformer.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/losses_sentence_transformer.md - `references/losses_sparse_encoder.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/losses_sparse_encoder.md - `references/model_architectures.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/model_architectures.md - `references/prompts_and_instructions.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/prompts_and_instructions.md - `references/training_args.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/training_args.md - `references/troubleshooting.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/references/troubleshooting.md - `scripts/mine_hard_negatives.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/mine_hard_negatives.py - `scripts/train_cross_encoder_distillation_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_cross_encoder_distillation_example.py - `scripts/train_cross_encoder_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_cross_encoder_example.py - `scripts/train_cross_encoder_listwise_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_cross_encoder_listwise_example.py - `scripts/train_sentence_transformer_distillation_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_distillation_example.py - `scripts/train_sentence_transformer_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_example.py - `scripts/train_sentence_transformer_make_multilingual_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_make_multilingual_example.py - `scripts/train_sentence_transformer_matryoshka_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_matryoshka_example.py - `scripts/train_sentence_transformer_multi_dataset_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_multi_dataset_example.py - `scripts/train_sentence_transformer_static_embedding_example.py` — https://raw.githubusercontent.com/huggingface/skills/main/skills/train-sentence-transformers/scripts/train_sentence_transformer_static_embedding_example.py - …and 3 more, listed in https://skillsdocs.com/api/v1/books/huggingface/skills/skills/train-sentence-transformers <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: train-sentence-transformers description: Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval / pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task. --- # Train a sentence-transformers Model **This SKILL.md is a router, not a manual.** It tells you which references and example scripts to load for your task. The actual content — recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting — lives in `references/` and `scripts/`. **Do not synthesize a training script from this file alone.** Open the per-type production template (`scripts/train_<type>_example.py`) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, `force=True`, `seed`, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet. ## 1. Identify the model type | Tag | Class | What it does | When to pick | |---|---|---|---| | **[SentenceTransformer]** | `SentenceTransformer` (bi-encoder) | Maps each input to a fixed-dim dense vector | Retrieval, similarity, clustering, classification, paraphrase mining, dedup | | **[CrossEncoder]** | `CrossEncoder` (reranker) | Scores `(query, passage)` pairs jointly | Two-stage retrieval (rerank top-100 from bi-encoder), pair classification | | **[SparseEncoder]** | `SparseEncoder` (SPLADE) | Sparse vectors over the vocabulary | Learned-sparse retrieval, inverted-index backends (Elasticsearch / OpenSearch / Lucene) | Tiebreakers when the request is ambiguous: "embedding model" / "vector search" / "similarity" → **[SentenceTransformer]**. "rerank" / "ranker" / "two-stage" → **[CrossEncoder]**. "SPLADE" / "sparse" / "inverted index" → **[SparseEncoder]**. If still unclear, ask. ## 2. Required reading **Read these in full before writing any code. Do not triage by perceived relevance.** ### Per-type — always required **[SentenceTransformer]** - `references/losses_sentence_transformer.md` — loss-to-data-shape mapping; `BatchSamplers.NO_DUPLICATES` requirement for MNRL-family; `Cached*` ↔ `gradient_checkpointing` incompatibility. - `references/evaluators_sentence_transformer.md` — evaluator-to-task mapping; `metric_for_best_model` key construction (named vs unnamed); per-evaluator `primary_metric` values. - `references/model_architectures.md` — encoder vs decoder vs static vs Router pipelines; pooling rules (mean / cls / lasttoken); auto-mean-pooling behavior for fresh-start MLM bases. - `scripts/train_sentence_transformer_example.py` — production template; copy this as your starting point. **[CrossEncoder]** - `references/losses_cross_encoder.md` — pointwise / pairwise / listwise / distillation; `pos_weight` derivation; `activation_fn=Identity()` mandatory for non-BCE losses (silent eval-rank collapse otherwise). - `references/evaluators_cross_encoder.md` — `CrossEncoderRerankingEvaluator` recipe; named-evaluator key format `eval_{name}_{primary_metric}`. - `scripts/train_cross_encoder_example.py` — production template; copy this as your starting point. **[SparseEncoder]** - `references/losses_sparse_encoder.md` — `SpladeLoss` wrapper requirement; FLOPS regularizer weights; smoke-test active-dim ramp behavior. - `references/evaluators_sparse_encoder.md` — `SparseNanoBEIREvaluator` (English-only) and the in-domain alternative; `eval_{name}_{primary_metric}` key format. - `scripts/train_sparse_encoder_example.py` — production template; copy this as your starting point. ### Cross-cutting — always required (regardless of task) - `references/training_args.md` — `TrainingArguments` knobs, precision rules (load fp32 + autocast bf16/fp16; never `torch_dtype=bfloat16`), `warmup_steps` (float) vs deprecated `warmup_ratio`, `save_steps` must be a multiple of `eval_steps` for `load_best_model_at_end`, schedulers, HPO, tracker, resume, hub-push variants. - `references/dataset_formats.md` — column-matching rules (label name auto-detection; column-order-not-name); reshaping recipes; hard-negative mining options. - `references/base_model_selection.md` — discovery commands; per-type model namespaces; ModernBERT-family `max_seq_length=8192` trap; `datasets >= 4` script-loader rejection; non-English starting-point shortcuts. - `references/troubleshooting.md` — symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one; the "Metrics don't improve" and "Hub push fails" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after. ### Cross-cutting — load when applicable - `references/hardware_guide.md` — VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs. - `references/hf_jobs_execution.md` — required when running on HF Jobs. - `references/prompts_and_instructions.md` — required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding `query: ` / `passage: ` style prefixes. ### Variant scripts (open when the task matches) - **[SentenceTransformer]** `scripts/train_sentence_transformer_<matryoshka|multi_dataset|with_lora|distillation|make_multilingual|static_embedding>_example.py`. - **[CrossEncoder]** `scripts/train_cross_encoder_<distillation|listwise>_example.py`. - **[SparseEncoder]** `scripts/train_sparse_encoder_distillation_example.py`. - Hard-negative mining CLI — `scripts/mine_hard_negatives.py`. ## 3. Defaults Override only if the user specifies otherwise: - **Local execution.** Pitch HF Jobs only if local hardware can't fit the job. - **Single run.** After it completes, propose experimentation if the user would benefit (weak/marginal verdict, "see how high you can push it" framing, etc.). Iteration rules in `references/training_args.md` (Experimentation section). - **Public Hub push at end-of-run, wrapped in try-except.** On HF Jobs (ephemeral env) ALSO enable in-trainer push (`push_to_hub=True` + `hub_strategy="every_save"`); details in `references/hf_jobs_execution.md`. ## 4. Constraints the produced script must satisfy These are non-negotiable contracts. Implementation lives in the production templates and references — do not reinvent. - Capture the pre-training evaluator score as `baseline_eval` **before** `trainer.train()`. - Emit a single end-of-run line: `VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=...`. A monitor scrapes for this. - Silence `httpx`, `httpcore`, `huggingface_hub`, `urllib3`, `filelock`, `fsspec` to WARNING (otherwise HF download URLs flood the agent's context). - Tee logs to `logs/{RUN_NAME}.log`. - End with `model.push_to_hub(...)` wrapped in `try/except`. - Smoke-test before any long run (`max_steps=1` + tiny dataset slice). The production templates show one common pattern (`SMOKE_TEST` env var). - **[CrossEncoder]** Include `EarlyStoppingCallback(patience>=3)` — CE rerankers often peak mid-training and regress. - **[SparseEncoder]** Log `query_active_dims` / `corpus_active_dims` on the verdict line; high nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. `..._query_active_dims`); use suffix matching to pluck them — see the SPARSE production template for the exact pattern. ## 5. Workflow 1. Identify the model type (§1). Ask if ambiguous. 2. Load the §2 required-reading files for that type. 3. Open `scripts/train_<type>_example.py` and copy it as your starting point. 4. Replace `MODEL_NAME`, `DATASET_NAME`, `RUN_NAME`, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against `references/losses_<type>.md`; cross-check the `metric_for_best_model` key against `references/evaluators_<type>.md` (named evaluators format the key as `eval_{name}_{primary_metric}`). 5. Smoke-test (`max_steps=1`). 6. Run. 7. After the run, append to `logs/experiments.md` and propose iteration if the verdict is weak/marginal. ## Prerequisites ```bash pip install "sentence-transformers[train]>=5.0" # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal pip install trackio # optional tracker; or wandb / tensorboard / mlflow hf auth login # or set HF_TOKEN with write scope (for Hub push) ``` GPU strongly recommended. CPU works only for demos and `[SentenceTransformer]` `StaticEmbedding`. <!-- chapter:end slug=train-sentence-transformers --> --- <!-- chapter:begin slug=transformers-js position=25 --> ## 25. transformers-js - **Source:** https://github.com/huggingface/skills/blob/main/skills/transformers-js/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/transformers-js.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `references/CACHE.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/CACHE.md - `references/CONFIGURATION.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/CONFIGURATION.md - `references/EXAMPLES.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/EXAMPLES.md - `references/MODEL_ARCHITECTURES.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/MODEL_ARCHITECTURES.md - `references/MODEL_REGISTRY.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/MODEL_REGISTRY.md - `references/PIPELINE_OPTIONS.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/PIPELINE_OPTIONS.md - `references/TEXT_GENERATION.md` — https://raw.githubusercontent.com/huggingface/skills/main/skills/transformers-js/references/TEXT_GENERATION.md <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: transformers-js description: Use Transformers.js to run state-of-the-art machine learning models directly in JavaScript/TypeScript. Supports NLP (text classification, translation, summarization), computer vision (image classification, object detection), audio (speech recognition, audio classification), and multimodal tasks. Works in browsers and server-side runtimes (Node.js, Bun, Deno) with WebGPU/WASM using pre-trained models from Hugging Face Hub. license: Apache-2.0 metadata: author: huggingface version: "4.x" category: machine-learning repository: https://github.com/huggingface/transformers.js compatibility: Requires Node.js 18+ (or compatible Bun/Deno runtime) or modern browser with ES modules support. WebGPU requires runtime and hardware support; WASM is the broad fallback. Internet access is needed for downloading models from Hugging Face Hub (optional if using local models). --- # Transformers.js - Machine Learning for JavaScript Transformers.js enables running state-of-the-art machine learning models directly in JavaScript across browsers and server-side runtimes (Node.js, Bun, Deno), with no Python server required. ## When to Use This Skill Use this skill when you need to: - Run ML models for text analysis, generation, or translation in JavaScript - Perform image classification, object detection, or segmentation - Implement speech recognition or audio processing - Build multimodal AI applications (text-to-image, image-to-text, etc.) - Run models client-side in the browser without a backend ## Installation ### NPM Installation ```bash npm install @huggingface/transformers ``` ### Browser Usage (CDN) ```javascript <script type="module"> import { pipeline } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers'; </script> ``` ## Core Concepts ### 1. Pipeline API The pipeline API is the easiest way to use models. It groups together preprocessing, model inference, and postprocessing: ```javascript import { pipeline } from '@huggingface/transformers'; // Create a pipeline for a specific task const pipe = await pipeline('sentiment-analysis'); // Use the pipeline const result = await pipe('I love transformers!'); // Output: [{ label: 'POSITIVE', score: 0.999817686 }] // IMPORTANT: Always dispose when done to free memory await pipe.dispose(); ``` **⚠️ Memory Management:** All pipelines must be disposed with `pipe.dispose()` when finished to prevent memory leaks. See examples in [Code Examples](./references/EXAMPLES.md) for cleanup patterns across different environments. ### 2. Model Selection You can specify a custom model as the second argument: ```javascript const pipe = await pipeline( 'sentiment-analysis', 'Xenova/bert-base-multilingual-uncased-sentiment' ); ``` **Finding Models:** Browse available Transformers.js models on Hugging Face Hub: - **All models**: https://huggingface.co/models?library=transformers.js&sort=trending - **By task**: Add `pipeline_tag` parameter - Text generation: https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending - Image classification: https://huggingface.co/models?pipeline_tag=image-classification&library=transformers.js&sort=trending - Speech recognition: https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&library=transformers.js&sort=trending **Tip:** Filter by task type, sort by trending/downloads, and check model cards for performance metrics and usage examples. ### 3. Device Selection Choose where to run the model: ```javascript // Run on CPU (default for WASM) const pipe = await pipeline('sentiment-analysis', 'model-id'); // Run on GPU (WebGPU) const pipe = await pipeline('sentiment-analysis', 'model-id', { device: 'webgpu', }); ``` ### 4. Quantization Options Control model precision vs. performance: ```javascript // Use quantized model (faster, smaller) const pipe = await pipeline('sentiment-analysis', 'model-id', { dtype: 'q4', // Options: 'fp32', 'fp16', 'q8', 'q4' }); ``` ## Supported Tasks **Note:** All examples below show basic usage. ### Natural Language Processing #### Text Classification ```javascript const classifier = await pipeline('text-classification'); const result = await classifier('This movie was amazing!'); ``` #### Named Entity Recognition (NER) ```javascript const ner = await pipeline('token-classification'); const entities = await ner('My name is John and I live in New York.'); ``` #### Question Answering ```javascript const qa = await pipeline('question-answering'); const answer = await qa({ question: 'What is the capital of France?', context: 'Paris is the capital and largest city of France.' }); ``` #### Text Generation ```javascript const generator = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX'); const text = await generator('Once upon a time', { max_new_tokens: 100, temperature: 0.7 }); ``` **For streaming and chat:** See **[Text Generation Guide](./references/TEXT_GENERATION.md)** for: - Streaming token-by-token output with `TextStreamer` - Chat/conversation format with system/user/assistant roles - Generation parameters (temperature, top_k, top_p) - Browser and Node.js examples - React components and API endpoints #### Translation ```javascript const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M'); const output = await translator('Hello, how are you?', { src_lang: 'eng_Latn', tgt_lang: 'fra_Latn' }); ``` #### Summarization ```javascript const summarizer = await pipeline('summarization'); const summary = await summarizer(longText, { max_length: 100, min_length: 30 }); ``` #### Zero-Shot Classification ```javascript const classifier = await pipeline('zero-shot-classification'); const result = await classifier('This is a story about sports.', ['politics', 'sports', 'technology']); ``` ### Computer Vision #### Image Classification ```javascript const classifier = await pipeline('image-classification'); const result = await classifier('https://example.com/image.jpg'); // Or with local file const result = await classifier(imageUrl); ``` #### Object Detection ```javascript const detector = await pipeline('object-detection'); const objects = await detector('https://example.com/image.jpg'); // Returns: [{ label: 'person', score: 0.95, box: { xmin, ymin, xmax, ymax } }, ...] ``` #### Image Segmentation ```javascript const segmenter = await pipeline('image-segmentation'); const segments = await segmenter('https://example.com/image.jpg'); ``` #### Depth Estimation ```javascript const depthEstimator = await pipeline('depth-estimation'); const depth = await depthEstimator('https://example.com/image.jpg'); ``` #### Zero-Shot Image Classification ```javascript const classifier = await pipeline('zero-shot-image-classification'); const result = await classifier('image.jpg', ['cat', 'dog', 'bird']); ``` ### Audio Processing #### Automatic Speech Recognition ```javascript const transcriber = await pipeline('automatic-speech-recognition'); const result = await transcriber('audio.wav'); // Returns: { text: 'transcribed text here' } ``` #### Audio Classification ```javascript const classifier = await pipeline('audio-classification'); const result = await classifier('audio.wav'); ``` #### Text-to-Speech ```javascript const synthesizer = await pipeline('text-to-speech', 'Xenova/speecht5_tts'); const audio = await synthesizer('Hello, this is a test.', { speaker_embeddings: speakerEmbeddings }); ``` ### Multimodal #### Image-to-Text (Image Captioning) ```javascript const captioner = await pipeline('image-to-text'); const caption = await captioner('image.jpg'); ``` #### Document Question Answering ```javascript const docQA = await pipeline('document-question-answering'); const answer = await docQA('document-image.jpg', 'What is the total amount?'); ``` #### Zero-Shot Object Detection ```javascript const detector = await pipeline('zero-shot-object-detection'); const objects = await detector('image.jpg', ['person', 'car', 'tree']); ``` ### Feature Extraction (Embeddings) ```javascript const extractor = await pipeline('feature-extraction'); const embeddings = await extractor('This is a sentence to embed.'); // Returns: tensor of shape [1, sequence_length, hidden_size] // For sentence embeddings (mean pooling) const extractor = await pipeline('feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX'); const embeddings = await extractor('Text to embed', { pooling: 'mean', normalize: true }); ``` ## Finding and Choosing Models ### Browsing the Hugging Face Hub Discover compatible Transformers.js models on Hugging Face Hub: **Base URL (all models):** ``` https://huggingface.co/models?library=transformers.js&sort=trending ``` **Filter by task** using the `pipeline_tag` parameter: | Task | URL | |------|-----| | **Text Generation** | https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending | | **Text Classification** | https://huggingface.co/models?pipeline_tag=text-classification&library=transformers.js&sort=trending | | **Translation** | https://huggingface.co/models?pipeline_tag=translation&library=transformers.js&sort=trending | | **Summarization** | https://huggingface.co/models?pipeline_tag=summarization&library=transformers.js&sort=trending | | **Question Answering** | https://huggingface.co/models?pipeline_tag=question-answering&library=transformers.js&sort=trending | | **Image Classification** | https://huggingface.co/models?pipeline_tag=image-classification&library=transformers.js&sort=trending | | **Object Detection** | https://huggingface.co/models?pipeline_tag=object-detection&library=transformers.js&sort=trending | | **Image Segmentation** | https://huggingface.co/models?pipeline_tag=image-segmentation&library=transformers.js&sort=trending | | **Speech Recognition** | https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&library=transformers.js&sort=trending | | **Audio Classification** | https://huggingface.co/models?pipeline_tag=audio-classification&library=transformers.js&sort=trending | | **Image-to-Text** | https://huggingface.co/models?pipeline_tag=image-to-text&library=transformers.js&sort=trending | | **Feature Extraction** | https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js&sort=trending | | **Zero-Shot Classification** | https://huggingface.co/models?pipeline_tag=zero-shot-classification&library=transformers.js&sort=trending | **Sort options:** - `&sort=trending` - Most popular recently - `&sort=downloads` - Most downloaded overall - `&sort=likes` - Most liked by community - `&sort=modified` - Recently updated ### Choosing the Right Model Consider these factors when selecting a model: **1. Model Size** - **Small (< 100MB)**: Fast, suitable for browsers, limited accuracy - **Medium (100MB - 500MB)**: Balanced performance, good for most use cases - **Large (> 500MB)**: High accuracy, slower, better for Node.js or powerful devices **2. Quantization** Models are often available in different quantization levels: - `fp32` - Full precision (largest, most accurate) - `fp16` - Half precision (smaller, still accurate) - `q8` - 8-bit quantized (much smaller, slight accuracy loss) - `q4` - 4-bit quantized (smallest, noticeable accuracy loss) **3. Task Compatibility** Check the model card for: - Supported tasks (some models support multiple tasks) - Input/output formats - Language support (multilingual vs. English-only) - License restrictions **4. Performance Metrics** Model cards typically show: - Accuracy scores - Benchmark results - Inference speed - Memory requirements ### Example: Finding a Text Generation Model ```javascript // 1. Visit: https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending // 2. Browse and select a model (e.g., onnx-community/gemma-3-270m-it-ONNX) // 3. Check model card for: // - Model size: ~270M parameters // - Quantization: q4 available // - Language: English // - Use case: Instruction-following chat // 4. Use the model: import { pipeline } from '@huggingface/transformers'; const generator = await pipeline( 'text-generation', 'onnx-community/gemma-3-270m-it-ONNX', { dtype: 'q4' } // Use quantized version for faster inference ); const output = await generator('Explain quantum computing in simple terms.', { max_new_tokens: 100 }); await generator.dispose(); ``` ### Tips for Model Selection 1. **Start Small**: Test with a smaller model first, then upgrade if needed 2. **Check ONNX Support**: Ensure the model has ONNX files (look for `onnx` folder in model repo) 3. **Read Model Cards**: Model cards contain usage examples, limitations, and benchmarks 4. **Test Locally**: Benchmark inference speed and memory usage in your environment 5. **Filter by Library**: Use `library=transformers.js` to find compatible models: https://huggingface.co/models?library=transformers.js 6. **Version Pin**: Use specific git commits in production for stability: ```javascript const pipe = await pipeline('task', 'model-id', { revision: 'abc123' }); ``` ## Advanced Configuration ### Environment Configuration (`env`) The `env` object provides comprehensive control over Transformers.js execution, caching, and model loading. **Quick Overview:** ```javascript import { env, LogLevel } from '@huggingface/transformers'; // View version console.log(env.version); // e.g., '4.x' // Common settings env.allowRemoteModels = true; // Load from Hugging Face Hub env.allowLocalModels = false; // Load from file system env.localModelPath = '/models/'; // Local model directory env.useFSCache = true; // Cache models on disk (Node.js) env.useBrowserCache = true; // Cache models in browser env.cacheDir = './.cache'; // Cache directory location // Optional: override logging level (default is LogLevel.WARNING) env.logLevel = LogLevel.INFO; // Optional: custom fetch for auth headers, retries, abort signals, etc. env.fetch = (url, options) => fetch(url, { ...options, headers: { ...options?.headers, Authorization: `Bearer ${HF_TOKEN}`, }, }); ``` **Configuration Patterns:** ```javascript // Development: Fast iteration with remote models env.allowRemoteModels = true; env.useFSCache = true; // Production: Local models only env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = '/app/models/'; // Custom CDN env.remoteHost = 'https://cdn.example.com/models'; // Disable caching (testing) env.useFSCache = false; env.useBrowserCache = false; ``` For complete documentation on all configuration options, caching strategies, cache management, pre-downloading models, and more, see: **→ [Configuration Reference](./references/CONFIGURATION.md)** ### ModelRegistry (v4) `ModelRegistry` gives you visibility and control over model assets before loading a pipeline. Use it to estimate download size, check cache status, inspect available dtypes, and clear cached artifacts for a specific task/model/options tuple. ```javascript import { ModelRegistry } from '@huggingface/transformers'; const task = 'feature-extraction'; const modelId = 'onnx-community/all-MiniLM-L6-v2-ONNX'; const modelOptions = { dtype: 'fp32' }; // List required files for this pipeline const files = await ModelRegistry.get_pipeline_files(task, modelId, modelOptions); // Check if assets are already cached const cached = await ModelRegistry.is_pipeline_cached(task, modelId, modelOptions); // Inspect precision formats available for this model const dtypes = await ModelRegistry.get_available_dtypes(modelId); console.log({ files: files.length, cached, dtypes }); ``` For production patterns and full API coverage, see **[ModelRegistry Reference](./references/MODEL_REGISTRY.md)**. ### Standalone Tokenization (`@huggingface/tokenizers`) For tokenization-only workflows, use `@huggingface/tokenizers`. It is a separate lightweight package useful when you need fast tokenization/encoding without loading full model inference pipelines. ```bash npm install @huggingface/tokenizers ``` ```javascript import { Tokenizer } from '@huggingface/tokenizers'; ``` ### Working with Tensors ```javascript import { AutoTokenizer, AutoModel } from '@huggingface/transformers'; // Load tokenizer and model separately for more control const tokenizer = await AutoTokenizer.from_pretrained('bert-base-uncased'); const model = await AutoModel.from_pretrained('bert-base-uncased'); // Tokenize input const inputs = await tokenizer('Hello world!'); // Run model const outputs = await model(inputs); ``` ### Batch Processing ```javascript const classifier = await pipeline('sentiment-analysis'); // Process multiple texts const results = await classifier([ 'I love this!', 'This is terrible.', 'It was okay.' ]); ``` ## Runtime-Specific Considerations ### WebGPU Usage WebGPU provides GPU acceleration in browsers and server-side runtimes (when supported): ```javascript const pipe = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX', { device: 'webgpu', dtype: 'fp32' }); ``` **Note**: Use `webgpu` when available and fall back to WASM/CPU when not supported in the current runtime. ### WASM Performance WASM is the most compatible execution backend across runtimes: ```javascript // Optimized for browsers with quantization const pipe = await pipeline('sentiment-analysis', 'model-id', { dtype: 'q8' // or 'q4' for even smaller size }); ``` ### Progress Tracking & Loading Indicators Models can be large (ranging from a few MB to several GB) and consist of multiple files. Track download progress by passing a callback to the `pipeline()` function: ```javascript import { pipeline } from '@huggingface/transformers'; // Track progress for each file const fileProgress = {}; function onProgress(info) { if (info.status === 'progress_total') { console.log(`Total: ${info.progress.toFixed(1)}%`); return; } console.log(`${info.status}: ${info.file ?? ''}`); if (info.status === 'progress') { fileProgress[info.file] = info.progress; console.log(`${info.file}: ${info.progress.toFixed(1)}%`); } if (info.status === 'done') { console.log(`✓ ${info.file} complete`); } } // Pass callback to pipeline const classifier = await pipeline('sentiment-analysis', null, { progress_callback: onProgress }); ``` **Progress Info Properties:** ```typescript interface ProgressInfo { status: 'initiate' | 'download' | 'progress' | 'progress_total' | 'done' | 'ready'; name: string; // Model id or path file?: string; // File being processed (per-file events) progress?: number; // Percentage (0-100, for 'progress' and 'progress_total') loaded?: number; // Bytes downloaded (only for 'progress' status) total?: number; // Total bytes (only for 'progress' status) } ``` For complete examples including browser UIs, React components, CLI progress bars, and retry logic, see: **→ [Pipeline Options - Progress Callback](./references/PIPELINE_OPTIONS.md#progress-callback)** ## Error Handling ```javascript try { const pipe = await pipeline('sentiment-analysis', 'model-id'); const result = await pipe('text to analyze'); } catch (error) { if (error.message.includes('fetch')) { console.error('Model download failed. Check internet connection.'); } else if (error.message.includes('ONNX')) { console.error('Model execution failed. Check model compatibility.'); } else { console.error('Unknown error:', error); } } ``` ## Performance Tips 1. **Reuse Pipelines**: Create pipeline once, reuse for multiple inferences 2. **Use Quantization**: Start with `q8` or `q4` for faster inference 3. **Batch Processing**: Process multiple inputs together when possible 4. **Cache Models**: Models are cached automatically (see **[Caching Reference](./references/CACHE.md)** for details on browser Cache API, Node.js filesystem cache, and custom implementations) 5. **WebGPU for Large Models**: Use WebGPU for models that benefit from GPU acceleration 6. **Prune Context**: For text generation, limit `max_new_tokens` to avoid memory issues 7. **Clean Up Resources**: Call `pipe.dispose()` when done to free memory ## Memory Management **IMPORTANT:** Always call `pipe.dispose()` when finished to prevent memory leaks. ```javascript const pipe = await pipeline('sentiment-analysis'); const result = await pipe('Great product!'); await pipe.dispose(); // ✓ Free memory (100MB - several GB per model) ``` **When to dispose:** - Application shutdown or component unmount - Before loading a different model - After batch processing in long-running apps Models consume significant memory and hold GPU/CPU resources. Disposal is critical for browser memory limits and server stability. For detailed patterns (React cleanup, servers, browser), see **[Code Examples](./references/EXAMPLES.md)** ## Troubleshooting ### Model Not Found - Verify model exists on Hugging Face Hub - Check model name spelling - Ensure model has ONNX files (look for `onnx` folder in model repo) ### Memory Issues - Use smaller models or quantized versions (`dtype: 'q4'`) - Reduce batch size - Limit sequence length with `max_length` ### WebGPU Errors - Check browser compatibility (Chrome 113+, Edge 113+) - Try `dtype: 'fp16'` if `fp32` fails - Fall back to WASM if WebGPU unavailable ## Reference Documentation ### This Skill - **[Pipeline Options](./references/PIPELINE_OPTIONS.md)** - Configure `pipeline()` with `progress_callback`, `device`, `dtype`, etc. - **[Configuration Reference](./references/CONFIGURATION.md)** - Global `env` configuration for caching and model loading - **[ModelRegistry Reference](./references/MODEL_REGISTRY.md)** - Inspect files, cache status, dtypes, and clear cache before loading pipelines - **[Caching Reference](./references/CACHE.md)** - Browser Cache API, Node.js filesystem cache, and custom cache implementations - **[Text Generation Guide](./references/TEXT_GENERATION.md)** - Streaming, chat format, and generation parameters - **[Model Architectures](./references/MODEL_ARCHITECTURES.md)** - Supported models and selection tips - **[Code Examples](./references/EXAMPLES.md)** - Real-world implementations for different runtimes ### Official Transformers.js - Official docs: https://huggingface.co/docs/transformers.js - API reference: https://huggingface.co/docs/transformers.js/api/pipelines - Model hub: https://huggingface.co/models?library=transformers.js - GitHub: https://github.com/huggingface/transformers.js - Examples: https://github.com/huggingface/transformers.js-examples ## Best Practices 1. **Always Dispose Pipelines**: Call `pipe.dispose()` when done - critical for preventing memory leaks 2. **Start with Pipelines**: Use the pipeline API unless you need fine-grained control 3. **Test Locally First**: Test models with small inputs before deploying 4. **Monitor Model Sizes**: Be aware of model download sizes for web applications 5. **Handle Loading States**: Show progress indicators for better UX 6. **Version Pin**: Pin specific model versions for production stability 7. **Error Boundaries**: Always wrap pipeline calls in try-catch blocks 8. **Progressive Enhancement**: Provide fallbacks for unsupported browsers 9. **Reuse Models**: Load once, use many times - don't recreate pipelines unnecessarily 10. **Graceful Shutdown**: Dispose models on SIGTERM/SIGINT in servers ## Quick Reference: Task IDs | Task | Task ID | |------|---------| | Text classification | `text-classification` or `sentiment-analysis` | | Token classification | `token-classification` or `ner` | | Question answering | `question-answering` | | Fill mask | `fill-mask` | | Summarization | `summarization` | | Translation | `translation` | | Text generation | `text-generation` | | Text-to-text generation | `text2text-generation` | | Zero-shot classification | `zero-shot-classification` | | Image classification | `image-classification` | | Image segmentation | `image-segmentation` | | Object detection | `object-detection` | | Depth estimation | `depth-estimation` | | Image-to-image | `image-to-image` | | Zero-shot image classification | `zero-shot-image-classification` | | Zero-shot object detection | `zero-shot-object-detection` | | Automatic speech recognition | `automatic-speech-recognition` | | Audio classification | `audio-classification` | | Text-to-speech | `text-to-speech` or `text-to-audio` | | Image-to-text | `image-to-text` | | Document question answering | `document-question-answering` | | Feature extraction | `feature-extraction` | | Sentence similarity | `sentence-similarity` | --- This skill enables you to integrate state-of-the-art machine learning capabilities directly into JavaScript applications without requiring separate ML servers or Python environments. <!-- chapter:end slug=transformers-js --> --- <!-- chapter:begin slug=trl-training position=26 --> ## 26. trl-training - **Source:** https://github.com/huggingface/skills/blob/main/skills/trl-training/SKILL.md - **Raw:** https://raw.githubusercontent.com/huggingface/skills/main/skills/trl-training/SKILL.md - **Markdown:** https://skillsdocs.com/huggingface/skills/trl-training.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html <!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. --> --- name: trl-training description: Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands. license: Apache-2.0 metadata: version: "1.0.0" author: huggingface commands: trl sft, trl dpo, trl grpo, trl kto, trl rloo, trl reward categories: machine-learning, llm-training, reinforcement-learning tags: rlhf, supervised-fine-tuning, dpo, grpo, huggingface, transformers documentation: https://huggingface.co/docs/trl/en/clis --- # TRL Training Skill You are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models. ## Overview TRL provides CLI commands for post-training foundation models using state-of-the-art techniques: - **SFT** (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets - **DPO** (Direct Preference Optimization): Align models using preference data - **GRPO** (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards. - **RLOO** (Reinforce Leave One Out): Online RL training with generation-based rewards - **Reward Model Training**: Train reward models for RLHF TRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem. ## Core Commands ### trl sft - Supervised Fine-Tuning Fine-tune language models on instruction-following or conversational datasets. **Full training:** ```bash trl sft \ --model_name_or_path Qwen/Qwen2-0.5B \ --dataset_name trl-lib/Capybara \ --learning_rate 2.0e-5 \ --num_train_epochs 1 \ --packing \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --eos_token '<|im_end|>' \ --eval_strategy steps \ --eval_steps 100 \ --output_dir Qwen2-0.5B-SFT \ --push_to_hub ``` **Train with LoRA adapters:** ```bash trl sft \ --model_name_or_path Qwen/Qwen2-0.5B \ --dataset_name trl-lib/Capybara \ --learning_rate 2.0e-4 \ --num_train_epochs 1 \ --packing \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --eos_token '<|im_end|>' \ --eval_strategy steps \ --eval_steps 100 \ --use_peft \ --lora_r 32 \ --lora_alpha 16 \ --output_dir Qwen2-0.5B-SFT \ --push_to_hub ``` ### trl dpo - Direct Preference Optimization Align models using preference data (chosen/rejected pairs). **Full training:** ```bash trl dpo \ --dataset_name trl-lib/ultrafeedback_binarized \ --model_name_or_path Qwen/Qwen2-0.5B-Instruct \ --learning_rate 5.0e-7 \ --num_train_epochs 1 \ --per_device_train_batch_size 2 \ --max_steps 1000 \ --gradient_accumulation_steps 8 \ --eval_strategy steps \ --eval_steps 50 \ --output_dir Qwen2-0.5B-DPO \ --no_remove_unused_columns ``` **Train with LoRA adapters:** ```bash trl dpo \ --dataset_name trl-lib/ultrafeedback_binarized \ --model_name_or_path Qwen/Qwen2-0.5B-Instruct \ --learning_rate 5.0e-6 \ --num_train_epochs 1 \ --per_device_train_batch_size 2 \ --max_steps 1000 \ --gradient_accumulation_steps 8 \ --eval_strategy steps \ --eval_steps 50 \ --output_dir Qwen2-0.5B-DPO \ --no_remove_unused_columns \ --use_peft \ --lora_r 32 \ --lora_alpha 16 ``` ### trl grpo - Group Relative Policy Optimization Train models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards. **Basic usage:** ```bash trl grpo \ --model_name_or_path Qwen/Qwen2.5-0.5B \ --dataset_name trl-lib/gsm8k \ --reward_funcs accuracy_reward \ --output_dir Qwen2-0.5B-GRPO \ --push_to_hub ``` ### trl rloo - Reinforce Leave One Out Online RL training where the model generates text and receives rewards based on custom criteria. **Basic usage:** ```bash trl rloo \ --model_name_or_path Qwen/Qwen2.5-0.5B \ --dataset_name trl-lib/tldr \ --reward_model_name_or_path sentiment-analysis:nlptown/bert-base-multilingual-uncased-sentiment \ --output_dir Qwen2-0.5B-RLOO \ --push_to_hub ``` ### trl reward - Reward Model Training Train a reward model to score text quality for RLHF. **Full training:** ```bash trl reward \ --model_name_or_path Qwen/Qwen2-0.5B-Instruct \ --dataset_name trl-lib/ultrafeedback_binarized \ --output_dir Qwen2-0.5B-Reward \ --per_device_train_batch_size 8 \ --num_train_epochs 1 \ --learning_rate 1.0e-5 \ --eval_strategy steps \ --eval_steps 50 \ --max_length 2048 ``` **Train with LoRA adapters:** ```bash trl reward \ --model_name_or_path Qwen/Qwen2-0.5B-Instruct \ --dataset_name trl-lib/ultrafeedback_binarized \ --output_dir Qwen2-0.5B-Reward-LoRA \ --per_device_train_batch_size 8 \ --num_train_epochs 1 \ --learning_rate 1.0e-4 \ --eval_strategy steps \ --eval_steps 50 \ --max_length 2048 \ --use_peft \ --lora_task_type SEQ_CLS \ --lora_r 32 \ --lora_alpha 16 ``` ## Configuration Files TRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file. **Example config (sft_config.yaml):** ```yaml model_name_or_path: Qwen/Qwen2.5-0.5B dataset_name: trl-lib/Capybara learning_rate: 2.0e-5 num_train_epochs: 1 per_device_train_batch_size: 8 gradient_accumulation_steps: 2 output_dir: ./sft_output use_peft: true lora_r: 16 lora_alpha: 16 report_to: trackio ``` **Launch with config:** ```bash trl sft --config sft_config.yaml ``` **Override config values:** ```bash trl sft --config sft_config.yaml --learning_rate 1.0e-5 ``` ## Distributed Training TRL integrates with Accelerate for multi-GPU and multi-node training. **Multi-GPU training:** ```bash trl sft \ --config sft_config.yaml \ --num_processes 4 ``` **Use predefined Accelerate configs:** TRL provides predefined configs: `single_gpu`, `multi_gpu`, `fsdp1`, `fsdp2`, `zero1`, `zero2`, `zero3` ```bash trl sft \ --config sft_config.yaml \ --accelerate_config zero2 ``` **Custom Accelerate config:** ```bash # Generate custom config accelerate config # Use custom config trl sft --config sft_config.yaml --config_file ~/.cache/huggingface/accelerate/default_config.yaml ``` **Fully Sharded Data Parallel (FSDP):** ```bash trl sft --config sft_config.yaml --accelerate_config fsdp2 ``` **DeepSpeed ZeRO:** ```bash trl sft --config sft_config.yaml --accelerate_config zero3 ``` ## Troubleshooting ### CUDA Out of Memory - Reduce `--per_device_train_batch_size` and increase `--gradient_accumulation_steps` - Enable `--use_peft` for LoRA training - Use `--gradient_checkpointing` to save memory - Try smaller model or longer sequence truncation ### Dataset Loading Issues - Verify dataset exists: check Hugging Face Hub or local path - Check dataset format matches expected columns - Use `--dataset_config` for multi-config datasets - Inspect dataset: `from datasets import load_dataset; ds = load_dataset(name)` ### Model Loading Issues - Verify model exists on Hugging Face Hub - Check if gated model requires authentication: `hf auth login` - For local models, provide absolute path - Ensure sufficient disk space and memory ### Slow Training - Enable dataset `--packing` for short sequences - Use larger `--per_device_train_batch_size` if memory allows - Enable `--tf32` for faster computation on Ampere GPUs - Use `--bf16` on supported hardware - Consider multi-GPU training with `--num_processes` ### Generation Issues (GRPO/RLOO) - Check prompt format in dataset - Adjust `--temperature` and `--top_p` for generation - Verify the reward function (for GRPO/RLOO) ## Additional Resources - **Documentation**: https://huggingface.co/docs/trl - **GitHub**: https://github.com/huggingface/trl - **Examples**: https://github.com/huggingface/trl/tree/main/examples ## Best Practices 1. **Start with SFT**: Always fine-tune base models with SFT before preference alignment 2. **Use LoRA for efficiency**: Enable `--use_peft` for faster training and lower memory 3. **Monitor training**: Use `--report_to trackio` (or `--report_to wandb` or `--report_to tensorboard`) for tracking 4. **Save checkpoints**: TRL automatically saves checkpoints in `--output_dir` 5. **Test on small datasets first**: Verify pipeline works before full training 6. **Use configuration files**: Create YAML configs for reproducibility 7. **Leverage Accelerate**: Use multi-GPU training for faster iteration When helping users with TRL: - Always check which training method is appropriate for their use case - Verify dataset format matches the expected schema - Recommend starting with smaller models for testing - Suggest LoRA for resource-constrained environments - Point to specific documentation sections for advanced features <!-- chapter:end slug=trl-training -->