Setting the file. One moment.
Chapter 11 · Huggingface Community Evals
Subchapter 11.4
examples/USAGE_EXAMPLES.mdMarkdown2 KBView on GitHub
This document provides practical examples for running evaluations locally against Hugging Face Hub models.
inspect-ai local runsinspect-ai with vllm or Transformers backendslighteval local runs with vllm or acceleratemodel-index.eval_resultsIf you want to run these same scripts remotely, use the hugging-face-jobs skill and pass one of the scripts in scripts/.
cd skills/hugging-face-evaluation
export HF_TOKEN=hf_xxx
uv --versionFor local GPU runs:
nvidia-smiuv run scripts/inspect_eval_uv.py \
--model meta-llama/Llama-3.2-1B \
--task mmlu \
--limit 10uv run scripts/inspect_vllm_uv.py \
--model meta-llama/Llama-3.2-8B-Instruct \
--task gsm8k \
--limit 20uv run scripts/inspect_vllm_uv.py \
--model microsoft/phi-2 \
--task mmlu \
--backend hf \
--trust-remote-code \
--limit 20uv run scripts/lighteval_vllm_uv.py \
--model meta-llama/Llama-3.2-3B-Instruct \
--tasks "leaderboard|mmlu|5" \
--max-samples 20uv run scripts/lighteval_vllm_uv.py \
--model meta-llama/Llama-3.2-3B-Instruct \
--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
--max-samples 20 \
--use-chat-templateuv run scripts/lighteval_vllm_uv.py \
--model microsoft/phi-2 \
--tasks "leaderboard|mmlu|5" \
--backend accelerate \
--trust-remote-code \
--max-samples 20When local hardware is not enough, switch to the hugging-face-jobs skill and run one of these scripts remotely. Keep the script path and args; move the orchestration there.