Setting the file. One moment.
Skill 122 · Evaluation Methodology
Subchapter 122.1
references/rubrics.mdMarkdown23 KBView on GitHub
This document contains the full anchored rubrics used by the eval-judge agent (Layer 2) to
score skills on each of the four dimensions it assesses. Each dimension uses a 0.0–1.0 scale
with five anchor points. The judge interpolates between anchors based on the evidence gathered
from reading SKILL.md and any files.
references/These rubrics are the authoritative scoring standard. When calibrating expectations, filing score disputes, or training new judge models, use these anchors as ground truth.
Weight in composite: 0.25 (highest)
Layer blend (deep depth): static 15%, judge 25%, Monte Carlo 60%
Triggering accuracy measures whether the skill’s description field in the frontmatter
causes Claude Code to invoke the skill at the right times. A skill with perfect triggering
accuracy fires on every prompt that genuinely needs it (high recall) and never fires on
prompts where it is irrelevant (high precision). The score is conceptually the F1 of
precision and recall across a representative prompt distribution.
The judge generates 10 mental test prompts: 5 that should trigger the skill and 5 that should not. It assesses whether the description would lead Claude Code’s routing model to activate (or not activate) for each prompt. The F1 score of this 10-prompt evaluation becomes the dimension score.
The judge also considers whether the description provides actionable trigger signals rather than just naming or describing the skill in passive terms.
0.0 – 0.19 (Grade F) — Unusable trigger
The description is absent, empty, or so vague that it provides no routing signal. Examples:
A skill at this level will almost never be autonomously invoked. It may be invoked if the user explicitly names it, but that defeats the purpose of a plugin ecosystem.
0.20 – 0.39 (Grade F/D) — Weak trigger
The description exists and is somewhat meaningful but has major gaps:
Example of a 0.30-scoring description:
“PluginEval quality methodology — dimensions, rubrics, statistical methods.”
This names the topic but provides no trigger signal. The routing model cannot infer when to use it.
0.40 – 0.59 (Grade D/C) — Partial trigger
The description has some trigger signal but is imprecise:
Example of a 0.50-scoring description:
“PluginEval quality methodology — dimensions, rubrics. Use when understanding evaluation.”
Better — has a trigger phrase — but “understanding evaluation” is too generic. It would catch some legitimate uses but also fire for unrelated evaluation tasks.
0.60 – 0.79 (Grade C/B) — Good trigger
Description clearly identifies when to invoke the skill with only minor gaps:
Example of a 0.70-scoring description:
“PluginEval quality methodology. Use this skill when understanding how plugin quality is measured or when interpreting evaluation results.”
Good — two explicit trigger contexts — but misses calibration and stakeholder scenarios.
0.80 – 1.00 (Grade A/B) — Excellent trigger
Description is precise and comprehensive:
Example of a 0.90-scoring description:
“PluginEval quality methodology — dimensions, rubrics, statistical methods. Use this skill when understanding how plugin quality is measured, interpreting evaluation results, calibrating scoring thresholds, or explaining quality badges to stakeholders.”
Four specific, distinct contexts. Fires on exactly the right prompts.
Weight in composite: 0.20 (second highest)
Layer blend (deep depth): static 10%, judge 70%, Monte Carlo 20%
Orchestration fitness measures whether a skill behaves as a pure worker in the agent → skill hierarchy. A skill should receive a delegated task, execute it using its own instructions, and return structured output. It should NOT:
This dimension is almost entirely judge-assessed (70% judge weight) because static analysis cannot reliably detect orchestration intent from surface patterns alone.
The judge reads the SKILL.md in full and asks: does this skill’s instruction set define a worker (receives task → executes → returns output) or an orchestrator (plans → delegates → aggregates)? It looks for specific signals in both directions.
Worker signals (positive):
Orchestrator signals (negative):
0.0 – 0.19 (Grade F) — Standalone agent
The skill is written as a fully autonomous agent that manages its own tool calls, sub-task delegation, and workflow coordination. It has no defined input/output contract. It reads like an agent system prompt, not a worker instruction set.
Example characteristics:
0.20 – 0.39 (Grade F/D) — Mixed roles
The skill mixes worker and orchestrator responsibilities. It does some work itself but also contains orchestration logic. The boundaries are unclear.
Example characteristics:
0.40 – 0.59 (Grade D/C) — Functional worker with structural issues
The skill is mostly a worker but the output format is not structured for supervisor consumption. The calling agent cannot easily parse or route on the output.
Example characteristics:
0.60 – 0.79 (Grade C/B) — Clean worker, minor gaps
The skill functions as a clean worker. Inputs and outputs are documented. The instructions produce output that a supervisor agent can consume. Minor issues remain.
Example characteristics:
0.80 – 1.00 (Grade A/B) — Pure worker
The skill is a composable, contract-defined worker. It is clear what it takes in and what it produces. The output format is specified in a way that a calling agent can rely on.
Example characteristics:
Good signals (push score up):
Bad signals (push score down):
Weight in composite: 0.15 (third highest)
Layer blend (deep depth): static 0%, judge 40%, Monte Carlo 60%
Output quality measures whether the skill’s instructions would guide Claude to produce correct, complete, and useful output across a representative range of real-world tasks. This dimension is entirely empirical — static analysis cannot assess whether instructions will produce quality outputs, so the layer blend is 0% static.
At deep depth, Monte Carlo simulation (60% blend) produces actual outputs from real prompts and scores them. At standard depth (judge only), the judge simulates three tasks mentally.
The judge selects three realistic tasks that the skill is designed to handle — varying from simple to complex. For each task, it mentally executes the skill’s instructions and assesses whether the resulting output would be:
The average across three tasks becomes the dimension score.
0.0 – 0.19 (Grade F) — Instructions produce incorrect output
Following the skill’s instructions would lead Claude to produce wrong answers or actively harmful output. The instructions contain factual errors, logical contradictions, or directives that produce the opposite of the intended result.
Example characteristics:
0.20 – 0.39 (Grade F/D) — Incomplete, major gaps
Instructions produce output for simple cases but fail on anything non-trivial. Major aspects of the skill’s domain are unaddressed. A user following this skill would get partial help for basic requests and no help for moderate complexity.
Example characteristics:
0.40 – 0.59 (Grade D/C) — Adequate for basic cases
Instructions produce reasonable output for straightforward tasks but struggle with any complexity. The skill is usable but requires the user to fill in significant gaps.
Example characteristics:
0.60 – 0.79 (Grade C/B) — Good for most cases
Instructions produce quality output for the majority of realistic tasks. A few edge cases or complex scenarios may be handled suboptimally but the core use cases work well.
Example characteristics:
0.80 – 1.00 (Grade A/B) — Excellent across the board
Instructions are comprehensive, specific, and produce high-quality output for even complex or edge-case tasks. The skill represents a genuine expertise distillation.
Example characteristics:
When assessing code examples and technical instructions, the judge verifies:
Weight in composite: 0.12 (fourth highest)
Layer blend (deep depth): static 30%, judge 55%, Monte Carlo 15%
Scope calibration measures whether the skill is the right size for its purpose. Too thin (stub) and it provides no value. Too broad (bloated) and it wastes tokens, confuses the model, and overlaps with sibling skills. The ideal skill is exactly as large as it needs to be — comprehensive for its defined domain, not a line longer.
This dimension requires human judgment (55% judge blend) because “right size” is context-dependent. A skill covering a complex framework legitimately needs more content than a skill covering a simple utility function.
The judge assesses scope by asking:
The judge also considers the skill’s category (reference documentation, workflow assistant, code generator, etc.) when calibrating expectations.
0.0 – 0.19 (Grade F) — Stub
The skill is a placeholder. It has a name and description but the body contains less than 50 lines or covers fewer than half of its stated domain. Someone invoking this skill would receive fragmentary guidance insufficient to complete any real task.
Example characteristics:
0.20 – 0.39 (Grade F/D) — Too narrow
The skill covers its domain but only the surface layer. Important aspects exist but are mentioned without sufficient depth to be actionable. The skill is not a stub but it is thin enough that users will frequently run into unaddressed scenarios.
Example characteristics:
0.40 – 0.59 (Grade D/C) — Slightly off-scope
The skill is either moderately under-scoped (missing a few important aspects) or slightly over-scoped (includes content that belongs in a different skill). The content that exists is reasonable in quality but the overall package is not well-calibrated.
Example characteristics:
0.60 – 0.79 (Grade C/B) — Well-scoped with minor issues
The skill covers its domain well. Important aspects are addressed at appropriate depth. One or two gaps remain, or there is a small amount of tangential content, but these are minor issues.
Example characteristics:
0.80 – 1.00 (Grade A/B) — Perfectly calibrated
The skill is exactly what it needs to be. It covers all important aspects of its domain at the right depth, with no padding and no gaps. Every section earns its place. The skill could be used as a reference implementation for its category.
Example characteristics:
references/ for supporting material that doesn’t belong in the
main execution pathScope expectations vary by skill category. Use these as baseline calibration guides:
| Category | Target lines (SKILL.md) | Pattern |
|---|---|---|
| Reference / Documentation | 200–500 | Deep coverage + references/ for extended material |
| Workflow / Process | 150–300 | Step-by-step + decision points + worked example |
| Code generator | 100–200 | Instructions + references/ for templates |
| Diagnostic / Debugging | 200–400 | Decision trees + failure modes + procedures |
| Integration / Configuration | 150–350 | Setup + options + copy-paste examples |
| Coordination / Planning | 100–200 | Decisions + checklists + handoff protocol |
When running with judges > 1, PluginEval reports Cohen’s kappa to measure agreement
between judge instances. Target kappa ≥ 0.70 for a stable, well-defined skill.
| Kappa range | Interpretation |
|---|---|
| ≥ 0.80 | Strong agreement — skill is clearly written |
| 0.60 – 0.79 | Moderate agreement — skill has some ambiguous sections |
| 0.40 – 0.59 | Fair agreement — skill needs clarity improvements |
| < 0.40 | Poor agreement — skill is ambiguous or judges are not calibrated |
Low kappa on a specific dimension points to the area needing clarification. Low triggering_accuracy kappa usually means the description maps to multiple different interpretations of when to use the skill.
The gold corpus (initialized via plugin-eval init) provides Platinum and Gold-badged
skills as calibration anchors. Before running a batch evaluation, compare your expected
scores against one or two corpus entries to verify your judge is calibrated correctly.
If your judge consistently scores a known Platinum skill below 85 on any dimension, check for model version drift or prompt injection in the skill content that may be confusing the judge.
Judge model upgrades can shift scores by ± 5–10 points on subjective dimensions (output_quality, scope_calibration). After any model upgrade, re-certify the top 10 corpus entries to establish new baseline calibration. If drift exceeds 5 points on any dimension, update the anchored examples in this rubric document to reflect the new model’s scoring behavior.