Judge Rubric

1You are an expert judge evaluating the quality of an LLM experiment analysis report.
2
3## Experiment Context
4
5Baseline experiment ID: {{input_data.baseline}}
6Candidate experiment ID: {{input_data.candidate}}
7Question asked: {{input_data.question}}
8
9## Expected Answer
10
11{{expected_output.answer}}
12
13Key points that should be covered:
14{{expected_output.key_points}}
15
16## Generated Report
17
18{{output_data.report}}
19
20---
21
22## Scoring Instructions
23
24Score each dimension from 0 to 10. Be strict — reserve 9–10 for exceptional quality.
25
26### 1. answer_accuracy (0–10)
27
28Does the report's conclusion or verdict align with the expected answer?
29
30- 9–10: The report reaches the same conclusion as the expected answer with correct supporting evidence (specific metrics, counts, percentages). Nuances match.
31- 7–8: The verdict is correct but misses some nuance or uses approximate rather than exact numbers.
32- 5–6: The report is partially correct — gets the direction right but is vague, hedged, or misses a key part of the answer.
33- 3–4: The report is mostly wrong or contradicts the expected answer in a meaningful way.
34- 0–2: The report does not answer the question, produces no output, or gives the opposite conclusion.
35
36### 2. key_points_coverage (0–10)
37
38Does the report surface the key findings listed in the expected key points?
39
40- 9–10: All key points are explicitly addressed with matching evidence and correct interpretation.
41- 7–8: Most key points are covered; one is missing or has incorrect detail.
42- 5–6: Half or more key points are present but some are missing or shallow.
43- 3–4: Only one or two key points are touched, others are absent or wrong.
44- 0–2: Key points are not addressed or the report is empty/trivial.
45
46### 3. evidence_quality (0–10)
47
48Are the report's claims grounded in specific, quantified evidence from the experiment data?
49
50- 9–10: Every major claim cites specific numbers (metric pass rates, event counts, percentages, delta values). Reasoning is traceable to data.
51- 7–8: Most claims are quantified; a few are qualitative without supporting numbers.
52- 5–6: Some quantification present, but significant portions rely on vague language ("some events", "a few cases", "seems better").
53- 3–4: Claims are mostly qualitative with minimal data reference.
54- 0–2: No quantification; report is purely speculative or empty.
55
56### Weighted aggregate (for your reasoning only — do not output this score)
57
58Suggested weights: answer_accuracy × 0.5 + key_points_coverage × 0.3 + evidence_quality × 0.2
59
60---
61
62Return a JSON object with exactly these keys:
63
64{
65  "answer_accuracy": <integer 0–10>,
66  "key_points_coverage": <integer 0–10>,
67  "evidence_quality": <integer 0–10>,
68  "reasoning": "<2–4 sentences explaining the scores, referencing specific parts of the report>"
69}