1You are evaluating an AI-generated root cause analysis (RCA) report for a production LLM application.23The report was generated by analyzing LLM observability traces for the following application:45**Application**: {{input_data.ml_app}} / {{input_data.eval_name}}6**Timeframe**: {{input_data.timeframe}}7**Mode**: {{input_data.mode}} (eval, errors, or generic — determines what signal was used)89---1011## Expected Findings1213The following key findings and recommendations were identified by a human analyst as ground truth for this scenario:1415**Key findings** (root causes and failure patterns that should be diagnosed):16{{expected_output.key_findings}}1718**Expected recommendations** (actionable changes that should be proposed):
31Score each dimension from 0 to 10. Use the full range.
32
33### 1. diagnosis_accuracy (0–10)
34
35Does the report correctly identify the root causes described in the expected findings?
36
37- **10**: All key root causes from expected findings are identified, correctly named, and traced to the right span or system component. No significant false diagnoses.
38- **7–9**: Most key root causes identified. Minor misattributions or missing one secondary cause.
39- **4–6**: Some root causes identified but important ones are missing, or the diagnosis is at the symptom level ("model gave bad answer") rather than tracing to a specific root cause (e.g., "system prompt lacks date format instructions in the parent agent span").
40- **1–3**: Very few root causes identified. Report mostly describes what failed without diagnosing why.
41- **0**: No root causes identified, or diagnoses are entirely wrong.
42
43### 2. evidence_grounding (0–10)
44
45Are claims backed by trace evidence — span IDs, quoted judge reasoning, specific content from trace fields?
46
47- **10**: Every significant claim references a span ID with a trace link. Judge reasoning is quoted verbatim (Eval Signal) or stack traces are cited (Error Signal). Trace deep-dives show what was fetched (system prompt content, tool parameters, retrieved documents, etc.).
48- **7–9**: Most claims are grounded. A few assertions are unsupported.
49- **4–6**: Some evidence cited, but key claims are asserted without trace references. Or evidence is generic ("the trace showed issues") rather than specific.
50- **1–3**: Little to no trace evidence. Report is mostly assertions.
51- **0**: No evidence cited at all.
52
53### 3. actionability (0–10)
54
55Are recommendations specific and immediately actionable, with before/after content from the traces?
56
57- **10**: Each recommendation specifies exactly what to change (system prompt text, tool parameter, routing logic), includes a before/after quote from the actual trace, and explains the causal link to the failure.
58- **7–9**: Most recommendations are specific. One or two are somewhat vague or lack before/after text.
59- **4–6**: Recommendations are present but generic (e.g., "improve the system prompt") without specific text changes or references to actual trace content.
60- **1–3**: Recommendations are vague advice (e.g., "improve retrieval quality") with no specifics.
61- **0**: No recommendations, or recommendations are irrelevant.
62
63### 4. completeness (0–10)
64
65Does the report cover all required sections and address the breadth of failure modes?
66
67- **10**: Report includes all expected sections (Signal Summary, Failure Taxonomy, Detailed Analysis per top failure mode, Prioritized Action Plan, Limitations). The Signal Summary matches the mode used (eval health table for Eval Signal, error type table for Error Signal, anomaly table for Generic). Covers all key failure modes from expected findings.
68- **7–9**: All major sections present. One failure mode underexplored or one section thin.
69- **4–6**: Core sections present but several are shallow. Failure taxonomy is incomplete (e.g., only 1–2 modes when 4+ were expected).
70- **1–3**: Large sections missing or severely truncated. Failure analysis is superficial.
71- **0**: Report is empty, extremely brief, or entirely off-topic.
72
73---
74
75## Output
76
77Return a JSON object with the following fields:
78- `diagnosis_accuracy`: integer 0–10
79- `evidence_grounding`: integer 0–10
80- `actionability`: integer 0–10
81- `completeness`: integer 0–10
82- `reasoning`: 2–4 sentences explaining the scores, citing specific strengths and weaknesses in the report