Setting the file. One moment.
Skill 06 · Agent Observability Experiment Bootstrap
Subchapter 6.3
references/python/evaluator-styles/class.mdMarkdown2 KBView on GitHub
--evaluator-style class (advanced — for evaluators that need state or async I/O)BaseEvaluator subclasses with . Always return — never a bare value. State-bearing evaluators usually have richer reasoning to surface anyway.
evaluate(self, context: EvaluatorContext) -> EvaluatorResultEvaluatorResultfrom ddtrace.llmobs import BaseEvaluator, EvaluatorContext, EvaluatorResult
class FaithfulnessJudge(BaseEvaluator):
def __init__(self):
super().__init__(name="faithfulness")
# TODO(user): initialize any client or state here
def evaluate(self, context: EvaluatorContext) -> EvaluatorResult:
# context exposes: input_data, output_data, expected_output, metadata
# TODO(user): replace placeholder logic with your faithfulness check
passed = context.output_data is not None
return EvaluatorResult(
value=1.0 if passed else 0.0,
reasoning="placeholder — replace with your faithfulness rubric",
assessment="pass" if passed else "fail",
metadata={"evaluator_version": "v1"},
)super().__init__(name=...) in __init__. The name is the column header in the Datadog Experiments UI.evaluate() runs in the experiment’s worker pool. Do NOT mutate self from evaluate() (thread safety) — state set in __init__ should be read-only thereafter.asyncio.run(...) inside evaluate() rather than making evaluate itself async. Keeps the experiment runner sync.If the evaluator is a one-line check (exact_match, length_under_500), use function style — the class boilerplate adds noise. See references/python/evaluator-styles/function.md.