Setting the file. One moment.
Skill 06 · Agent Observability Experiment Bootstrap
Subchapter 6.5
references/python/evaluator-styles/remote.mdMarkdown2 KBView on GitHub
--evaluator-style remote (LLM-as-Judge running server-side)RemoteEvaluator instances that point at a judge configured in the Datadog UI. The judge LLM call runs on Datadog’s side, not in the user’s experiment process — useful when the judge is shared across experiments or has its own quota / model selection.
from ddtrace.llmobs import RemoteEvaluator
# Create the judge in Datadog UI first: Agent Observability → Evaluations → New Evaluator
quality_judge = RemoteEvaluator(eval_name="<name-from-datadog-ui>")
# Optional: customize the payload the judge receives
custom_judge = RemoteEvaluator(
eval_name="<name>",
transform_fn=lambda ctx: {
"question": ctx.input_data.get("question"),
"answer": ctx.output_data,
"reference": ctx.expected_output,
},
)The judge must exist in the Datadog UI before the experiment runs. Emit a comment in the generated file telling the user:
# Before running this experiment:
# 1. Open Datadog → Agent Observability → Evaluations → Custom evaluators
# 2. Create an LLM-as-a-Judge evaluator. Note the eval_name you give it.
# 3. Paste that name into RemoteEvaluator(eval_name="...") below.For one-off rubrics tied to a single experiment, inline LLMJudge (under the function style) is simpler. See references/python/evaluator-styles/function.md.
If the user doesn’t have a judge configured in Datadog yet and won’t set one up, fall back to function style with an LLMJudge placeholder — at least the experiment runs end-to-end.