Evaluation Payload

Top-level fields

Valid (evaluation_type, output_type) pairs: (hog, boolean), (llm_judge, boolean), (hog, numeric), (llm_judge, numeric), (sentiment, sentiment). The output type cannot change after creation. Numeric evaluations require the llm-analytics-numeric-evaluations rollout flag for the project. Existing numeric evaluations remain editable when it is off.

target and target_config

• "target": "generation" runs once for each matching generation and uses an empty target_config.

• "target": "trace" runs once for the whole trace. target_config picks the settle strategy:

  • { "strategy": "fixed_window", "window_seconds": 10..7200 } waits a fixed delay after the first matching generation (defaults to 1800). A config without a strategy key means this.
  • { "strategy": "inactivity", "quiet_period_seconds": 10..1800, "max_age_seconds": 60..7200 } evaluates once no new activity arrived for the quiet period (defaults to 300), capped at max_age_seconds (defaults to 7200, must be at least the quiet period) from the first one.

• "target": "session" runs once for the whole $ai_session_id session, after the session settles. target_config picks the settle strategy with session-sized bounds:

  • { "strategy": "inactivity", "quiet_period_seconds": 10..86400, "max_age_seconds": 60..604800 } evaluates once no new session activity arrived for the quiet period (defaults to 3600), capped at max_age_seconds (defaults to 86400, must be at least the quiet period) from the first one. This is the default strategy for a session target.
  • { "strategy": "fixed_window", "window_seconds": 10..604800 } waits a fixed delay after the first matching generation (defaults to 1800).
  • A session evaluation only fires for events that carry $ai_session_id. Producers set it on every generation or on none, so an SDK that never sets it will never trigger a session evaluation. $ai_session_id is not $session_id, PostHog’s product-analytics session; the two are unrelated.
  • A session is evaluated at most once per evaluation, for as long as the completed run stays inside Temporal’s retention window. A session that resumes long after being evaluated may be evaluated again.

• Conditions always match generation properties. For a trace or session target, that matching generation schedules the eventual whole-unit evaluation.

• Sentiment evaluations cannot use the trace or session target.

evaluation_config by type

llm_judge

hog

bytecode is compiled and written by the server on save — never pass it. Send only source.

sentiment

output_config

boolean output

allows_na: true lets the evaluator return N/A in addition to true/false. N/A differs from an execution error that skips a run. true_is_failure: true treats a raw true result as a failure and a raw false result as a pass. Leave it false for the default mapping.

numeric output

Return a finite number, not a boolean or numeric string. Optional min and max are inclusive; an out-of-bounds score skips that run. step guides scoring without rounding results. allows_na defaults to false; set it to true to allow null from Hog. Set passing_rule to {"operator": "gte", "threshold": 7} (at least 7) or use lte (at most). Without a passing rule, scores remain ungraded and reports are unavailable. Changing the rule reinterprets historical scores; previously generated reports stay unchanged.

sentiment output

Empty object: {}.

model_configuration (llm_judge only)

provider and model are required. Pin provider_key_id to run on one specific key, or set it to null to run on the team’s active key — both save a valid draft. To enable an unpinned eval, the active key must be ok and use the same provider (llma-evaluation-config-get shows it). A disabled draft may be saved without a usable key, but it cannot be tested or enabled until a key can be resolved.

conditions

A list of condition sets. OR between sets, AND within a set’s properties. Omitting conditions (or an empty list) matches every $ai_generation. A generation target evaluates each match; a trace target evaluates each matching trace once.

Each property filter: { "key": "...", "type": "event" | "person", "operator": "...", "value": ... }. Common operators: exact, is_not, icontains, not_icontains, regex, gt, lt, is_set, is_not_set. There is no time/date field inside conditions — scope by event timestamp upstream if needed, and sample volume with rollout_percentage.

Full examples

Hog (no provider key required)

The same Hog source works for a whole trace. Change only the target fields:

Or evaluate when the trace goes quiet instead of after a fixed delay:

The same Hog source also works for a whole session, once no new activity has arrived for the quiet period:

New Hog source should use the shared evaluation_events and target globals. Top-level generation globals such as input, output, properties, and event, plus the trace-only events and trace globals, remain available for compatibility with saved evaluators. Session evaluations do not carry the legacy events / trace globals: session Hog source only receives evaluation_events and target. item.input_text and item.output_text are best-effort readable projections of common provider payloads; use raw item.input and item.output when exact structure matters.

LLM judge

provider_key_id: null runs on the team’s active key and always saves a valid draft. Pin it to an ok-state key uuid from llma-provider-key-list to run on one specific key. To enable an unpinned eval, the team’s active key must be ok and its provider openai.