Config Eval Playbook

Config fields (create/update): name, startNodeName, endNodeName, dataTableId, metrics.

startNodeName must be a node with an incoming connection — the first node the trigger feeds into, never the trigger itself. An eval run replaces the trigger with a dataset-driven one, so naming the trigger as the start node fails to compile. For a single-agent workflow, startNodeName and endNodeName are usually the same agent node.

create, update, and delete show an approval card automatically. Call the tool and act on the result; do not ask for chat approval beforehand.

Metric Fields

actualAnswer, expectedAnswer, and userQuery are expressions and must begin with = (e.g. ={{ $json.output }}). Without the = the value is stored as literal text and the judge scores {{ $json.output }} verbatim instead of the resolved output. Only omit = for a genuinely fixed constant string.

Worked Example — correctness eval on an agent workflow

1. Ensure a dataset exists with an input column and a ground-truth column:

   Copy

   data-tables(action="list")
   // if none fits:
   data-tables(action="create", name="Support agent eval dataset",
     columns=[{ name: "input", type: "string" },
              { name: "expected_output", type: "string" }])
   // then seed rows with data-tables insert

2. Create the config eval, linking the dataset by id:

   Copy

   eval-config(
     action="create",
     workflowId="<wf_id>",
     name="Support agent correctness",
     startNodeName="AI Agent",        // first node after the trigger — not the trigger itself
     endNodeName="AI Agent",
     dataTableId="<dt_id>",
     metrics=[{
       name: "Correctness",
       preset: "correctness",
       credentialId: "<cred_id>",   // provider is derived from this credential
       model: "<model_id>",
       actualAnswer: "={{ $json.output }}",
       expectedAnswer: "={{ $json.expected_output }}"
     }]
   )

3. Report: evaluation name, workflow, start/end nodes, dataset name + id, metric.

Worked Example — helpfulness eval (no ground truth)

Use helpfulness when there is no single correct answer. The dataset needs only the input column; no ground-truth column is required.

Changing or Removing

• To change metrics, nodes, or the dataset, use update with the configId from list/get. update replaces the full config — resend every field you want to keep, not only the changed ones.
• To read current state before editing, use get with the configId.
• To remove an eval, use delete with the configId.

Boundary Reminder

This tool never touches the canvas. Do not add EvaluationTrigger or Evaluation nodes; config evals are attached through the evaluation-config API only. Build and seed datasets exclusively through the data-tables tool.