Sourcing Cases

Replaying a thread at run time (seed.mode: "replay") is a secondary mode: a local check that the failure is real, not something to commit. See Replaying a thread first.

Two connections make either possible:

• LangTracer — discover. It analyses Instance AI conversations and clusters them into capability-gap themes (“what fails, at scale”). Use it to find high-frequency real failures worth encoding, instead of guessing a failure mode.
• LangSmith — verify. Eval runs and production conversations land in LangSmith as traces. When a finding or a flaky result is ambiguous, read the trace to confirm what happened — which tool calls fired, and their payloads.

Connect the LangTracer MCP

The hosted instance exposes a streamable-HTTP MCP at <base>/api/mcp, authed with an lt_… bearer (kept in .env.local: LANGTRACER_API_KEY; the hosted base is the LANGTRACER_URL value). Register it the way your harness registers MCP servers — e.g. in Claude Code:

--scope local keeps the key out of committed config. MCP servers load at session start, so reconnect to pick it up. (The LangSmith MCP is usually already connected.)

The MCP returns ids; the driver needs pages. Every thread, cluster, and case you cite while sourcing has a web URL off the same base — <base>/conversations/<threadId>, <base>/clusters/<id>, <base>/test-cases/<id> — so cite them as links, not bare ids. Full table in Share links, never bare ids.

Discover → verify → encode

1. Scan cluster themes — list_cluster_runs / get_latest_cluster_run return capability-gap themes with a label, summary, and mechanism: the real, recurring failure modes.
2. Pull the conversations — list_conversations (e.g. verdict:"bad", analyzed:"yes") → get_conversation (raw trace) + get_conversation_analysis (findings). get_linear_ticket_context links a thread to its ticket. To target execution failures, add funnelDrop:"05" (built + launched an execution that never succeeded). Caveat: the funnel drop is a conversion signal, not a build-quality one — most funnelDrop:"05" threads are healthy builds that correctly routed to credential setup, or the user simply abandoned, so the drop alone tells you nothing about the build. Pair it with verdict:"bad" to filter down to threads where the analyser actually flagged a build/execution defect (in one sample: 131 raw drops → 18 once verdict:"bad" was added).
3. Verify before trusting a finding. The analyser keys off tool-call spans (was build-workflow called on this turn?), which it reads reliably — but a content-dependent claim (“invented ID”, “missing node”) can be wrong when it couldn’t see the built workflow. Confirm against the raw trace before building a case around it.
4. Optionally record the selection as an observation. Offer to save why this thread was selected and what the developer observed. This is a best-effort LangTracer note, not a gate: skip it when the developer declines or has not stated a preference, and continue if the write fails. See Optional: record the selection as an observation.
5. Encode a durable synthetic case — turn the confirmed failure into an authored case (SKILL.md, case-shapes.md). The failure mode is the anchor; the conversation is yours to write, in the user’s voice. Write down the precondition, not just the failure mode — the state the thread was in when the call went wrong, which is usually narrower than the theme label suggests and is often stated outright in the assistant text just before the failing call. Get this wrong and the case grades green because it never set the situation up; see First reproduce, then reclassify.
6. Push it to a curated suite (don’t commit the JSON) with eval:langtracer-push — see Push to a lang-tracer suite. An inline seed rides along with the case. Exception: a replay case is refused and listed under skipped: — it’s reconstructed from a trace at run time, so it dies when that trace is pruned and has no durable home; that’s exactly why step 5 turns the confirmed failure into a durable synthetic case.

Optional: record the selection as an observation

After the raw trace confirms the finding, offer the developer one optional record before drafting:

• Why did you choose this thread? Capture why it is useful eval coverage, such as impact, recurrence, reproducibility, or a gap in the current suite.
• What did you observe? Capture the concrete behavior, the expected behavior, and the run or turn that contains the strongest evidence.

If the developer opts in, call LangTracer’s create_observation tool. Map the record like this:

• sourceThreadId: the selected imported thread.
• sourceRunId and anchorRunId: the evidence run, when the finding is narrower than the whole conversation.
• kind: capability_gap for a confirmed failure, regression for successful behavior worth protecting, or binary_check when that is the intended output.
• description: why the developer selected this thread.
• openCodeNote: what the developer observed, in reviewer voice.
• expectedBehavior, failurePattern, and proposedCheck: the rule, the observed deviation, and the check the eval should enforce.
• severity and failureClass: include them only when the evidence supports the classification.
• isTestCaseCandidate: true because the developer selected it for eval authoring.

This record is optional. Do not pause an autonomous run to request it when the driver has not already opted in. In checkpoint mode, offer it once with the selection proposal. A declined offer, unavailable MCP tool, or failed write must not stop the eval workflow. Report a successful observation by id with the source conversation link; otherwise omit it from the handoff.

Calibration can sharpen the note. If an observation exists, use update_observation only for mutable details such as the expected behavior, failure pattern, proposed check, severity, failure class, or reviewer note. Its kind and source provenance are immutable, so classify the source evidence before creation. Both observation tools are LangTracer-local. Do not create or update LangSmith feedback for this record.

Was the workflow handed over? (sourcing an attach opening)

Whether the opening turn carries attach is a fact about the thread, not a judgement call — and guessing wrong makes the case harder than reality. Read it two ways, depending on when the thread was imported:

• Recorded. Turn 0 of get_conversation carries resourceAttachments: [{ "type": "workflow", "id": "…" }]. That’s the editor hand-off verbatim. Only kind and ids are stored, never the workflow’s name.
• Inferred, for threads imported before n8n traced it. The tell is an opening turn whose userMessage is empty — the editor’s context block is stripped before the trace, so a hand-off where the user typed nothing leaves a blank record. Corroborate with an early workflows[get] on a workflow the user never named.

Then use the id to recover the workflow itself: find the tool call whose input carries that workflowId (usually workflows[get], full: true) — its output is the workflow, nodes and connections and parameters, and that is what you scrub into seed.workflows. The id alone is only a join key; it addresses the user’s own instance, so it is worthless by itself.

Two things to expect:

• The agent may never have read it. Then you have an id and no content, and the workflow is yours to write — keep the topology plausible for the complaint.
• The opening often has no text at all. Keep it that way; see the empty-opening note in case-shapes.md.

Scrubbing a real workflow into a synthetic seed

A seeded case carries a workflow inside it, and that file is committed and runs for as long as the case lives. So the workflow has to be written, not copied: read the real one, then author a stand-in that keeps its technical shape and none of its personal data. One careful pass, with a person reading it, and the committed fixture is clean from the start — nothing further down the line will clean it up for you.

What to look for

Identifying data hides in more places than the obvious ones:

• Parameters — email addresses, phone numbers, channel names, folder paths, spreadsheet / document / table ids, internal hostnames, and any token someone pasted into a header or a query string. A token sitting in a parameter is not a credential, so nothing strips it for you.
• Free text inside the workflow — sticky notes and node notes, string literals in a Code node, sample rows in a Set node. Sample rows are often real records.
• The workflow’s own name, which often carries a company or a person (“Acme lead sync”, “Chase invoices — Dana”). This repo is public, so a real customer’s name must not appear anywhere in a case; use a placeholder like Acme Corp.
• The earlier messages you seed alongside it — someone’s own wording, their name, their employer, links they pasted. Write those turns yourself, in a user’s voice.
• Recorded tool calls, if you keep any. Measured on a real thread, this is where the leaks actually were — the workflow was clean, while the outputs of workflows[list] / get / get-as-code and data-tables[list] carried a live credential id, the project id, the person’s entire workflow inventory, table schemas down to every column name, and a webhook id. Each toolCallId also carries the source run id. Read the messages: a regex pass over that seed caught one category out of about eight.

Provenance is the one thing worth keeping: note the source thread id in the case’s description so anyone can find the original later. An id points at the conversation without carrying any of it.

Which version of the workflow to start from

Take the workflow as it stood when the turn you’re testing began, not the state it ended the thread in.

• The assistant built it. list_conversation_workflow_builds lists each build with a seq and turnRunId, so you can find the one just before your turn, and get_conversation_workflow_build returns it. Its contentStatus says what you actually get: stored-json is ready to work with; compilable-source means only the builder’s code was kept, which parseSeedWorkflowCode (opens in a new tab) turns back into a workflow; unrecoverable and unrecoverable-intermediate mean there is nothing to read. About one build in five has no stored JSON, so don’t assume. And don’t reach for a later build instead — that one is the workflow after the fix, so the case would test the repair rather than the problem.
• The person already had it. When the editor hands a workflow to the assistant, the assistant looks it up by id — so the workflow you want is the result of the workflows[get] call whose input mentions that id.
• Nothing was stored. Older threads pre-date this being kept at all, and the read says so in a note rather than returning an empty list. An empty answer never means “there was no workflow”. Two ways out: replay the thread locally and take the workflow the harness rebuilds from the trace (see below), or write the stand-in yourself — the simplest set of nodes that can still show the problem. Hand-authored, there’s no original to compare against, so the before-you-ship checks are the only safety net left.

What to keep, and what to replace

Keep the technical shape, since that is what the case tests. Replace the values that point at a real person, company or account.

Swap like for like — a URL stays a URL, an id keeps the same look and length. That’s not only about readability:

• Loading a seed checks every node for id, name, type, a numeric typeVersion, a two-number position and a parameters object, and rejects the whole seed if one is missing. So don’t delete fields while you’re in there.
• Ids get swapped for fresh ones by a plain search-and-replace across the seed. Anything under 8 characters is rejected outright, and a short id that squeaks through is exactly how you end up rewriting unrelated text that happens to contain it.

On one node that comes out as: the host, the id and the key change; the expression, the reference to another node and the parameter names don’t.

Trim as you go. The seed is one field on the case, with a 256KB ceiling — load_skill bodies alone can account for most of it. Trimming has a rule of its own: keep the shape here too. Filtering a list output changes what the agent believes exists, and a { "note": "…" } where a result belongs invents a shape no tool ever returns. Shortening the text inside a block is fine; to get rid of a call, drop the whole tool-call block. If you keep a build-workflow call, its output.workflowId has to match a workflow the seed declares — see case-shapes.md.

Three easy mistakes

Renaming a node. A node’s name is how the rest of the workflow points at it — the wiring is keyed by name, and expressions call it by name ($('Old Name')). It can also turn up in recorded builder code and in the prose. Miss one reference and the workflow quietly stops working, which is the kind of over-cleaning that leaves the case testing nothing. Node names are rarely identifying on their own, so the default is to leave them be. If you do rename one, grep -c 'Old Name' <case>.json should come back 0. (Workflow names are a different story: each run gets its own copy with a [seed …] suffix, and a seed declaring two workflows with the same name is refused, because there’s no way to tell which one a mention refers to.)

Replacing the value the case is about. In http-keep-generic-credential-unknown-service, the host queue.fal.run matters precisely because n8n has no built-in credential for it — swap in a well-known host and the case tests the opposite thing. Same with the invented column names in flags-unverified-sql-identifiers. Expectations quote values too (“posts to #growth“, a particular model name), so a replacement has to be made in both places, or the expectation loosened the way others already are for values that don’t matter (“an unset or placeholder value is acceptable — it’s something to fill in at setup, not a build mistake”). Knowing what the case asserts is part of scrubbing it, which is why this is a person’s job and not a script’s.

Tidying up a node’s version. Split In Batches v2 lists its outputs as ['loop', 'done']; v3 lists them as ['done', 'loop']. Same two outputs, opposite meaning — so bumping the version quietly sends the loop the other way while the wiring still looks untouched. Leave versions as the conversation had them.

Check that only values changed

Compare the JSON you started from with the workflow you put in the case:

leaves prints one line per value in the file, so the diff is a plain list of what you changed. Every line should be a value you meant to replace. A line mentioning type, typeVersion, connections or a behaviour setting means the shape moved, not just the values; a line on one side only means something was added or dropped. Keys that aren’t part of the graph (settings, meta, pinData) are ignored, so the original compares cleanly against the trimmed version.

This is the third of Before you ship a seeded case. The other two — the problem is still there, and the case fails without the seed — are what catch a clean-up that kept the shape but lost the point.

What the harness already handles

You don’t need to do these by hand:

• Credential references on nodes resolve against the case’s credentials[] as the seed loads. A reference is kept only when exactly one seeded credential has that type and display name; any other reference is dropped. On a published: true workflow an unresolved reference fails the restore instead.
• Data tables are columns only. The seed format has no place for rows, and a rows key is rejected rather than quietly removed — so table contents can’t come along by accident.
• A seed workflow is only id, name, nodes, connections and an optional published flag. published: true activates the workflow on restore, the way the user’s publish left it. Pinned example data, instance metadata and settings never travel.
• Ids and workflow names are per run. Ids are replaced with fresh ones and names get a [seed <8hex>] suffix, with mentions updated in the prose — though not inside node definitions or recorded tool calls.

Everything else in the seed is yours to check.

Replaying a thread first (the secondary mode)

Scrubbing takes work, and sometimes you want to know the failure is real before you put that work in. That’s what seed.mode: "replay" is for: give it a thread id and the harness rebuilds the conversation from its trace at run time, then drives the turn you care about. Nothing about the thread lands in the repo — the case holds only the id.

Treat it as a local check, not a case. The trace it depends on ages out in about two weeks, so a committed replay case stops working; and it stands up someone’s real conversation on the eval instance, which is exactly what scrubbing exists to avoid. So run it, confirm the failure, then scrub the workflow it hands you into a seeded case that will still be there next quarter. Details and limits in case-shapes.md.

Sourcing a regression baseline (successful builds)

The discover→verify→encode flow above hunts failures. The complementary need — a broad regression baseline of things that already work — is sourced the opposite way: from conversations where a workflow was built and executed without errors. Use list_conversations with the conversion-funnel filter:

• funnelStep: '03' = built a workflow, '04' = launched an execution, '05' = execution succeeded, '06' = published (each step is a strict subset — also reached every earlier step). For “built + executed cleanly” cases, '05' is the signal; add language: 'eng' to keep prompts English.
• The result is large — it spills to a file. Triage by first prompt: jq -r '.data[] | "\(.threadId)\t\(.firstUserPrompt[0:180])"'. Skip “The execution failed…” debugging threads (they didn’t cleanly build) and off-topic app-build requests.
• Author a build case per selected thread (prompt in the user’s voice, grounded in the real firstUserPrompt; note the source thread id in description). These are regression-kind cases whose value is coverage of a working capability, not a currently-red gap — you rarely need get_conversation when the first prompt already specifies the build. Terse prompts almost always need a multi-turn director note (see case-shapes.md).

Two practical notes on get_conversation_analysis

• It hands you draft cases. The response’s aiAnalysis.structured.extractedCases[] are pre-drafted candidates — each with expectedBehavior, proposedCheck, and failurePattern that map almost 1:1 onto outcomeExpectations / processExpectations. Start from these rather than a blank case (still verify against the raw trace per step 3, and rewrite the prompt in the user’s voice). verdict and findings sit alongside them.
• The payload is large — tens of thousands of characters, enough to exceed a tool’s token cap and be spilled to a file. The useful part is aiAnalysis.structured; jq into that (or into .extractedCases) rather than reading the whole blob.