Case Shapes

Example cases in the corpus get renamed and churned, so this file names as few cases as possible — search the LangTracer suite by tag/field instead (the search_test_cases MCP tool, or export the suite and grep). The one stable pointer worth naming: applies-each-change-when-asked (in the baseline suite) for a well-built director conversation.

The schema (harness/schema.ts (opens in a new tab)) enforces the rules you must respect:

• Seeding lives in one slot, seed, whose mode is inline or replay — so the modes are mutually exclusive by construction.
• A case needs a conversation or a seed with mode: "replay" (which supplies the live turn from the trace).
• A case needs at least one executionScenario, processExpectation, or outcomeExpectation.
• buildExpectations is a forbidden key (fails loudly) — it was split into processExpectations / outcomeExpectations.

Behaviour / process cases (multi-turn director notes)

These test how the agent converses, not just what it builds: does it ask the right clarifying question, avoid re-asking, honour a mid-build correction, respect plan approval, batch bundled changes? They’re graded by processExpectations and are often build-only (no executionScenarios). Tag them behaviour and search the suite for that tag for patterns.

How multi-turn works

Mode is chosen automatically from conversation:

• Single-prompt (auto-approve): one user turn, no assistant turns — the prompt is sent and every confirmation is auto-approved. Plain build cases. Caveat: only confirmations are auto-approved — a genuine clarifying ask-user question is never answered, so the build hangs until the per-iteration timeout and reports as BUILD FAILED: Run timed out with no scored result (nothing to grade). If a prompt is vague enough that the agent is likely to ask a setup/topology question before building (an unspecified data source, delivery channel, or one-vs-two-workflow split), author it multi-turn with a [bracketed] director note in turn 1 that pre-answers those questions so the agent proceeds to build. A single-prompt build case only works when the prompt leaves nothing the agent must ask about. Real sourced prompts are frequently terse (“i want to create a webhook”, “convert a topic into a YouTube script”) and almost always trigger a clarifying question — default a terse sourced prompt to multi-turn with a director note that pre-answers the setup/topology it omits.
• Multi-turn: anything else. A user-proxy LLM plays the user — answers questions, audits the agent’s plan against your script, and sends follow-ups (capped by messageBudget).

Rules that trip people up:

1. assistant turns are reference only — never delivered to the builder. They tell the proxy the expected flow. The load-bearing content is your user turns + director notes; lead with those.
2. Don’t fabricate assistant “done” turns to sequence refinements. The proxy sees the whole script as soft context on every decision, so acks like “Got it / Updated” interleaved between user changes signal that work is already handled → it drops steps, reorders, or jumps to the end. Instead encode the ordering inside one director note and let the proxy send each change and wait for the build (this is exactly what applies-each-change-when-asked does — one bracketed, ordered list; a single opening reference ack is fine).
3. conversation[0] is sent to the builder raw. Never put a director note in the opening turn — it leaks verbatim into the build prompt. Notes belong only in the proxy-driven turns ([1]+).
4. The proxy defers credentials by default. A credential slot — whether on the standalone credential card (credentialRequests) or a workflow setup-wizard card (setupRequests, an entry with credentialType) — is auto-declined (“I’ll set them up later”) unless a director note governing that exact moment asks the user to engage — see “Engaging the credential-setup card” below. The workflow setup wizard is the one that matters for a normal build: live testing found the builder routes credential resolution through it during a workflow build, never through the standalone tool. The standalone tool is real and live-verified too (all three of manual/auto/skip), but only via a standalone credential-connect request with no build attached (e.g. “connect my Slack account now, before I build anything”) — see the tool’s own doc comment in utils/user-proxy/tools.ts for the captured shapes.

Director-note vocabulary ([bracketed] in a user turn)

Text inside [square brackets] is a director note — how the user behaves at that moment, never spoken verbatim. It overrides the proxy’s default (“always answer, inventing a plausible value”). The proxy’s system prompt (utils/user-proxy/prompts.ts (opens in a new tab)) recognises this vocabulary:

The text of a turn may be an array of strings (joined with newlines) so a long director note stays readable in JSON. A note governs only what it covers; elsewhere the proxy answers every question with a plausible placeholder (mocks make placeholders fine). Setup cards are filled via the wizard (apply_setup_wizard) or dismissed — never answered as questions.

Sanity rule: a behaviour case is only worth shipping if its processExpectations would fail on the misbehaviour you’re guarding against. If the agent could ignore your intent and still pass, the assertion is too loose.

Credential cases

By default a build sees no credentials: the harness pins every build thread’s credential view to the case’s declared set (empty unless declared), so concurrent cases and whatever else lives on the instance can never leak in.

Declared credentials are created for real before the build, the thread’s view is pinned to exactly that set, and they’re deleted after the run. Tokens are a placeholder by default; for a live token (verification runs for real instead of mocked) set the type’s EVAL_* env var — e.g. EVAL_SLACK_ACCESS_TOKEN, EVAL_NOTION_API_KEY, EVAL_GITHUB_ACCESS_TOKEN, EVAL_GMAIL_ACCESS_TOKEN, EVAL_TEAMS_ACCESS_TOKEN. name is optional (duplicates get a #2 suffix).

Only a closed set of types is valid — declaring anything else fails at case-load with a pointer to add a template. From credentials/seeder.ts (opens in a new tab): slackApi, notionApi, githubApi, gmailOAuth2, microsoftTeamsOAuth2Api, whatsAppTriggerApi, httpHeaderAuth, httpBasicAuth. Need another? Add a CredentialTemplate to seeder.ts (a defaultName, optional envVar, and buildData(token)); that extends SUPPORTED_CREDENTIAL_TYPES, which the schema validates against.

Engaging the credential-setup card (TRUST-349)

By default the proxy defers any credential slot — standalone card (credentialRequests) or workflow setup-wizard card (setupRequests, a credentialType entry) — with an empty/no-op response. This happens before the LLM is even called (confirmation-payload.ts‘s tryInfrastructureResponse for the standalone card; deterministic.ts‘s credential-only-request check for the wizard), so it’s the same deterministic behavior for every case that doesn’t opt in.

To make the simulated user engage instead, add a director note that names the credential/OAuth/connect vocabulary at the moment the card would appear (matched by hasCredentialEngagementDirection in utils/user-proxy/index.ts):

manual is one action that covers three cases, driven entirely by how many existingCredentials the resolved type’s request carries — no separate “create” action, the harness decides automatically:

Simplified examples of each (not committed in the repo — cases live in the LangTracer suite once pushed, per “Push to a lang-tracer suite” in the parent skill; these are illustrative, trimmed of the full calibrated wording):

Zero existing — the harness creates one:

Two existing — must pick the one the director note names:

Three existing — proves disambiguation isn’t hard-coded to “exactly two”:

All three are complete as written in one respect that earlier guidance got wrong: none of them asserts anything about the credential’s connection test. Both routes a credential takes into a build now authenticate, so an expectation acknowledging a connection-test failure reds every correct build — see “Credential validity” below.

The wire shapes (verified live against both tools — credentials.tool.ts‘s handleSetup state machine and workflows.tool.ts‘s setup-wizard equivalent):

Every eval credential holds a placeholder token unless you set the type’s EVAL_*_ACCESS_TOKEN env var (see “Credential cases” above). The parent umbrella (TRUST-348) requires “no stored provider credentials in any phase,” so a real token is the wrong fix. Instead the harness resolves the connection test as passing, the same way for both routes a credential can take into a build:

• Declared in credentials[] — models a credential the user already has connected, so it authenticates.
• Created by the simulated user on an engaged setup card — authenticates too, since the product won’t apply a card whose credential failed its test.

Do not assert a connection-test failure for either; such an assertion reds on every correct build. See “Credential validity” below for how to script a card-created credential that deliberately does not authenticate.

auto is reachable but inert — the product genuinely rebuilds the agent and returns needsBrowserSetup:true, but this harness has no Computer Use tools attached, so the conversation stalls afterward (expected, not a bug). Keep any case scripting auto a local smoke test, never part of the gated suite — it will time out.

Credential validity: a credential works by default

The product will not apply a setup card whose credential fails its connection test — the frontend’s isCredentialComplete returns isCredentialTestedOk, so Apply stays disabled until the test passes. “The user completed the setup card” therefore implies “the credential authenticates.” The same holds for a credential declared in credentials[]: it stands for one the user connected before the conversation started. Both carry a placeholder token that would fail for real, so the harness resolves the test as successful for both — otherwise every such case models a state a real user cannot reach.

Mechanically: the harness registers the case’s seeded credential ids on the thread up front, the proxy adds the ones it sets up mid-run (the types it lists in workingCredentialTypes), both go on the same bypassCredentialTest list of the eval allowlist endpoint, and the credential adapter resolves their test as successful without contacting the provider. The token is untouched — only the test result is synthesized, and only for credentials that case created. Nothing changes about “no stored provider credentials”.

To script a credential that does NOT authenticate, say so explicitly in the direction, naming which one:

The proxy then omits that type, its test runs for real, and it fails. Note what this models: not “the card was applied with a broken credential” (unreachable), but a credential that stopped authenticating — expired, revoked, scope changed. A card is currently the only way to get a failing credential: declaring one in credentials[] gives you a working one.

Non-vacuity for these cases is deterministic, not judged. A bypassed test is deliberately indistinguishable from a real pass in everything the agent sees — any hint would make it hedge, which is the behaviour such a case exists to rule out. So the judge cannot tell whether the bypass fired; check the run’s credential-test-bypassed proxy decision stat instead (it appears in the [proxy: ...] segment of the build log line). On a mixed card, the count is the assertion: two credentials with one scripted invalid should show exactly 1.

Seeded cases (start mid-conversation)

A seeded case puts prior history into the build thread before the live turn, so the eval drives only the turn under test. Use it to set up the situation you want to test — history up to some point, then a message that should trigger (or correct) a behaviour.

Pick the lightest mode that fits:

Both modes are implemented and wired (harness/conversation-seed.ts + harness/langsmith-seed.ts, threaded through the runner). The literals match lang-tracer’s metadata.seed verbatim, so nothing translates between the repos.

What the seed does — and does not — exercise

The seeded portion is replayed, not re-run. The prior messages are written into the thread as they stand (marked seeded: true so the judge and checks can tell them apart), and the workflows, data tables and agents they reference are created on the instance — so when the live turn runs, the agent sees the workspace the case says it should. Data tables are created schema-only, no rows: row values are the most sensitive thing a table holds, and they stay off the eval instance.

One thing the restore can’t reproduce: the sandbox is empty. The agent re-reads state from the database rather than editing source it “wrote”, so a seeded case is harder than the real turn was — never grade one on cost, turn count or messageBudget.

The consequence to internalise: nothing you assert can change what already happened in the seeded turns — the agent didn’t produce them, it’s only reacting to them now. So target your expectations at the live turn and everything built or said after the seed: what the agent does with the restored state, how it responds to the triggering message, what the workflow looks like after the correction. Asserting on the seeded prelude itself proves nothing.

Which mode — and when to avoid replay

Default to a synthetic case (an authored prompt + director script, or an inline seed prelude): it’s durable, carries no real user data, never expires, and you control the setup exactly. Reach for replay only when the misbehaviour genuinely needs real prior context that’s impractical to synthesize — a long accumulated thread, specific built workflows/tables — and the issue is in a later turn. (A turn-0 issue can’t be isolated by seeding: it lands inside the seed, so you’d bake the bug into the prelude.) Two standing costs keep it a last resort, not a default:

• Data handling. A replay stands someone’s own conversation up on the eval instance. The most sensitive parts are removed first — data-table row values are kept out and redacted from the history, node credentials stripped — but that pass can’t be assumed exhaustive, so treat what comes back as potentially personal data and follow your team’s data-handling policy. Cleanup deletes the thread, workflows and tables when the build finishes, though it is best-effort: a crashed run can leave them on the instance. Scrubbing the workflow into an inline seed is how the whole concern goes away for good.
• Transience. It depends on LangSmith trace retention (~14 days); the case stops running once the source trace ages out (tag it seeded, keep it out of full/pr).

If a plain prompt + director script can reproduce the situation, prefer that.

mode: "replay" — rebuild a thread for a local run

The case carries only the opaque thread id — no conversation content lands in the repo. At run time the harness pulls the thread’s runs from LangSmith, reconstructs the message log, recreates the workflows/tables it built, and splits at the last user message: everything before is the seed, that message is sent live. project defaults to instance-ai. Optional endpoint pins a US-tenant source host during the US→EU migration; optional liveTurnRunId pins which user turn goes live.

• Cross-workspace, zero config. A production thread can be replayed in a staging eval — the harness enumerates the workspaces your LANGSMITH_API_KEY can reach and finds the one holding the thread. It only reads the source; the eval writes its own traces/datasets to its own workspace. What it rebuilds still lands on the eval instance, so the data-handling note above applies.
• Continue past the live turn. Add a conversation to keep driving after the trace’s last message replays (first authored turn = expected assistant reply as proxy reference; subsequent user turns become follow-ups). Omit it to replay just the live turn and stop.
• Transient — don’t commit it, keep out of CI. LangSmith base-tier traces retain ~14 days and threads can be deleted or pruned, so a committed replay case goes dead the moment its trace disappears. Treat it as a local, throwaway reproduction: don’t commit it — run it to confirm the failure, then encode a durable synthetic case as the artifact. If you do keep one for a local run, tag it seeded, not full/pr; the resolver fails loudly when a trace has aged out.
• Multi-workflow limitation. Verification targets the primary created workflow (workflowsCreated[0]); if the live turn creates several, assert on the first or lean on processExpectations.
• Only agent-built workflows are restored. Reconstruction recreates workflows the agent built in-thread (a build event before the boundary) — not a workflow that pre-existed the conversation. So a debugging/diagnosis thread (“why does my HTTP node fail?”), where the agent only inspects or patches an existing workflow, seeds with no workflow to inspect. Reproduce the target workflow yourself (a synthetic case whose executionScenarios precondition builds the stand-in), or grade the live turn with processExpectations only.
• Can’t be pushed to a lang-tracer suite either. eval:langtracer-push refuses a replay case and lists it under skipped: — a suite is a durable home and this seed isn’t. Combined with the don’t-commit rule above, a replay case has no durable home by design — the durable artifact is always the synthetic case you derive from it. (An inline seed has no such problem: it pushes with the case and lives in the suite like any other.)

mode: "inline" — durable synthetic fixture

For a synthetic, sanitised seed pinned in git (never a real user’s conversation): author the prior messages, plus the artifacts they reference, in the case body (schema in harness/conversation-seed.ts (opens in a new tab) — messages + optional workflows, dataTables and agents). Real conversations belong in replay, which keeps their content out of the repo.

Two constraints that bite: an artifact id must be ≥8 characters (the id remap refuses shorter ones), and a seeded build-workflow tool call’s output.workflowId must match the seeded workflow’s id — otherwise the remap separates them and the agent can’t find the workflow it should act on. The same applies to a seeded build-agent result’s output.agentId.

The seed sits in the case body, not a sibling file, so it travels with the case whether it comes off disk, out of a LangTracer suite, or from a dispatched case body.

{role, text} shorthand — a prose prelude

When the prelude is just “what was discussed” — no tool calls, no workflows — write a message as {role, text} and the schema expands it to a full envelope:

text also takes an array of lines (newline-joined), like a conversation turn. The expansion stamps createdAt itself — ascending, in the past — so a shorthand message can’t order after the live turn. Shorthand and full envelopes mix freely in one array; a full envelope keeps its authored createdAt — unless the authored stamps don’t already ascend and sit in the past, in which case the whole sequence is restamped onto ascending pre-live slots. A future stamp would sort a seeded turn after the live turn, and a non-ascending sequence (a shorthand turn appended after later-stamped envelopes, say) would present the history in an order the graded transcript never had; restamping only the offending entry would reorder it against the array the transcript is graded from. A near-miss (say text: 123) is deliberately not expanded — it fails at load instead of becoming a message the transcript builder would silently drop.

agents — “here’s an agent you already built, now change it”

An n8n Agent is not a workflow, so it has its own slot: a project-scoped resource with a config plus authored skill bodies. Declare it and the restore creates it at its pinned id in the thread’s project, with its skills, before the live turn:

config and skills are the exact shapes GET …/agents/v2/:id/config and …/skills return, so you can author a seed from an agent you built by hand: build it on a dev instance, fetch both, scrub, paste. Things worth knowing:

• The thread is bound to the seeded agent, exactly as the conversation that built it would have left it, so the live turn’s build-agent call continues that agent directly. Without the binding the call is rejected (Unknown agentRef) and the model recovers from the agent id in its seeded history — measured at 3/3 runs recovering correctly, but it burns a turn, and the rejection message offers “create a new agent” as its first option.
• Names are not uniquified and leftovers are not evicted, unlike seeded workflows. An agent is addressed by id, so a same-named copy can’t misdirect the live turn; the name is also woven through skill prose, where a rename would rewrite instructions the case grades.
• Grade the agent, not a workflow. The agent’s config + skills render into the judge context, so outcomeExpectations cover them. Assert on the change the live turn makes — and assert the untouched parts survive, which is how you catch a rebuild-from-scratch masquerading as an edit.
• Skill bodies are prose — scrub them like conversation content, not like config. A skill carries far more free text than a workflow does: instructions and references[] are whole markdown documents, and they routinely name real teams, customers, internal tools, ticket queues, Slack channels and escalation contacts. Rewrite them into neutral equivalents (Acme Corp, #support) rather than trimming, and reread the full body — a workflow-shaped scan of names and ids will miss a paragraph.
• Skill ids must match: every config.skills[].id needs an entry in the skills map, or the agent renders with a dangling reference.
• Two agents can’t share an addressing key. Names are slugified to address the agent (Support Bot and support-bot both become support-bot), so a seed whose agent names differ only by case, spacing or punctuation is refused rather than silently dropping one from the registry.
• Credential ids are blanked on restore, the agent counterpart of stripping a seed workflow’s node credentials. An id from the instance you authored on addresses nothing here, so you can paste a fetched config as-is and the restore empties them. The seeded agent therefore arrives unconfigured for credentials — fine for grading its config and skills, but it is not runnable as seeded. Declare what the live turn should see in the case’s own credentials[].
• Requires the agents module. A seeded agent restore fails loudly (as a framework issue) on an instance where agents are disabled, rather than running the case unseeded.

folders — “look at the ODW folder”

A folder case grades how the agent finds the contents of a folder the user names (CONTEXT-86: the workflow list tool takes folderPath or folderId). Declare the folders and place the seed workflows in them:

Things worth knowing:

• Leave decoys at the root. A folder case is only informative when the folder’s contents differ from what a name search would return. Keep workflows outside the folder, and pick names a query filter for the folder name would not match.
• Grade the mechanism and the result. One expectation for the folderPath or folderId call, one for reporting exactly the folder’s workflows, one for not asking the user what the folder holds. The miss path (no folder exists) is a separate case: LangTracer case 699 grades it, and it needs no folders slot.
• Names are verbatim and leftovers are evicted, like seeded projects. The live turn names the folder, so the created name has to match exactly; a root folder of the same name that existed before the run started is deleted before the restore, with every workflow and subfolder in it. There is no seed marker on the name, so a same-named folder a human made on that instance goes too: use an eval instance, and a name a real project would not use. A previous iteration’s folder, created during the run, is left alone.
• Run it alone. A folder case and its miss-path sibling (case 699) share the project, so one invocation per case. With several iterations the previous folder can still be live for a few seconds while it is judged; keep the name distinctive.
• Nesting works. A folder’s own parentFolderId names another seed folder. Parents are created first. The folderPath for a nested folder is Parent/Child, so a folder name cannot contain /.
• Rules are checked at case load: ids of at least 8 characters, unique; every parentFolderId names a declared folder; no cycles; trimmed names that pass n8n’s folder-name rules; at most 20 folders.
• Needs a licensed instance (feat:folders) with N8N_INSTANCE_AI_FOLDER_EXPLORATION_ENABLED=true. An unlicensed instance fails the restore with a hint rather than running the case without its folder. Locally, after /rest/e2e/reset: PATCH /rest/e2e/feature {"feature":"feat:folders","enabled":true}.
• Keep it on disk for now. The LangTracer case-write API does not store seed.folders or workflows[].parentFolderId, so the push refuses the case until n8n-io/lang-tracer adds both.

Which opening shape — the agent is handed the workflow, or it has to find it

Two real conversations look the same in a case file but test different things, and picking wrong makes the case harder than reality.

Handed it. The user is looking at a workflow and opens the assistant: “why is this failing?”, “add error handling”. They never name it — the editor sends the workflow along as a resource reference and the agent resolves it by id. Declare that with attach on the opening turn:

The id is the one the seed declares; the harness swaps in the per-run id, so you track nothing. Only the opening turn may carry attach (an attachment is a hand-off, not something a user re-sends), and it must name a workflow the inline seed declares — both are refused at load rather than ignored.

The opening often has no text at all — the user opens the assistant on a workflow and waits for it to speak first. Keep "text": "" when that’s what happened; it’s the faithful shape, and openings with no user text jumped from 1% to 31% of the corpus when the editor hand-off shipped, so it is not an edge case. Note that an empty text is valid only alongside attach: the chat API rejects a message that is empty with nothing attached (“Either message or attachments must be provided”), so the two stand or fall together.

Has to find it. The user refers to the workflow in words: by name (“the Wait node in Generate leads failed”) or loosely (“the batch image workflow”). No attach — finding it is part of what the case tests. This also works when the seeded history already shows the agent building it, since the id is in its own record.

Get this wrong in the “handed it” direction — omit attach on a conversation that really had one — and the agent has to guess from prose that deliberately names nothing. It will list workflows and pick, or ask which one, and you will score a clarification failure the real user never hit.

Seeded faults must really fail

seed.priorRuns[].hints steer the mock for the staged run’s outbound calls and nothing else. The builder’s own runs are never mocked, and Code, Set, IF, Filter and Merge run for real. So:

• The fault must fail in n8n. A Code node that returns a plain object does not fail; n8n wraps it as one item. Return an array of plain values, reference a node that does not exist ($('Missing')), or throw.
• The fault fires before any external call. The builder’s rerun reaches a fixture host and dies there (DNS, 401), so a fault behind an HTTP node is never reached. Never grade the rerun itself; grade the fix with outcomeExpectations and the harness’s executionScenarios.
• An empty branch needs a Filter that keeps nothing. Split Out over an empty array still emits a placeholder item.
• Walk the item flow before asserting a count. A Slack node fed three items posts three times.

Before you ship a seeded case — three checks

1. The defect still bites. A seed whose workflow isn’t broken any more makes the case a silent no-op that passes forever.
2. Run it once with the seed removed. It must fail. Copy the case, delete seed, run both. If the no-seed copy also passes, the seed isn’t carrying the test. Name the copy so it doesn’t share a --filter substring with the real case (control-noseed-<slug> works; <slug>-noseed would match both).
3. The workflow’s skeleton is untouched if you scrubbed it from a real one: node types, versions and connection topology byte-identical, only string leaves moved.

Don’t grade a seeded case on cost, turn count, messageBudget, or “fixed it in one build”. A seeded thread starts with an empty sandbox, so the agent re-reads the workflow from the database and re-derives SDK source a real resumed session would still have on disk. The bias is harder than reality, so those numbers read worse for a reason that has nothing to do with the builder.

Context cases (long conversations, token cost, compaction)

A context case asks whether the agent still works once the conversation is long. Two questions, two mechanics.

The judge already has the token numbers

Every case’s judge context carries token ground truth, so no new field is needed to grade cost:

• Each transcript turn header reads Turn 2 (3 steps, 47446 tokens in [41220 cached], 812 tokens out).
• A Total block sums every LLM step across the whole build, main run and resumes, plus a cache read/write split and a Fixed overhead line — the opening step’s input, which is instructions and tool schemas plus one user message, so it reads as the floor cost of any turn on that instance. Use it to separate “the prompt got bigger” from “this conversation got expensive”.

So processExpectations may reference consumption directly. Write them as statements about behaviour the numbers reveal, not as thresholds:

Absolute token budgets rot: a prompt edit moves every number and the case goes red for a reason that has nothing to do with the agent. Grade the shape — what got re-read, what came from cache, which turn is the expensive one.

Two limits to know before you write a cost expectation:

• It needs N8N_INSTANCE_AI_RUN_DEBUG_ENABLED=true on the instance under test. Without it the blocks render (no run debug captured) and the expectation is unjudgeable — it fails or passes at random.
• The capture hooks only the orchestrator’s own stream. A workflow build runs inside that loop, so a workflow case’s numbers are its real cost. A delegated Agent build (the build-agent tool) runs on a separate stream with no hooks, so its tokens are missing from every block. Don’t write a cost expectation on an data/agents/ case.

requiresMemoryCompaction — grade the post-compaction window

Observational memory compacts a long thread: an Observer summarises it into observation rows, the summary is rendered into the system prompt, and the early turns are masked out of the agent’s window. A case that tests whether a decision survives that has a setup problem — production compacts at 30k tokens of visible message content, which is a conversation too long to hand-author.

Set the flag and the harness handles it:

Two things happen, both per-thread, so no other case in the run is affected:

1. The harness sends a low observer threshold with every turn of that thread only: 1,000 tokens of visible message content, the request field’s floor. The seed plus the live turns must cross it, or the observer never runs — a short seed does not test memory, it reports not judged (below). Size the seed generously; a few real turns of tool output cross 1,000 tokens easily. The Observer’s prompt, masking and cursor logic are the production ones at any threshold.
2. After the build, the harness checks the premise actually held. If it did not, every expectation on the case is reported not judged — an incomplete verdict, excluded from scoring, not a red.

The premise check is the point of the flag. Uncompacted, the raw early turns are still in the window, so the agent answers off them and every expectation passes for free — a green that tests nothing.

The evidence is structural. GET /rest/instance-ai/eval/threads/:threadId/memory returns the observation rows and the compaction cursor from their own tables. A cursor means the observer ran and everything up to lastObservedMessageId is masked out; the rows are what replaced it. Missing either, the case had nothing to test and is reported not judged. Nothing reads the system prompt, so this needs no debug flag and a prompt or SDK rename cannot quietly turn “never compacted” into the answer.

The judge never sees the premise check — it grades the conversation, not whether the harness configured the scenario. A misconfigured lane must not read as a quality regression.

Grade the summary, not just the reply

The judge does see the observation rows, as ground truth:

So write expectations at both layers:

The first grades the memory system, the second grades the agent. Grading only the reply is weaker than it looks: the agent can answer correctly from the unmasked tail, or guess well, and the case goes green while memory dropped the detail. The [MARKER] is the Observer’s own priority label, so “retained as CRITICAL” is checkable too.

Writing the conversation

The shape that makes a compaction case real:

1. State the anchors once, in a user turn near the start of the seed, and never restate them. Anything the agent repeats later lands after the cursor and survives for the wrong reason.
2. Make live turn 1 off-topic. Its answer must not re-surface the anchors into the post-cursor window. Compaction runs at the end of it.
3. Ask for the anchors back in live turn 2. That turn runs against the compacted context.
4. Force the antecedent — a lost anchor must produce a confident wrong answer, not a vague one. “Posting goes through HTTP Request, not the native Slack node” works because the Slack node is the obvious guess. “The threshold is 4700” works because a guess reaches for 5000. See negative expectations need a forced antecedent.

Tag these context-consumption and put them in a context dataset, so a tier that needs run debug enabled can select them.