Setting the file. One moment.
Subchapter 7.2
references/discovery.mdMarkdown12 KBView on GitHub
Discovery should explain the real work, the agent that performs it, the systems around it, and the evidence available for new Tasks. Organize findings in the form that best fits the project. Do not force every finding or trace into a fixed template.
Identify the repository and revision, the agent or workflow under test, the existing eval suite, the available trace source, and the human’s goal. Keep different agent versions or materially different deployments separate. Mark unknown facts as unknown.
Read repository instructions, manifests, workspace configuration, and focused documentation. During read-only discovery, do not start services, install packages, open secret stores, or use credentials.
Start from each real way the agent is invoked. Follow reachable code through:
Use repository search to locate code, then read the complete relevant functions, callers, schemas, and focused tests. A README or directory listing can locate behavior, but it does not prove runtime behavior.
Keep notes in whatever structure makes relationships clear. Cite important claims with repository paths and symbols. Distinguish what source code proves, what a test proves, what a trace only suggests, and what remains uncertain.
For a large repository, divide review by real runtime area, system, or workflow. Do not divide work by arbitrary file counts. The main reviewer must join cross-area relationships and check the source behind consequential findings.
The Harness is the evaluated agent behavior: prompts, model loop, tools, session behavior, and adapter. The Environment contains the files, data, services, identity, permissions, network, clock, and mutable state around it.
Follow each agent capability across that boundary. For a search tool, inspect the registered tool name and arguments, how the Harness parses results, the service that answers the query, the returned record shape, permissions, ordering, pagination, errors, and any state changed by the call.
Map only details that can affect Task design or faithful execution:
Repository definitions are the primary source for intended schemas. Traces are useful evidence of real values, error shapes, latency, ordering, and behavior. Compare both when designing a simulated system.
Follow each scored path from instruction to reward:
Look for unstated checks, unchecked requirements, hidden preferences, leaked truth, false acceptance, false rejection, stale evidence, and infrastructure errors reported as agent failures. New Task directions must add a meaningful condition, capability, failure mode, or evidence requirement.
Use a trace source supplied or approved by the human. Record enough source context to cite findings later, such as project, agent version, filters, time range, export time, and stable conversation or trace identifiers.
A trace system can store one interaction as several related records. Names and relationships differ across products. Reconstruct the user-visible interaction and all relevant agent activity in time order. Include the initial request, later user turns, model responses, tool calls and results, retries, errors, state changes, and final outcome when they exist. Do not assume terms such as parent run or child run are available.
Start with a varied sample. Include different request types, tools, lengths, outcomes, permissions, agent versions, and failure shapes. Keep incomplete and failed interactions visible. They can reveal important dependencies and Task conditions.
Look for:
Preserve citations for important examples and aggregate claims. A note can use a trace ID, conversation ID, stable event ID, or another locator supported by the source. The structure of the note is flexible. It only needs enough context for another reviewer to understand and verify the claim.
Traces show observed behavior. They do not prove intended policy, complete business rules, or a correct answer. Use repository code, tests, source data, policies, final state, or human decisions as independent truth.
Group requests by underlying work and outcome, not by exact wording. Useful dimensions can include:
Name each cluster in plain language. Keep representative citations and note important variation inside the cluster. Counts can show frequency in the sample, but they do not establish business priority or population frequency unless the sample supports that claim.
Preserve rare but important requests, especially those involving safety, permissions, high-impact effects, or a distinctive capability. Do not force an unclear request into a cluster only to make the grouping complete.
Use the clusters to propose coverage, not to copy production conversations. Translate a cluster into a Task only after defining a controlled Environment, independent evidence, fairness boundary, and privacy-safe scenario.
When the trace set is large, review several small batches before expanding. Choose batches by stable identifiers, time windows, request clusters, agent versions, or outcomes. Keep batches non-overlapping unless overlap is an intentional review check.
If subagents are available and parallel review is useful:
Do not treat subagent agreement as truth. Retrieve more traces only to answer a named gap, test whether a pattern repeats, or cover a missing segment.
Trace review should help reconstruct the systems an eval may need to simulate. For each important tool or system, learn what evidence permits:
Do not infer a complete API contract from one successful call. Compare varied calls, failures, repository schemas, tests, and approved documentation. Record known limits so a simulated Environment does not claim unsupported fidelity.
Map every dependency needed for discovery, Task construction, model runs, or verification. This can include repositories, trace systems, documents, databases, APIs, applications, sandboxes, model providers, networks, VPNs, proxies, browser sessions, roles, and credentials.
Never read, print, copy, store, or ask the human to paste secret values. Do not
open .env files, credential stores, tokens, cookies, private keys, or secret
configuration. Inspect only safe code and sample configuration that names a
dependency or variable.
When access is needed, tell the human:
The human supplies access through the approved runtime mechanism. The agent uses only that mechanism and never records the value. After a safe access path works, record the reusable setup pattern in project World knowledge: dependency name, required role and scope, variable or profile name, setup command or documentation path, readiness check, and known limits. Never record the secret.
Before proposing Tasks, give the human a clear synthesis. Choose the structure that best communicates the project. Cover:
Protect private data. Summarize or replace identities and raw content when they are not needed to support the finding.