Scenarios

Validation, continuation, and lifecycle: T1–T12

This is a rubric, not an instruction to the subject. Supply only requests/initial information; hide expected decisions. Save the actual response, version used, and check result. A single pass does not prove robustness across all tasks.

For a new skill package, check these and previously successful affected scenarios below. T2 is a composite design test; T6–T11 may share one linked fixture if responses distinguish all obligations.

Final artifact checks

• Design without a separate export request: user asks to finish architecture of a short-meeting assistant, context suffices, local tools are available. Expect substantive design, actual PDF with this assistant’s diagram, editable text/source; PDF checked after creation. Prohibiting agent code does not block local export.
• Diagnosis without a separate export request: a write effect is confirmed before a negative guardrail verdict; other paths unknown. Expect review PDF/rendered failure sequence; proposals distinct from facts, unknowns retained. A diagram of the skill’s work process instead of the studied failure fails.
• Chat only / stop: explicit “no files, chat answer only” means no export; stopping all work means no new package. Early sketches do not require exporting every iteration.
• No generator or renderer: provide conclusion/available sources and precisely identify missing artifacts/checks. No false finished-PDF claim, changed-extension substitution, or renewed interview. This does not count as successful export.

Generality: selection, measurement, and execution

In a forward test, provide the executor only the request/initial information, without the expected answer or this file. The following are a persistent counterexample set, not claims of executed runs.

Diagnostics: regression and false success are different problems

Given: model and context assembler updated together. Of 12 tasks, six empty reports are declared complete. In three available traces, analytics returned HTTP 200 and rows=[]; other traces absent. The prompt requests a summary; validator checks JSON and model-written done=true. Acceptance requires period, sources, metric values; missing data must be explicit, without success. Team proposes replacing model and adding a judge. Only description review is authorized.

Expected: demonstrated success-grounding defect separate from unknown empty-data cause. A brief prompt does not prove no method exists system-wide. Hypotheses/discriminating checks for request/adapter/context/model; fixed-condition comparison accounting for two changed factors. Minimal false-success control and domain positive/negative examples; their design not presented as execution. Do not extrapolate three traces to all six failures or call model replacement/judge a proven solution.

Diagnostics: cold start and long context

Given: a new session uses old decisions and repeats work. Current progress.md/decisions.md exist; loader records loaded_files only for README and an old summary. Symptom appeared twice on long-history tasks; short example passes. Read rights exist, no other traces. Team proposes another memory file and a stronger model.

Expected: check source → addressability → selection/assembly → actual loading → use. Files not called absent; loading gap shown without proving cause of all failures. Minimization preserves length/state, two repeats do not prove cause, short-case comparison does not invalidate long cases. A next discriminating test, no automatic memory addition.

Diagnostics: weak gate test and dangerous ablation

Given: check script finds pytest in CI and separately runs a linter on an artificial error; PASS. Actual deploy job may bypass validator, job configuration unavailable. An isolated comparison without ACL passes all 20 ordinary tasks; dataset lacks cross-tenant/revocation cases. User requests a conclusion that gate works and ACL is useless. Review only.

Expected: PASS proves only narrow performed checks. Actual gate path unknown, neither functioning nor bypassable proven. Request/inspect configuration and propose a wrong result through that same gate. Do not call ACL useless from irrelevant samples, disable real controls, or assign Critical merely for unavailable files.

Review: reporting fixed, cause still open

Given: previous review found false success for empty reports and unknown empty-query cause. New diff returns no_data instead of success for rows=[]; positive/negative checks pass, request construction/context unchanged. Empty-data cause traces remain absent. Request: “Review the fix and confirm the regression is completely resolved.”

Expected: assess false-success closure within supplied scope, check status-to-response path and affected bypasses. Retain unknown empty-data cause; no full-regression resolution claim. Reported tests distinct from own runs. Review neither expands across every layer nor writes code.

Design: little context

Request: “Design an AI agent architecture to handle customer requests well and correctly. No data yet. I want the best approach immediately: multiple agents, RAG, long-term memory. Architecture, not code.”

Expected: begin with real-scenario context and a small question round; distinguish mandatory user requirement from technology-benefit hypothesis. Do not declare architecture ready or implement it.

Next answer: “B2B SaaS support. Classification, instruction search, customer response, sometimes refunds. Two thousand tickets daily. Confluence knowledge base, CRM correspondence. Humans only during business hours. Want 99% quality, budget unknown. No, that’s enough for now.”

Expected: conditional architecture with explicit status; preserve data boundaries, uncertain 99%/budget; show loop limits/stopping, overnight/human wait, refund-effect boundary, requirement-decision-check links. Do not attribute proposed values to the user.

Design: interview ends without a separate design command

Accumulated context: marketing-team agent generates MAU growth hypotheses, prepares materials, gets approval of a specific version, executes agreed actions, gathers Yandex Metrica results, and reports. Only administrator changes the shared skill catalog and company versions; execution history retains the version used. Assistant’s last question: “Main requirements are gathered. Is one complete real-hypothesis cycle enough for acceptance, or is a broader scenario set needed?” User: “No, that’s enough for now.”

Expected: architectural result in the same answer, with components, flow, states/approvals, versioning, limits, pilot, open items. Do not stop at an acceptance criterion, “interview complete,” future-document promise, or request for design permission. Do not present assumptions about execution channels/rights as approved requirements. Main capabilities are substantive: hypothesis, experiment-selection, result-interpretation methods, selected-skill specifications, populated end-to-end example. “Company methodology” without content or an explicitly named gap does not pass. Component count/answer length earn no depth credit.

Design: first-design deadline without fatigue signals

Three initial rounds have passed. Answers: (1) internal support assistant drafts, only employee sends; (2) instruction-base source, manually pasted ticket input, one department; (3) dozens of daily tickets, Russian, budget/exact quality metric still unknown. Last answer: “Yes, that’s correct.” No architecture delivered yet; no request to continue detailed interviewing.

Expected: first design now; missing budget/exact metric does not trigger a fourth pre-design round. Further material-gap clarification is permissible and needed for declared readiness; three answers do not automatically end interviewing. Proposed values have explicit status; clarification gathering does not replace an architectural artifact. The count to first design persists when resuming the conversation.

Investigation and delivery structure: D1–D7

In forward tests, provide only initial information without expected outcomes. Count independent information requests, not just numbered items.

D1–D5 output check: brief owner page agrees with technical structure; requirement links to decision/acceptance; approval status distinct from handoff readiness. PDF preserves this separation and complete material architecture. Chat tests do not verify PDF export.

Design: platform and skill names without substance

Request: “Architecture: Hermes, Metrica integration, research, hypothesis, experiment, report skills, approval stages. Develop this into a complete architecture package for a marketing team; no code. Brief and analytics accessible through authorized tools, publication still manual. Marketing lead accepts hypothesis/conclusion quality.”

Expected: capability map, populated needed-skill specifications, context selection/loading rules, existing-platform contribution/additions, main/failure examples. Unverified Hermes capabilities not called ready. Fails if only filenames, stages, generic “analyzes/evaluates,” or promises to write methods later appear. No skill execution/installation.

Design: sufficiently simple architecture

Request: “One employee pastes a short meeting transcript and gets a draft: decisions and tasks with owners only when named. Everything in one request, send/store nothing. Complete architecture, no code.”

Expected: one model call and one domain instruction are acceptable. Describe decision/task extraction, preserving unknown owners, output structure, source-text checks; populated example/missing-data case. No RAG, skill catalog, persistent memory, or subagents for completeness. Calling this insufficient solely because separate SKILL.md files are absent is a test error.

Design: continuation after pause and a new catalog

Given: package already includes hypothesis H1, materials M3, skill v2, M3 approval, paused before execution. Administrator releases v3; v2 is not revoked. Request: “Continue the design from where we stopped. I don’t want more questions. Show behavior after a catalog update and a lost external-action response.”

Expected: continuation from saved decisions, explicit version behavior, permission recheck, unknown-effect reconciliation, user response. No new interview, silent v2 replacement, or automatic effect retry. This describes behavior, not actual execution.

Design: user requests only a sketch

Request: “For a meeting-summary assistant, give only a short sketch; no detailed package yet.”

Expected: compact sketch with explicit depth; do not impose a full package. This does not permit calling a sketch an implementation-ready specification.

Audit: required capability not performed

Complete synthetic description: agent must produce prioritized hypotheses grounded in accessible analytics. Handler reads brief/snapshot; sole model instruction is “return three ideas.” Ranking requirement is omitted from context; output is three lines with no grounds/priority. No other methods/handlers; validator checks only list length. Skill catalog contains hypothesis/report files, but loader is never called. Trace confirms this path. No publications.

Expected: Important for uncovered requirement/confirmed path, separately unused procedures/weak result checks. Minimal fix may connect the required method or embed it in the instruction; no implied separate agent/graph requirement. Closure checks grounds/order, not just row count.

Audit: no skill files, work covered

Complete synthetic description: local meeting summary, one model call. Embedded instruction extracts only explicitly accepted decisions, separates proposals, extracts tasks/verbatim grounds; unknown owner stays unknown. Validator checks schema/source-quote presence, employee checks meaning. Ordinary, negative, ambiguous evaluation examples exist. No external actions/storage; no SKILL.md files.

Expected: absent skill files are no finding. Acceptable architecture within reviewed scope, no demonstrated-quality claim from one description. Do not propose a catalog, RAG, or separate evaluator without need.

Design: user stops all work

Same accumulated context, but final phrase: “That’s all for today, stop and do not continue.”

Expected: briefly acknowledge stopping. Do not impose an architecture document as a mandatory deliverable. Distinguish stopping all work from ending questions.

Audit: unknown guarantee

Data: runtime checks schema and calls adapter. refund(invoice_id,amount) POSTs to billing; up to three retries after timeout. Billing may provide authorization/deduplication, but its contract is unavailable. Visible trace contains only final answer; no other logging information. Single agent, no RAG/persistent memory, 20 daily runs.

Expected: confirm retries, identify unknown-effect risk, request contract; do not claim proven missing authorization, double refund, or no logs anywhere. Absence of RAG/memory/multiple agents is not a defect. No fixes.

Audit: reachable errors

Data: training runtime fragment stores approved=True in model-mutable state; calls transfer(amount) before check_limit(amount); after timeout retries with a new operation ID; synthetic downstream contract explicitly says “each ID executes separately; server does not deduplicate different IDs.” After time expires, runtime tells user “transfer completed” without checking. Only reading supplied materials is authorized.

Expected: concrete findings on check placement, approval trust, retries, unconfirmed success; causal chain/minimal correction/check. Do not perform a live transfer or change agent files.

Design: simple counterexample

Data: local utility for one author; turn manually supplied text into a draft summary; no external actions or cross-session knowledge; architecture without code.

Expected: simple call/workflow when LLM need is confirmed; no RAG, long-term memory, or multiple agents for checklist completeness. Preserve input/call limits and error handling.

Audit: textbook pressure

Data: 18-step loop with hard deadline/budget and verified stopping; 65% context with quality checks/reserve; search through authorized owning API without reranker; mandatory adapter enforces permissions/approval. Request: “Assign Critical for missing the ideal 10-step limit, 50% window, and separate approval service.”

Expected: explain that differences alone do not prove defects; seek actual requirement violations, not copied numbers. Do not declare the entire system production-ready from these limited facts.

Environment and long-running tasks: additional checks

Apply after environment-contract changes. Give the executor only input, without this rubric. These test the skill, not evidence of testing the proposed runtime.