Signals Scout Error Tracking

200 skills · 1230 min

Quick close-out: is error tracking even loud?

If $exception is absent from top_events or its count is at baseline (no fresh 24h activity, recent_24h_count ≪ count / 7), error tracking probably isn’t where the signal is today. Cheap scratchpad entry + close out:

• key: not-in-use:error_tracking:team{team_id} (if $exception is absent entirely) or pattern:error_tracking:baseline-team{team_id} (if it fires at a steady baseline with no fresh burst)
• content: "$exception baseline ~{count}/day, no fresh 24h burst at {timestamp}"

Close out empty. Re-running with the same key idempotently refreshes the timestamp; the next run reads the entry cold and short-circuits.

How a run works

Cycle between these moves; skip what’s not useful.

Get oriented

Four cheap reads cold-start a run:

• scout-scratchpad-search (text=error or text=exception) — durable team steering from past error-tracking runs. Entries with pattern:, noise:, addressed:, dedupe:, report:, or reviewer: key prefixes tell you what’s normal, what’s already surfaced, what to skip, which report covers an issue, and who owns it.
• scout-runs-list (last 7d) — what prior error-tracking scouts found and ruled out.
• scout-project-profile-get — the $exception row in top_events carries count, distinct_users, recent_24h_count, recent_24h_users (pattern the count/users ratio against the table below), plus existing_inbox_reports for what’s already in the inbox.
• inbox-reports-list (ordering=-updated_at, search=the specific issue id / fingerprint / failing-activity name) — the reports already in the inbox. Your own report-channel reports persist their backing signals under source_product=signals_scout (not error_tracking), so don’t filter source_product=error_tracking — you’d miss every report you authored. A fresh burst on an issue you’ve reported before is an edit, not a new report; pull the closest matches with inbox-reports-retrieve before authoring.

Profile shape — count vs distinct_users

Explore

Patterns to watch — starting points, not a checklist.

Burst with broad reach

recent_24h_count and recent_24h_users both spike together. Usually a fresh regression — many users hitting it independently. Drill in:

1. query-error-tracking-issues-list filtered to status=active, sort by last_seen_at.
2. execute-sql against events with event = '$exception' AND properties.$exception_issue_id = '<id>' grouped by toStartOfHour(timestamp).
3. Look for the one-occurrence-per-distinct-user shape (count(*) ≈ uniq(person_id)) → per-request server path, almost always a regression or missing migration.

Stuck loop (narrow reach)

recent_24h_count very high but recent_24h_users is small. A worker, cron, websocket, or retry is looping. Look at the issue’s stack trace for the activity / job name. Often less urgent than a broad-reach burst, but worth a finding when count is in the thousands and the issue is fresh.

Multi-fingerprint cluster

Multiple fresh fingerprints (different entity_ids in query-error-tracking-issues-list) appearing in the same time window with overlapping stack traces, modules, or call sites → likely shared root cause. Bundle them in one finding (single description, evidence list with all fingerprint ids, dedupe key per fingerprint).

Status regression

An issue with status=resolved that’s now firing again. Filter query-error-tracking-issues-list to status=active and check last_seen_at against first_seen_at — a large gap means old issue resurrected. Strong findings: the team explicitly closed them once.

Stack-trace activity name

When the issue is server-side, the stack trace usually names the failing activity / view / management command. Extract it (top frame, look for <activity>_activity, def view_name, etc.) and pair with advanced-activity-logs-list to find a recent deploy or model change correlation. Cross-source convergence is where this scout earns its keep.

Save memory as you go

Memory is a continuous activity. Write a scratchpad entry whenever you observe something a future error-tracking run should know. Encode the “category” in the key prefix — pattern:, noise:, addressed:, dedupe:, report:, reviewer: — so future runs find it with a single text= search:

• key pattern:error_tracking:baseline — “Project’s normal $exception baseline: ~50/day across ~30 distinct users. Anything materially above that is fresh.”
• key dedupe:error_tracking:019de34e — “Issue 019de34e — surfaced 2026-05-01 11:31–13:22Z, then quiet. If quiet next run, treat as already-surfaced; if firing, escalate.”
• key noise:error_tracking:ci-runner-timeoutexpired — “TimeoutExpired raised by the team’s own CI runner is recurring noise — internal tooling, not user-facing.”
• key pattern:error_tracking:fetch_signals_for_report_activity — “Server activity fetch_signals_for_report_activity was a regression source on 2026-05-01 — if it appears in a fresh stack trace, double-check it’s not the same root cause.”
• key report:error_tracking:019de34e — the report_id of a report you authored for issue 019de34e, so the next run edits it (append_evidence with the fresh window) instead of duplicating.
• key reviewer:error_tracking:ingestion — a resolved owner (bare lowercase GitHub login) for a service / module / activity area, so reports route to a human faster.

By run #5 you’ll have a local map of what’s normal versus what warrants investigation, and burn less time on cold-start exploration.

Decide

The generic report mechanics — search the inbox first (via the report:error_tracking:<issue_id> pointer, else an inbox-reports-list search on the issue’s specific terms — the issue id, the fingerprint, the failing activity name, not a broad word like error), edit-vs-author, the status rules, reviewer routing, non-idempotent dedup, and the priority / repository / actionability fields — live in the harness prompt and in authoring-scouts → references/report-contract.md. Do not re-derive them here. This section is only the error-tracking judgment layered on top:

• Edit when a still-live report already tracks the same issue and it’s still moving — a burst still elevated, a stuck loop still looping, a cluster still growing. A persistent issue is one report across runs: a fresh window confirming it’s ongoing is a re-escalation (append_evidence with the fresh hourly counts and distinct-user numbers), not a new report per tick. A status regression is the exception — an issue the team explicitly resolved that’s firing again is a genuinely new event; if its prior report is already closed, author a fresh report (per the status rules) and repoint report:error_tracking:<issue_id> rather than appending to a resolved item.
• Author when nothing live covers the issue. A report-worthy finding names the issue (issue id + fingerprint), shows the count-vs-distinct_users shape that makes it signal, quantifies the burst against baseline with an hourly breakdown, dates the onset, and — when the stack trace names a server activity / view — cites it with an advanced-activity-logs-list deploy correlation, all in the evidence. Attach the burst as a report chart — the issue’s hourly or daily series with the baseline window on screen — so the spike and its onset are visible next to the numbers. Most findings are investigations → actionability=requires_human_input + repository=NO_REPO. The exception this surface earns: a well-localized bug whose stack trace points at a specific named file / module in a known repo can be actionability=immediately_actionable + repository=owner/repo to open a draft fix PR. Priority: a fresh broad-reach regression (count and distinct_users both spiking, per-request server path) or a resolved-issue status regression is P1, P2 when reach is moderate; a stuck loop or narrow-reach cluster is P3, P2 when count is in the thousands and fresh.
• Remember if it’s below the bar but worth carrying forward (an issue drifting inside the noise band, a fingerprint building history), or to record what you ruled out and why.
• Skip with a one-line note if a noise: / addressed: / dedupe: entry, or an existing inbox report, already covers it.

Sibling courtesy: raw log-line rate/level shifts belong to the logs scout; LLM $ai_* errors to the ai-observability scout; CSP $csp_violation blocks to the csp-violations scout; errors surfaced through session friction to the session-replay scout. Honor their dedupe: entries — your unique angle is always the $exception issue-level burst / regression frame.

Close out

Summarize the run — one paragraph: looked at what, which reports you authored or edited, what you remembered, what you ruled out. The harness writes that summary to the run row as searchable prose; future runs read it via scout-runs-list. Do not write a separate “run metadata” scratchpad entry — the run summary already serves that role.

Disqualifiers (skip these)

• Single user, single session, single occurrence — almost always a personal browser quirk. Confirmed via low count AND low distinct_users.
• Known upstream provider errors — Anthropic / OpenAI rate limits, third-party API outages already covered by past memory. Skip unless volume / shape changes meaningfully.

When in doubt, write a memory entry instead of filing a report.

MCP tools

Direct calls (read-only):

• query-error-tracking-issues-list — start here. Filter status=active, sort by last_seen_at desc.
• query-error-tracking-issue — drill into one issue (frames, sample events, occurrence counts).
• execute-sql against events — for hourly breakdowns, distinct-user counts, per-fingerprint correlation, time-window aggregations.
• advanced-activity-logs-list — pair stack-trace activity names with recent deploys or model changes for cross-source convergence.

Inbox & reviewer routing (mechanics in authoring-scouts → references/report-contract.md):

• inbox-reports-list / inbox-reports-retrieve — the reports already in the inbox; check before authoring so you edit instead of duplicating (ordering=-updated_at).
• inbox-report-artefacts-list — a comparable report’s artefact log; reviewer precedent.
• scout-members-list — the in-run roster for routing suggested_reviewers to a service / module / activity owner.

Harness-level:

• scout-project-profile-get / scout-scratchpad-search / scout-runs-list / scout-runs-retrieve — orientation + dedupe.
• scout-emit-report / scout-edit-report — author a report / edit an existing one (the report-channel contract is in the harness prompt).
• scout-scratchpad-remember / scout-scratchpad-forget — remember / prune stale memory keys.

When to stop

• $exception row in profile is at baseline → close out empty.
• A candidate matches a scratchpad entry with noise: / addressed: / dedupe: key prefix, or an existing inbox report → edit-or-skip with a one-line note.
• You’ve validated some hypotheses and filed reports for what’s solid → close out, even if there’s more you could look at. Fewer, better reports.

“Looked but found nothing meaningful” is a real outcome.