Authoring Scouts

Authoring Signals scouts

A scout is a scheduled agent that wakes on its own interval, looks at one PostHog project, decides what’s genuinely worth surfacing, and writes it into the Signals inbox as a report — or closes out empty, which is a real outcome. PostHog ships a fleet of canonical scouts (a cross-product generalist plus per-surface specialists). This skill helps you and your agent adapt those canonical scouts to a specific project, or author new scouts from scratch for a use case the fleet doesn’t cover.

A scout’s output is the report channel: it lists emit_report / edit_report in its frontmatter allowed_tools and authors or edits full inbox reports 1:1 directly. The canonical fleet runs this way, and every new scout should too — always include the allowed_tools opt-in when authoring one. Where that output lands is a separate, per-scout config decision: the report goes to the Signals inbox, and the same report can be delivered to a Slack channel or DM at the same time (output_destinations under Run posture) — so don’t rule a scout out of a job because the user wants the result in Slack. (A historical signal-emitting channel — weak emit-signal findings a pipeline consolidated — still exists in the harness for scouts that never opted in, but it is deprecated: don’t author new scouts on it, and opt an old one in rather than extending it.)

A scout is an LLMSkill that holds a SignalScoutConfig. The harness loads the body verbatim as the agent’s system prompt, and progressively reads any bundled reference files on demand. The config row is what makes a skill a scout. Any valid skill name works, so the signals-scout- prefix is optional. The prefix controls one thing: the coordinator globs signals-scout-* to auto-register a config for a skill that has none. A skill with any other name needs its config created alongside it, which is what scout-create does.

The job before the writing

Don’t write a scout in the abstract. Ground it in the target project first — a scout is only as good as its fit to the data it watches. (The scout tools were recently renamed from signals-scout-* to scout-*; if a scout-* name comes back unknown, the server may still expose it under the legacy signals-scout-* name — search the tool catalog and call whichever name it returns.)

1. Read the project. posthog:scout-project-profile-get returns the deterministic snapshot the scout itself cold-starts from: products in use, top events with reach/burst metrics, integrations, existing inbox counts. If the scout watches a specific event, confirm it exists and check its shape with posthog:read-data-schema. A scout for an event the project doesn’t capture is dead on arrival.
2. See what already runs. posthog:scout-config-list lists every existing scout on the project with its schedule, enabled, and emit posture, plus each scout’s description (pulled from the skill’s frontmatter) so you can tell what a scout watches without loading its body. Don’t duplicate a surface a canonical scout already covers — adapt that one instead.
3. Read the closest canonical scout. It’s your template and your reference shape. Pull it with posthog:skill-get {"skill_name": "signals-scout-<x>"} (per-team rows) or read it from the repo at products/signals/skills/signals-scout-*/. The generalist (signals-scout-general) is the broad template; if your scope is domain-tight, pick the specialist closest to your surface — list the live roster with posthog:skill-list {"search": "signals-scout"} (specialists exist for most product surfaces: error tracking, logs, AI observability, experiments, feature flags, session replay, web analytics, surveys, and more).
4. Skim the inbox. posthog:inbox-reports-list shows what reports are actually landing — calibrate so your scout adds signal, not noise.

Choose the path

There are two independent decisions: what you’re building, and where it lives.

What

Where

Adapting-in-place tradeoff: editing a canonical scout’s row for your team marks it diverged — you stop receiving upstream improvements to that scout. If you only need an additional behavior, prefer authoring a new, differently-named scout (signals-scout-<your-scope>) and leaving the canonical one intact.

See references/lifecycle-and-testing.md (opens in a new tab) for the exact skills-store calls, the build/lint commands, and how seeding works.

Write the scout

First pick the shape. references/scout-patterns.md (opens in a new tab) is a cookbook of the reference architectures scouts fall into — anomaly watcher, liveness/absence watcher, zero-result/unmet demand, watchlist explore/exploit, cross-product correlation, recommendation/gap, warehouse-backed source, custom single-event, open-text theme, adversarial/abuse concentration, external-tool/code, state∩code intersection, custom issue-tracker/work-queue, daily digest/roll-up, triage over a pre-detected stream, first-person dogfooding/probe, recurring measurement/LLM-judge, maintainer/steward, owner-scoped book/queue, trigger-to-brief enrichment, dispatcher/campaign, fleet meta-scout/reviewer — each mapped to a canonical scout or a proven custom shape you can copy as scaffolding. It also makes the key point that a scout can watch any source PostHog ingests into the data warehouse, not just analytics events (a Slack channel sync, a billing system, a CRM, a support inbox), plus external systems reachable from the sandbox, other agents’ output, and the scout fleet itself. And where a built-in signals source already covers the surface (GitHub and Linear issues), the issue-tracker pattern says where that source stops and a scout starts paying for itself. A scout does not have to stop at describing a fix: the maintainer / steward pattern is how a scout holding write_scopes keeps a family of dashboards, alerts, warehouse views, or scanner prompts healthy itself, with a write ladder that says which changes it applies, which it verifies and reports, and which it hands to a human. Find the closest pattern, then write the body.

Follow references/scout-anatomy.md (opens in a new tab) — it has the frontmatter schema (including the allowed_tools report-channel opt-in every scout needs), the canonical body structure (quick close-out → orient → domain discriminator → explore patterns → save-memory → decide → disqualifiers → close-out), the lean-body rule, and copy-ready skeleton templates for both a specialist and the generalist.

Write the body feature-forward. The body is a prompt a run reads in full, every run, for as long as the scout lives — so it states what the scout watches and how it decides, and nothing else. Four things do not belong in it:

• No rollout state. A preview flag, an early-access gate, a “recently renamed” tool, a port that is happening one scout at a time. It is true this month and misleading next month, and a run cannot tell which. Write the behavior the scout should follow now; if a capability may be absent, say how the run tells, not when it is due.
• No backlog. Work that is planned, a field that does not exist yet, a lane somebody means to add. A run cannot act on it, so it only spends tokens and invites the run to wait for something.
• No project-only facts. A metric, dashboard, event, id, or reviewer that exists in one project. A canonical scout ships to every project, so a fact from one of them is wrong for almost every run. Text about an optional product says how the run tells whether the project uses it, and never assumes it does.
• No duplicated harness mechanics. The report-channel contract, the run gates, and the close-out format come from the harness prompt, which every report-channel scout is given. A copy in the body drifts from the real one and the run then holds two versions of the same rule. Keep the body to your scout’s own domain framing.

The same four apply to a bundled reference: it is read at run time too, so it carries no more rollout state or backlog than the body.

Two craft references the whole fleet reasons in terms of — a good scout’s Decide and memory sections are built on them, so read them before writing those sections:

• references/report-contract.md (opens in a new tab) — the report tools (scout-emit-report / scout-edit-report), the report bar (author 1:1 only for a finding you’d own end-to-end), suggested_reviewers routing, the dedup-via-report_id discipline (the channel isn’t idempotent — reconcile against existing reports via the vanilla inbox-reports-list / inbox-reports-retrieve before authoring), and the accepted caveat that the pipeline may later rewrite an authored title/summary. This is how your scout decides what clears the bar and how to file it.
• references/dedupe-and-memory.md (opens in a new tab) — the four-states classifier (net-new / material-update / already-covered / addressed-or-noise), the scratchpad key-prefix vocabulary, and the cross-project noise patterns. This is how your scout avoids re-filing and learns across runs.
• references/report-checks.md (opens in a new tab) — the follow-up checks a scout attaches to a report so “did the fix hold?” is measured later instead of remembered. This is how your scout closes the loop on a finding after somebody acts on it.

A report is backward-looking; a check is the opposite direction — an expectation plus a time to test it, written onto a report with scout-report-check-create. Give your scout a checks section when its findings are the kind whose fix shows up in data later. Three rules belong in the body, and the reference has the rest:

• List the report’s existing checks first (scout-report-check-list), every time. An open check for the same claim makes a second one noise, and a report holds at most five open checks.
• metric_threshold wherever one number settles the claim and an event or action series can carry it. The coordinator measures it itself, with no scout run.
• agent when no single number settles it, or when the number lives outside events — a log rate, a fix whose effect shows in which entities fire rather than how many, a claim that needs a stack trace read.

The single most important design decision in any scout is its signal-vs-noise discriminator — the cheap profile-shape read that separates “worth investigating” from “baseline”. For error tracking it’s the count vs distinct_users ratio; for CSP it’s reach over raw count. Your new scout needs its own. Name it explicitly near the top of the body so every run anchors on it.

(The one exception: a measurement scout on the structured-output channel holds no bar — it applies a rubric to every sampled item, and the rubric takes the discriminator’s slot as the design surface to name, dogfood, and calibrate. See the recurring measurement / LLM-judge pattern in references/scout-patterns.md.)

A second design consideration applies to a metric-shaped scout — one that scores, ranks, or reports a named, reusable measure, whether a business measure (MRR, churn risk, usage revenue, activation) or operational telemetry it computes every run to monitor or report (cost per run, failure or error rates, latency, throughput). If the project has an approved metric for that measure (metric-list shows what exists), name it in the body and run it with data-catalog-metric-run, so the scout’s number matches the one the team already reports.

Run posture (config)

A scout’s schedule and emit behavior live on its SignalScoutConfig, separate from the skill body. For a brand-new scout, pass these settings in the nested config object of the posthog:scout-create call, including creating it disabled or in dry-run before it ever runs. The endpoint creates the skill and config atomically, always opts the scout into the report channel, and safely re-applies config fields when the same definition is retried. Otherwise the coordinator auto-registers an enabled config on the default every-24-hours schedule on its next tick (up to ~30 min). For an existing scout, tune with posthog:scout-config-update (find the id via -config-list):

• run_interval_minutes — 30 to 43200. Default 1440 (every 24 hours). Slow a chatty or expensive scout by raising this.
• run_cron_schedule: a five-field cron expression ('30 9 * * *', '0 9 * * 1-5') evaluated in the project’s timezone; occurrences must be at least 30 minutes apart. When set it takes precedence over run_interval_minutes, so a “slow it down” edit on a cron scout has to change or clear the cron (set null to return to the rolling interval). A new or edited schedule anchors on the edit time and waits for its next slot rather than catching up on a past one. Reach for it when the run should land at a wall-clock time: a digest before standup, a weekday-only watch.
• enabled — false pauses the scout entirely (coordinator skips it).
• emit — defaults to true: the scout writes its reports straight to the inbox. The standard flow is to make a scout and let it write — seeing what actually lands is the fastest way to calibrate it. Set emit=false (dry-run) only when you want to be extra careful: the scout still runs and logs its reasoning but writes nothing to the inbox. Reach for dry-run on a scout you expect to be chatty, expensive, or high-stakes; for most scouts, just writing and watching the inbox is the better loop.
• network_access — defaults to trusted: the scout’s sandbox can only reach the platform’s trusted-domain allowlist (PostHog, GitHub, common package registries), which covers the MCP loop and gh but blocks everything else. Set full for a scout whose skill needs to read arbitrary external sites, e.g. documentation, papers on arxiv.org, or a vendor status page. Applies from the scout’s next run, and changes are activity-logged.
• auto_pause_exempt — defaults to false. A scout whose reports nobody engages with (no open, rating, or action — the cloud web inbox records reads; other clients don’t yet) is warned and then paused automatically (pause_reason=ignored) — every run costs a sandbox agent, so a scout producing output no human consumes shouldn’t keep running forever. A scout that is merely quiet is only flagged (pause_reason=no_output, a warning that never advances to a pause), since a watch scout’s silence can be its job. -config-list shows the warning as status=pending_pause and the pause as status=paused_by_system; setting enabled=true again resumes the scout with a fresh grace window before the sweep may judge it again. Set auto_pause_exempt=true up front for a watchdog scout whose whole job is to stay quiet, so it never even picks up the quiet flag.

Steering with notes (no authoring needed)

Sometimes you don’t want to change the scout — you want to tell it something. That’s what scout notes are for: short steering messages any team member (or an agent acting for one) leaves for the fleet, which every run picks up as prior context alongside its scratchpad and run history. Reach for a note instead of an edit when the steer is feedback, a pointer, or context with a shelf life:

• Feedback on output: “the staging traffic spike you keep flagging is known noise, stop reporting it”.
• A pointer: “dig into the EU signup funnel this week — we think something regressed”.
• Context the scout couldn’t know: “we shipped a new checkout on Tuesday, treat conversion shifts after that as expected”.

The tools (reads on the public signal_scout:read scope; because scouts read notes verbatim, writing or deleting one requires the same authorization as editing a scout’s skill — the llm_skill:write scope plus skill editor access):

• posthog:scout-notes-create {"content": "...", "skill_name": "signals-scout-web-analytics"} — address one scout by its exact skill name (roster via scout-config-list; the skill must already exist, so a typo’d target is rejected instead of silently steering no one), or omit skill_name for a general note every scout sees. Optionally set expires_at so a time-boxed note (“watch closely this week”) retires itself. skill_name: "pipeline:report-research" addresses the report pipeline’s research stage instead of any scout, for guidance about how the reports it builds from clustered signals get researched, judged, and routed (“route billing-adjacent reports to the billing folks”). Reports a scout authors directly never pass through that stage, so a rule for them belongs in a fleet-wide note (omit skill_name) or a per-scout one. It is the only pipeline:* audience; any other value is rejected.
• posthog:scout-notes-list — browse the active notes; pass skill_name to see what a given scout will read. Expect system-derived notes alongside the human ones: a dismissal or snooze note, a Discuss question, a thumbs-rating note, and a reviewer add/remove on a report are each forwarded to the relevant scout as a note, labelled by origin (report_dismissal, report_discussion, report_feedback, report_reviewer_correction; a note left directly is human). Derived notes expire on their own after ~30 days. Three caveats: a Discuss question or rating note only forwards when the person had the notes-write authorization themselves (skill-editor access plus, on a scoped credential, signal_scout:write and llm_skill:write); a reviewer correction only forwards when the person had skill-editor access on the canonical project (the reviewer change itself still lands on the report); and the list hides every derived origin from a caller without report read access (task:read), since they quote report content. A missing derived note can mean either. A dismissal note is still on the report and a Discuss question on its discussion task, but a rating note exists only as the forwarded scout note (plus an analytics event), so a rating note that did not forward is gone.
• posthog:scout-notes-delete {"id": "..."} — retire a note that’s been acted on or no longer applies.
• posthog:scout-run-now {"id": <config_id>, "note": "..."}: steer one run only. The note (up to 1,000 characters) is read by that run next to the durable notes and is never delivered to a later run as a note, so use it for “check this now” instead of a note every scheduled run would keep reading. It does stay visible in that run’s metadata (run_note) when a later run reads its history, so phrase it as a dated one-off (“today only: …”) rather than standing policy. It needs skill-editor access and, on a scoped credential, on top of the base , the same two scopes as a durable note (a 403 otherwise; drop the note to run without it), and spends a run like any manual dispatch.

How scouts treat notes: every run reads its notes in step 1 and is told to let a fresh note visibly shape what it investigates — but notes are advisory. They direct attention; they don’t lower the scout’s evidence bar or force a report, so a note saying “report X” still gets an honest investigation, not an automatic emit. The scout closes the loop in its run summary (which notes it acted on and how) and folds absorbed guidance into its scratchpad.

Choosing between a note and an edit: a note is the right tool for this project, right now steering and for trying a nudge before committing to it; a skill edit is the right tool once the steer is permanent policy (a disqualifier, a threshold, a scope change). A note that you keep re-leaving is a skill edit waiting to happen — promote it. Note lifecycle stays with humans: scouts never delete notes, so retire acted-on notes yourself (or set expires_at up front) to keep the channel high-signal.

Test loop

Dogfood the scout yourself before you ever spend a real run. You — the agent authoring the scout — have the same PostHog MCP tools a scout uses at runtime (execute-sql, read-data-schema, the per-product list tools, scout-project-profile-get). The cheapest, fastest iteration doesn’t touch a scout run at all: walk the scout’s own logic against the live project by hand. Confirm the watched event/entity exists and has the shape you assumed, run the discriminator to check it actually separates signal from noise on this project’s data, and run each explore pattern‘s queries to see what they surface. This loop is free and instant — refine the body against what you find, re-run the queries, repeat, until the scout’s logic holds up on real data. This is where the real iteration happens.

A few runtime tools cannot be dogfooded, so validate them in the first real run instead. Anything that takes a run_id only works from inside a run, and a tool the project has not been switched on for fails whoever calls it. posthog:scout-lighthouse-audit is the one to plan around: it loads a page in a real throttled browser and returns the lab metrics, the element the browser chose as the Largest Contentful Paint, where the LCP time went phase by phase, and the ranked savings estimates — which is how a page-performance finding names a cause rather than guessing one. It needs the run’s run_id, it is restricted to an allowlist of public pages (the browser signs in to nothing, so a page behind a login would report the login screen’s numbers as the page’s), it is limited to the projects switched on for it, and it is capped at five audits per run. A rejected call costs nothing, but once the page loads the slot is spent whatever the result, and every error message ends with how many audits the run has left, which tells a rejection apart from an exhausted budget. So: treat it as available only once a run’s own call succeeds, never write a body whose finding depends on it, and keep lab and field evidence labelled apart — one throttled cold load explains a finding, and the field percentile over real users is what establishes there is one. Then check the first real run’s transcript for the call and what it returned, the same way you would check a query you could not run by hand.

Only once you’re happy with the body do you spend an actual run. posthog:scout-run-now {"id": <config_id>} dispatches one run of the scout immediately, regardless of its schedule (find the id via -config-list). This is the initial real run — the scout executing end-to-end in the harness, writing scratchpad memory and (with the default emit=true) writing reports to the inbox. The run is asynchronous: the call returns a workflow id right away, so poll -runs-list / -runs-retrieve for the result. A few things to know:

• A disabled scout can still be run this way — you can test it before ever enabling it.
• A manual run does not change the scout’s schedule or last_run_at.
• It inherits every guard the scheduled path has: 403 if scouts aren’t enabled for the project, 429 if the project is over its Signals credits quota or daily run budget, 409 if a run for this scout is already in progress.
• It draws from the same daily run budget as scheduled runs — and a dry-run (emit=false) still consumes a run. There’s no free test run: every -run-now spends the project’s daily scout-run allowance, so firing the same scout repeatedly in a short window burns through the budget (and can leave the project’s scheduled scouts unable to run that day). Don’t use -run-now as your iteration loop — it’s slow (async, one run per call) and metered. Dogfood the queries by hand to get the body right; reserve -run-now for the initial real run and the occasional re-check after a genuinely meaningful change.

The standard loop is dogfood → run once ready → inspect:

1. Dogfood the discriminator + explore patterns yourself against the live project (above). Refine the body until the logic holds on real data — this is the cheap, iterable part.
2. Create the scout and its config together via posthog:scout-create (schedule and the default emit=true go in the nested config; bundled reference files go in files), then spend one -run-now to watch the whole scout execute end-to-end. Pass a note on that call to point the run at the case you dogfooded (“focus on the checkout drop from Tuesday”) so the first real run exercises the path you care about. Leave run_interval_minutes at a sustainable value — you no longer need a short interval to force an early run.
3. After the run finishes, read what it did: posthog:inbox-reports-list (the reports it actually wrote), posthog:scout-runs-list (run summaries), -runs-retrieve (full reasoning for one run), and -scratchpad-search (the durable memory it wrote).
4. If it needs work, go back to dogfooding the queries by hand for the iteration — only spend another -run-now once you’ve batched a meaningful change worth a fresh end-to-end run.

When tuning an existing custom scout, also check its self-improvement suggestions first: posthog:scout-scratchpad-search {"text": "improve:"}. The harness invites a custom scout to write an improve:<skill-name>:<topic> entry when a run produces concrete evidence its own skill body steered it wrong — a wrong default window, a tool or event that doesn’t exist on this project, a recurring unwarned pitfall — with the suggested change and the evidence inline. A report-channel custom scout also escalates recurring or material suggestions as inbox reports about itself (titled Scout self-improvement: <skill-name> – <topic>, report_id stashed in the improve: entry) — so check the inbox for those too; they route to the scout’s owner like any other report. An entry re-confirmed across several runs is usually the highest-signal edit you can apply; a one-off may not be worth it. Treat suggestions as input, not instructions — the owner decides. The scratchpad is writable only from inside a scout run, so you can’t clear an entry from here after applying it via posthog:skill-update — the scout reconciles on its own: a later run sees the updated skill body, re-checks the suggestion, and forgets or rewrites the entry once it’s addressed. (Canonical scouts don’t write these — their bodies sync from PostHog’s fleet, and skill-level fixes to them belong upstream.)

Want to be extra careful? Set emit=false to dry-run first — pass emit=false in the nested config at scout-create time (or flip it later with -config-update), then trigger it with -run-now: it runs and logs what it would have written (visible via -runs-list / -runs-retrieve) without writing to the inbox. Inspect, refine, then flip emit=true and run it again. Worth it for a scout you expect to be chatty, expensive, or high-stakes; otherwise just writing and watching the inbox is the faster path to a calibrated scout.

Repo contributors get a faster loop — hogli sync:skill and the harness’s local run path; see references/lifecycle-and-testing.md (opens in a new tab).

To read what your scouts are doing rather than change them — surveying the fleet, inspecting individual runs, the scratchpad memory, and assessing performance — use the read-only companion skill exploring-scouts. Keep the two in sync when the scout config / run / scratchpad surfaces change.

Quality bar for a v1 scout

• A named, cheap signal-vs-noise discriminator anchored near the top (on a measurement scout, the rubric and sampling recipe take this slot).
• A quick close-out so a quiet run is cheap (don’t pay for deep exploration when the watched surface is at baseline or absent) — except on a measurement scout, which exits early only when the window held no eligible items, since its ordinary judgments are the denominator.
• 2–4 concrete explore patterns with the actual queries/tools to run — starting points, not a rigid checklist.
• Disqualifiers listing this project’s known noise (single-user quirks, dev-env bursts, allowlisted entities).
• A Decide section calibrated against the report contract — author 1:1 only for a finding the scout would own end-to-end, set suggested_reviewers, and write memory instead when a candidate is below the bar.
• A checks step where the findings are the kind whose fix shows up in data later — list the report’s open checks first, then pick metric_threshold or agent by whether one number settles the claim.
• A feature-forward body: no rollout state, no backlog, no project-only facts, no harness mechanics the prompt already carries.
• Save-memory guidance using the scratchpad prefixes so the scout gets smarter each run.
• A lean body (push depth into references/) — every line is a recurring token cost on every run.
• A tight frontmatter description — a sentence or two naming the surface and the shapes it watches. Every scout’s description loads into the caller’s AI plugin together, so wordy descriptions waste token budget and get truncated; skip the fleet-wide boilerplate (report bar, durable memory, self-contained peer).