Lifecycle And Testing

• Discovery. A scout is a skill that holds a SignalScoutConfig, and the coordinator dispatches from those config rows. The signals-scout- name prefix is optional; it controls only auto-registration, below.
• Config. Each scout has one SignalScoutConfig per (project, skill_name) carrying its schedule (run_interval_minutes, default 1440, or a project-local run_cron_schedule that takes precedence when set), enabled, emit, network_access (trusted default, full for scouts that read arbitrary external sites), the rest of the run posture (output_destinations, structured_output_schema, write_scopes, mcp_gateway_server_ids, model, tags, auto_pause_exempt, display_name), and a last_run_at stamp. A config is auto-registered the first time the coordinator sees a signals-scout-* skill without one, so authoring a prefixed skill is enough to get a scout. A skill named anything else needs its config created with it. Create a fresh per-team scout and its config together with posthog:scout-create; the nested config object sets its schedule, emit posture, and destinations before it can run, and files bundles reference files in the same call. The lower-level posthog:scout-config-create remains available when a skill already exists without a config. Config responses also carry the scout’s description, read live from the skill’s frontmatter (not a config field you set), plus scout_origin (canonical or custom) and owners.
• Coordinator. A periodic Temporal workflow ticks (~every 30 min). Each tick it bounds candidates to projects enrolled via the signals-scout feature-flag allowlist, then dispatches every enabled scout whose schedule is due, most-overdue first, capped per tick. On a rolling interval, due means last_run_at is None (a never-dispatched scout is maximally overdue; manual runs do not set the stamp) or now - last_run_at ≥ run_interval_minutes. On a cron schedule, due means the first slot after the latest of last_run_at, the last schedule edit, and the config’s creation has passed, so a fresh or re-scheduled cron scout waits for its next slot instead of firing at once. There is no sampling — every due scout runs. last_run_at advances for everything dispatched.
• Run. Each dispatched scout becomes one sandboxed agent run with a hard budget of 15 minutes; a run still going at the wall is killed and its row marked failed. The body is the system prompt; the agent orients, explores, files reports or remembers, and writes a one-paragraph summary to the run row. A streak of consecutive scheduled failures longer than a twelve-hour outage could explain trips a breaker that pauses the scout (pause_reason=repeated_failures): the threshold is one more than the runs the schedule fits in twelve hours, clamped to 5–25 (five for a daily scout, thirteen for a rolling hourly interval, fifteen for an hourly cron, whose slots are counted over a window padded for daylight-saving shifts). Manual (scout-run-now) and workflow-triggered failures never count toward the streak, while a clean run from any trigger clears it. The coordinator then probes a paused scout once a day and resumes it on a clean run, unless the project is at its enabled-scout cap, in which case it stays paused until another scout is paused or deleted; enabling it by hand is refused at the cap too.

Pausing a scout = enabled=false. That records status=paused_by_user, which automatic lifecycle sweeps never resume or re-pause; enabled=true resumes from any pause, including a system-applied one (status=paused_by_system, cause in the read-only pause_reason). Config responses expose status and pause_reason read-only; writes flow through enabled. Slowing it = a larger run_interval_minutes (or, on a cron scout, a sparser run_cron_schedule; the cron wins while it is set). Dry-running it = emit=false. Letting it reach sites outside the trusted-domain allowlist = network_access="full". All of these via posthog:scout-config-update (get the id from -config-list), or set at creation time in the nested config object passed to posthog:scout-create.

Path A — per-team (skills store)

The common path for a user customizing scouts for their own project. A scout is just an LLMSkill row named signals-scout-*; create or edit it with the skills-store tools, and the harness globs it in on the next tick.

Notes:

• Prefer edits (find/replace) over a full body rewrite for tweaks — a full rewrite forces you to reproduce the whole body and risks silently dropping unrelated content. Each old must match exactly once. Every write bumps an immutable version; chain further edits via base_version.
• Divergence: once you edit a canonical scout’s row for your team, canonical sync treats it as diverged and stops force-updating it — you keep your edits but lose upstream improvements to that scout. To customize without diverging, duplicate the canonical scout into a new signals-scout-<your-scope> row and edit that; leave the original alone.
• Writing reports needs the signal_scout_report:write scope, and the scratchpad needs signal_scout_internal:write (the sandbox has both). Authoring a scout doesn’t require either — only the harness writes.

Path B — canonical (in-repo, for PostHog contributors)

Improving a scout for every enrolled project. Disk under products/signals/skills/signals-scout-*/ is the source of truth; lazy_seed mirrors changes onto each enrolled team’s LLMSkill rows on the next coordinator tick (or immediately via python manage.py sync_signals_scout_skills --all-enabled). Teams that hand-edited a row are diverged and left alone.

Authoring a new canonical scout is just creating signals-scout-<scope>/SKILL.md and merging — the next tick discovers it, seeds it onto enrolled teams, and auto-registers an enabled config on the default every-24-hours schedule. If you change the fleet shape (add/rename a scout, change the SKILL.md schema), update products/signals/skills/AGENTS.md. On master, CI builds and publishes dist/skills.zip to the downstream distribution repos (the ai-plugin bundle and the standalone skills repo) automatically.

Testing

Dogfood the scout yourself first — before spending any real run. The authoring agent has the same PostHog MCP tools a scout uses at runtime (execute-sql, read-data-schema, the per-product list tools, scout-project-profile-get), so the cheapest iteration is to walk the scout’s own logic against the live project by hand: confirm the watched entity exists and has the assumed shape, run the discriminator to check it separates signal from noise on this project’s data, and run each explore pattern‘s queries. Free and instant — refine the body, re-run the queries, repeat, until the logic holds on real data.

Only once you’re happy do you spend a real run. posthog:scout-run-now {"id": <config_id>} dispatches one run of the scout immediately, regardless of its schedule (get the id from -config-list) — the initial real run, the scout executing end-to-end in the harness. An optional note steers that run alone (read next to the durable notes and never delivered to a later run as a note, though it stays visible in that run’s metadata; needs llm_skill:write and skill-editor access, like a durable note), so you can aim the first run at the case you dogfooded. The run is asynchronous: the call returns a workflow id right away; poll -runs-list (pass skill_name to scope to this scout) / -runs-retrieve for the result. A disabled scout can still be run this way (test before enabling), and a manual run doesn’t touch the schedule or last_run_at. It inherits the scheduled path’s guards (403 not enabled, 429 over quota / daily run budget, 409 a run already in progress) and draws from the same daily run budget as scheduled runs — a dry-run (emit=false) counts too. There’s no free test run, and it’s slow (async, one run per call): firing the same scout repeatedly in a short window burns the project’s daily allowance (and can starve its scheduled scouts). Don’t iterate via -run-now — dogfood the queries by hand to get the body right, and reserve -run-now for the initial real run and the odd re-check after a genuinely meaningful change. The loop is dogfood → run once ready → inspect:

1. Dogfood the discriminator + explore patterns yourself against the live project (above), refining the body until the logic holds — the cheap, iterable part.
2. Create the scout and its config together via posthog:scout-create (the default emit=true goes in the nested config), leaving run_interval_minutes at a sustainable value — no short-interval trick needed. Then spend one -run-now to watch the whole scout execute end-to-end, and inspect once it finishes:
   • posthog:inbox-reports-list — the reports it actually wrote.
   • posthog:scout-runs-list — run summaries.
   • posthog:scout-runs-retrieve — the full reasoning for one run.
   • posthog:scout-scratchpad-search — the durable memory it wrote.
3. If it needs work, go back to dogfooding the queries by hand for the iteration, re-edit via skill-update, and spend another -run-now only once you’ve batched a meaningful change.

Extra-careful variant — dry-run first. For a scout you expect to be chatty, expensive, or high-stakes, set emit=false so it runs and logs what it would have written (visible in -runs-list / -runs-retrieve) without writing to the inbox. Trigger it with -run-now, inspect, refine, then scout-config-update to emit=true. For most scouts, writing straight away and watching the inbox is the faster calibration.

Repo contributors additionally get hogli sync:skill to run the scout against the local harness for a tighter loop before merging.