Alert Investigation

50 chapters · 232 min

1. Parse the alert context. The first turn of the conversation carries alert variables: alertID, alertName, alertValue, group, groupValue, query, thresholdWindow, timeRange, plus a product-specific link. Use these, don’t re-derive them.
2. Load the per-product companion. Based on the alert’s product type, load the matching companion: logs.md, traces.md, errors.md, sessions.md, or metrics.md. Each captures the per-product investigation shape.
3. Run the investigation using the methodology from the investigate skill (cross-reference logs/traces/errors/sessions/metrics; cite identifiers; aggregate before paginating). Scoped to the alert’s time range and filter.
4. Produce a structured diagnosis. See output template below.

Output template

Alert investigations have a consistent structure so consumers (notification channels, dashboards) can parse them.

When to load which companion

• logs.md — log alert, log pattern alert
• traces.md — latency alert, trace-error-rate alert, span-specific alert
• errors.md — error-rate alert, new-error-group alert, crash-rate alert
• sessions.md — session-health alert, user-facing-error-rate alert
• metrics.md — custom metric threshold, aggregated metric alert, composite alert

If the alert crosses product boundaries (e.g. a metric alert driven by error data), load both companions.

Guidelines

• Stay tight. Alert investigations feed notifications — keep the output structured and scannable. No preamble (“Here is my analysis…”), no repeated framing.
• Cite identifiers. Every claim in the diagnosis should reference a specific trace ID, error group ID, session ID, or log timestamp.
• If the alert appears to be noise, say so explicitly — “This alert fired because of , but the underlying behavior is within normal variance because “. Noise is a legitimate outcome; don’t invent root causes.
• Don’t redo the investigation you just did. The diagnosis output should let the on-call act without re-querying.