AWS Observability

Overview

Domain expertise for AWS observability across metrics, logs, and traces, for two products that share the CloudWatch name but are separate services with separate control planes, data models, and APIs:

Enabling Omni does not replace CloudWatch; log groups, metrics, and alarms keep working, and most customers use both.

Works best with the AWS MCP server (opens in a new tab) — enables running CLI commands, querying CloudWatch, and validating configurations directly. All guidance also works with standard AWS CLI access.

Note: Reference files contain specific runtime versions, quota values, and feature matrices that may change. When precision matters (e.g. deploying to production, choosing a runtime, or checking a quota), confirm values against current AWS documentation rather than relying solely on the values in these files.

Scope guard — is this an observability request?

This skill owns the scope decision for every request routed to it, and this section is its sole home.

If a request is NOT about AWS observability with CloudWatch or CloudWatch Omni (weather, trivia, general chit-chat, non-observability coding), decline it in one sentence: state plainly that it is out of scope, never fabricate an answer, and never claim a false capability limitation (no “no internet access”, no “no weather data”) — the reason is simply that it is out of scope. Then redirect by naming what this skill does cover: querying CloudWatch and CloudWatch Omni logs, traces, and metrics, building dashboards, configuring alerts, investigating a service, or agent evaluation.

Do not attempt the off-topic task, and do not call a tool or run a query in pursuit of it. Keep it brief: no lecture, no long refusal.

Step 0 — CloudWatch or CloudWatch Omni?

Decide this before routing. The natural wording (“set up an alert for high latency”, “build a dashboard”, “query my logs”) does not say which product the customer means.

1. The customer names the product — “Omni”, “Application Observability”, “Agent Observability”, a Space, Domain, Dataset, access grant, spaceId, cloudwatch-omni, Omni SQL, PromQL, views, context graph, evaluators, agent evaluation / scoring traces / online or continuous evaluation / gen_ai.evaluation, and OTel span vocabulary (traces.default, logs.default, spans, durationNano, status.code, resource attributes, service.name) → Omni. “Log Insights”, “log group”, “metric namespace”, “CloudWatch alarm”, “metric alarm”, “composite alarm”, “anomaly alarm”, “X-Ray”, “Application Signals”, “canary”, “CloudTrail”, “Dynamic Instrumentation” → CloudWatch. A “log group” named inside an agent-evaluation request is the online-evaluation data source to verify, not a CloudWatch signal. A bare “alarm” (or “alert”) with no other product signal is ambiguous — fall through to rule 3 and probe: a Space → Omni alert (alerts.md (opens in a new tab)); no Space → CloudWatch alarm (cloudwatch/alarms.md (opens in a new tab)). Exception: a “PromQL alarm“ is a CloudWatch alarm on OTel metrics (cloudwatch/alarms.md (opens in a new tab)) — “alarm” wins over “PromQL”.

2. Knowledge or how-to question (“what is an Omni alert”, “does Omni have an API”, “how do alerts differ from alarms”) → answer from the reference files directly. Do not probe the account, and do not divert to the other product. Whether a capability exists is a fact about the product, not the account. concepts.md (opens in a new tab) carries the full feature-equivalence matrix. 2a. Authoring / how-to alert request — “create / set up / write an Omni alert that fires when X”, “how should I alert on Y” — with no Space or Region supplied and no go-ahead to actually create it, is a HOW-TO request. Deliver the authoring guidance (the alert-authoring must-state checklist below, then alerts.md (opens in a new tab)) FIRST, from the reference files. Do NOT stall on a Region/Space clarifying question and do NOT fall back to a CloudWatch alarm for a request that explicitly says “Omni alert”. Probe the account only once the user supplies a Space/Region or asks you to create it. 2b. A telemetry object is already in the prompt — the user pastes, or the UI passes as context, a span, trace, log record, or query result and asks what it means, what errors it has, or how long it took. This is not a live-data request: do not probe the account, run a query, or ask the user to fetch it again. Read it in place per the “Reading a span or trace you already have” section of query/sql-logs-traces.md (opens in a new tab) and state every item in its must-state list. 2c. A query request whose live result is empty or unreachable (“show me the slowest spans”, “which traces failed”) is still answered with the methodology — the exact query and the field rules that make it correct (see the must-state callouts in query/sql-logs-traces.md (opens in a new tab)). Zero rows, a wrong-Region Space, or an unreachable endpoint is reported as a finding alongside the query, never as the whole answer.

3. Request that must act on live data, and the wording is ambiguous (not a Step 0.5 content question, which is answered from the catalog) → probe the target Region first:

   Copy

   aws___call_aws → aws cloudwatchomni list-domains
   aws___call_aws → aws cloudwatchomni list-spaces        # scope to the target Region

   • You MUST state, when resolving the ambiguity or asking the user, that a Space is one per account per Region, so the probe (and any Omni query) targets the Region the request concerns — not just the default Region — because an Omni query against the wrong Region returns an empty result easily misread as “no data”.
   • A Domain and a Space exist in that Region → Omni.
   • No Space → CloudWatch: alarms → cloudwatch/alarms.md (opens in a new tab), dashboards → cloudwatch/dashboards.md (opens in a new tab), queries → cloudwatch/log-insights.md (opens in a new tab), metrics → cloudwatch/metrics.md (opens in a new tab). If the customer explicitly asked for Omni and has no Space, that is a first-time setup — see step 4.
   • The probe itself errors (“not yet supported”, unknown service, endpoint does not resolve) → the CLI/SDK model in use lacks cloudwatchomni. That is not evidence Omni is absent and must not be reported as “Omni is unavailable”. The customer’s installed AWS CLI/SDK most likely predates the service. If the request carried any Omni signal, give the customer the upgrade command to run (AWS CLI v2 reinstall or brew upgrade awscli; pip install -U boto3 botocore) and have them re-run aws cloudwatchomni list-domains — exact steps in programmatic-access.md (opens in a new tab). Never run a package-manager upgrade or installer on the host yourself (brew, pip install -U, .pkg/MSI); it mutates the customer's machine beyond the request and can break unrelated tooling — hand over the command and continue with the guidance. Never substitute a CloudWatch or X-Ray command for an Omni request. If the request carried no Omni signal, do not block on the upgrade: proceed on the CloudWatch path (the pre-Omni default) and mention the upgrade only in passing.
   • Still inconclusive → ask the customer.

4. First-time Omni setup — creating a Domain or Space, granting access, provisioning ingestion, forwarding log groups into the Dataset, connecting Slack, or instrumenting an application or AI agent so traces reach a Space → STOP and route to the setting-up-cloudwatch-observability skill. This skill covers a Space that already has data. An instrumentation / ADOT / OTel-collector request that names neither Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on nor Omni or a Space is ambiguous — probe list-spaces in the target Region: a Space → route to the setting-up-cloudwatch-observability skill’s application-instrumentation reference (plain ADOT SDK, no add-on); no Space → cloudwatch/application-signals-onboarding.md (opens in a new tab).

Under-specified alert requests: When an alert or alarm request names what to watch (a symptom or a service) but not the inputs it needs — the threshold value and the evaluation period — ask for those rather than inventing them. Notifications are optional (per alerts.md), so ask for a notification destination only if the user wants to be notified. This holds on both paths (an Omni alert or a CloudWatch alarm).

Under-specified dashboard requests: When a dashboard request names what to show but not which metrics, panels, or layout, ground those against the data and confirm the panel set rather than inventing panels; a dashboard has no threshold, period, or notification. For ANY dashboard authoring/save request, also open dashboards.md (opens in a new tab) and surface its “Facts you MUST surface when building or saving an Omni dashboard” checklist (see the dashboard must-state subsection below). Which signals or panels a named resource type needs is a Step 0.5 catalog question, not a dashboards-file question.

Step 0.5 — Service-health investigation (routing)

A large share of real questions are phrased as an operator symptom, not as a tool: “is my <service> throttled / slow / erroring / unhealthy,” “which of my <service>s are <symptom>,” “what’s the health of my <service>,” “what does <service> depend on and which is broken,” “what signals / what should be on a dashboard or view for <service>.” These are observability-data questions — answer them from the telemetry surface, not from the resource’s control plane, and answer with the methodology (the correct signals, aggregation, scoping, and caveats) even when you also pull live numbers and even when no matching resource exists in the account.

Route by the symptom, then open the reference and surface every applicable item in its “facts you MUST surface” checklist — the checklist is the output contract, and it lives in the reference file, not here:

• A metric symptom about one AWS service (throttling, latency, error rate, restarts, saturation, “what signals should it get”, which signals or panels a named resource type needs on a dashboard or view) → query/promql-metrics.md (opens in a new tab). Open its “Service-health question — facts you MUST surface” section and surface every applicable item. Dashboard or view wording does not send this to either dashboards file (neither carries the signal facts) nor to rule 3’s probe — it is a content question.
• “What does <service> depend on / what’s broken downstream” / blast radius / who is affected / which direction do I walk the graph / what do CALLS, ACCESSES, RUNS_ON mean → context-graph.md (opens in a new tab). Open its “Dependency / blast-radius question — facts you MUST surface” section and surface every applicable item. A slowness or error symptom phrased in terms of the graph, dependencies, or edge types routes here, not to the metric bullet above.
• Individual slow or failing spans / traces (“slowest spans for <service>“, “which traces failed”) are trace SQL, not a metric aggregate → query/sql-logs-traces.md (opens in a new tab). Open its “Span Duration” and “Finding failed spans” sections and state every item in their must-state callouts — a request about a service’s latency or error rate (an aggregate signal) is the PromQL bullet above instead.

Authoring an Omni alert — surface the must-state checklist

When you author or advise on an Omni alert, open alerts.md (opens in a new tab) and surface the items relevant to the task from its “Facts you MUST surface when authoring an alert” section. That section holds the per-item detail and is the source of truth, so state what fits the request rather than restating it here.

Agent evaluation — surface the must-state callouts

When a request is about scoring traces, choosing an evaluator, reading stored scores, online/continuous evaluation, or evaluation datasets, open agent-evaluation.md (opens in a new tab) and state every applicable item from its “tell the user ALL of this” callouts — level rules (one level per call, the three level semantics, tool-call level needs tool spans), evaluator redirect and ground truth, where scores are stored and the wrong-table read, and online-evaluation data-source verification. The callouts are the output contract; the answer text must carry them, not just the plan.

Building or saving an Omni dashboard — surface the must-state facts

When you author, save, read back, or debug an Omni dashboard, open dashboards.md (opens in a new tab) and surface the items relevant to the task from its “Facts you MUST surface when building or saving an Omni dashboard” checklist. That checklist holds the per-item detail and is the source of truth.

Routing — CloudWatch (references/cloudwatch/)

Routing — CloudWatch Omni (references/cloudwatch-omni/)

Rows that act on live Space data assume Step 0 found a Space. Knowledge questions are answered from the file directly.

Files

references/cloudwatch/

references/cloudwatch-omni/