Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.3
references/cloudwatch-omni/app-basics.mdMarkdown13 KBView on GitHub
CloudWatch Application Observability — also called Omni — is an AI-native observability app. A customer investigates their systems by asking questions in conversation rather than by assembling queries and dashboards by hand.
This file is orientation, not procedure. It explains what the pieces are, how they connect, what order they get set up in, and how to tell an Omni request apart from a CloudWatch one. Every procedure lives in a sibling reference, linked below.
Always state these, in any answer drawn from this file:
- The setup order is Domain, then Space, then grants, then telemetry in, then instrumentation.
- Access Profiles are conditional, needed only when async workloads (alerts, integrations, agents) are involved. Name that condition, do not omit it.
- Say, in these words or close to them, that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see. Never describe the setup order without this warning.
- If the request names neither product (“add observability to my API”), ask which one they mean, framed CloudWatch Omni versus CloudWatch, and explain the setup-versus-use boundary in the same message so they can choose — then stop. Do not assume Omni, and do not assume CloudWatch.
Domain — the identity boundary. A Domain carries the authorization provider, either IAM or IAM Identity Center, and owns the endpoint URL customers reach Omni through. The endpoint derives from the Domain’s name, which is why the name is not cosmetic. An account has at most one Domain; an AWS Organization can instead have one Domain shared by every member account.
Space — the access boundary over the account’s CloudWatch Dataset. A Space lives
in exactly one account and one Region, is created under a Domain, and holds no
telemetry of its own. An account has at most one Space per
Region, so a customer wanting telemetry separated across Regions gets a Space in
each. Whether those Spaces may sit in a Region other than the Domain’s depends on
the Domain’s authorization provider: an IAM-only Domain allows it; an Identity
Center Domain requires each Space in the Domain’s own Region, and a customer who
needs Identity Center together with Spaces in several Regions uses the org-scoped
Domain instead (references/cloudwatch-omni/spaces-and-domains.md, Region rules).
Access grant — attaches a principal to one Space at a permission level. The
principal can be a person or group in Identity Center, an IAM role or user, or an
async workload. Grants are the only way anyone reaches a Space’s telemetry through Omni: a Space
with no grants is readable by nobody through Omni, including the account that created it. A grant never
spans Spaces, Regions, or accounts. Grants bound the Omni plane only — where a Space’s data came
from CloudWatch log groups, those groups stay readable through the native CloudWatch Logs APIs
under separate IAM, so a grant is not by itself a way to keep a principal away from that data
(see access-grants.md).
Access Profile — a named boundary for async workloads: an alert running an investigation, an integration calling another system, an agent evaluating traces. These act without a person in the loop to approve what they do, so the profile limits what Omni may do on the customer’s behalf. A profile carries no permissions itself — grants supply them.
Dataset — the account’s CloudWatch store that queries run against: one per
Region, arn:aws:cloudwatch:<region>:<account>:dataset/default, a CloudWatch
resource the Space reads rather than part of the Space. OTel metrics land in it
natively; logs and traces reach it when a dataset integration copies the account’s
log groups — including aws/spans — into it. A customer starting from nothing
usually needs both telemetry flowing into CloudWatch and that integration.
The relationships matter more than the definitions, because each of them determines something a customer will otherwise get wrong.
ADMIN over
every Space in the Domain.Working from nothing, the sequence is:
references/cloudwatch-omni/spaces-and-domains.md or references/cloudwatch-omni/org-domains.md.references/cloudwatch-omni/access-grants.md.references/cloudwatch-omni/instrumentation/collector.md), and create the dataset integration
so logs and traces are copied into the Dataset
(references/cloudwatch-omni/data-forwarding-and-centralization.md). A customer
starting from nothing needs both halves. If
traces are part of it, Transaction Search must be enabled in that Region first —
Omni reads spans from the aws/spans log group, and without it traces never arrive
while everything else looks healthy (account-level, per Region).references/cloudwatch-omni/instrumentation/instrumentation.md for applications and
references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md for AI
agents.Two things are conditional rather than sequential:
references/cloudwatch-omni/access-profiles.md.Constraints:
Two skills cover CloudWatch, and choosing wrongly is worse than any mistake inside a procedure — the agent proceeds confidently down the wrong path.
This skill is for setting Omni up: Domains, Spaces, grants, Access Profiles, ingestion, forwarding, instrumentation, and connecting Slack.
aws-observability is for CloudWatch — Log Insights, metrics, alarms,
dashboards, X-Ray, CloudTrail, Application Signals, synthetics — and also for
day-to-day use of Omni once it is set up: writing queries, building dashboards,
and configuring alerts.
Between the two skills the split is setup versus use, for Omni; everything
CloudWatch — setup (Application Signals, Dynamic Instrumentation) as much as use — lives
in aws-observability:
| The customer wants | Skill |
|---|---|
| To create or configure a Domain, Space, grant, or profile | This skill |
| To get telemetry flowing into Omni for the first time | This skill |
| To instrument an application or agent for Omni | This skill |
| To query telemetry, build a dashboard, or set up an alert in Omni | aws-observability |
| Anything about Application Signals, X-Ray, synthetics, or CloudWatch alarms | aws-observability |
| To investigate a live problem | aws-observability |
Constraints:
list-spaces
probe is not a substitute for asking. Probe only when the customer has already
named a concrete instrumentation/collector task (see the routing rules in
SKILL.md), not for a general “add monitoring” request.aws-observability → references/cloudwatch-omni/concepts.md uses,
so the customer hears one question whichever skill asks it. Do not describe it only
as a difference in interaction style.aws-observability. Naming the table above
in prose is enough; do not start any of it.aws-observability for use. If they named a CloudWatch feature (log group, alarm,
Log Insights, Application Signals, X-Ray, synthetics), route to aws-observability.aws-observability.aws-observability. Those are CloudWatch
features and are not part of Omni setup.references/cloudwatch-omni/spaces-and-domains.md — creating an account-scoped Domain and a Spacereferences/cloudwatch-omni/org-domains.md — a Domain shared across an AWS Organizationreferences/cloudwatch-omni/access-grants.md — giving principals access to a Spacereferences/cloudwatch-omni/access-profiles.md — bounding what an async workload may doreferences/cloudwatch-omni/instrumentation/collector.md — deploying a collector that exports to
CloudWatch’s OTLP endpointsreferences/cloudwatch-omni/data-forwarding-and-centralization.md — forwarding telemetry already
in CloudWatch into the Datasetreferences/cloudwatch-omni/instrumentation/instrumentation.md — instrumenting applicationsreferences/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md —
instrumenting AI agents