Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.39
references/cloudwatch-omni/spaces-and-domains.mdMarkdown31 KBView on GitHub
A Domain is the identity boundary: it carries the authorization provider and owns the endpoint customers reach Omni through. A Space is the workspace that holds telemetry and runs inside one account and Region. Setup is always Domain first, then Space, because the Space is created against a domain ID.
Always state these, in any answer drawn from this file: the quotas (one Domain per account, one Space per account per Region); that the domain name becomes the endpoint URL; that
create-spacenever verifies the role can be assumed; and, before any delete, that deleting a Space is destructive and needs the customer’s confirmation.Two questions this file answers in full, each with every item below:
- Can a Space be in a different Region from its Domain? — depends on the Domain’s provider: yes under an IAM-only Domain; no under an Identity Center Domain, whose Spaces must be in the Domain’s own Region; Identity Center plus Spaces in several Regions is what the org-scoped Domain is for. Never a bare “yes”.
- The Space was created successfully but fails with an authorization error the moment it is used. —
create-spacechecked only that the role ARN was present and in the caller’s account, never that the role can be assumed, so a wrong trust policy is accepted at create time; check the space access role’s trust policy for thecloudwatch.amazonaws.comprincipal, all three ofsts:AssumeRole,sts:TagSession, andsts:SetContext(assume-role alone is accepted and fails in use), and both confused-deputy conditions —aws:SourceAccountequal to the account,aws:SourceArnmatching thecloudwatchspace/resource with Region and space ID wildcarded, because the space ID does not exist when the role is created; a successful create is not proof the role is right, so read the Space back withget-spaceto verify; and recreating the Space is not the first remedy — a new Space with the same role fails the same way.
Two decisions change which operations you call. Settle both before touching anything.
Who owns the Domain:
| The customer wants | Path |
|---|---|
| A Domain for a single account | Account-scoped — create-domain, this file |
| A Domain shared across an AWS Organization | Org-scoped — create-domain-for-organization, management account only. See references/cloudwatch-omni/org-domains.md |
Which authorization provider:
| The customer authenticates with | identityProviders | Extra setup |
|---|---|---|
| IAM identities only | ["IAM"] | None |
| IAM Identity Center | ["IAM", "IDC"] | An Identity Center instance ARN, and the Domain MUST be created in that instance’s primary Region |
Constraints:
create-domain and
create-domain-for-organization produce different Domains with different
deletion operations.d- followed by up to 25 lowercase alphanumerics, a
domain ARN, and a domain endpoint URL. The endpoint URL derives from the
domain name, which is why the name is not cosmetic.Quotas to state up front:
create-domain conflicts.Constraints:
iam:PassRole on the space access role, for whoever creates the Space.
Without it the create fails after the role already exists.Region rules differ by provider, and the difference matters:
references/cloudwatch-omni/org-domains.md.Never answer “yes, Spaces can be in other Regions” without naming which provider that holds for. It is true for IAM-only and false for account-scoped Identity Center.
Constraints:
references/cloudwatch-omni/org-domains.md.Every operation runs through the aws___call_aws tool as an aws <service> <operation> CLI command — the service name in kebab-case, the operation in
kebab-case, and each input-shape member passed as a --kebab-key flag. Policy
documents appear below as standalone JSON blocks — pass each as the corresponding
flag value (--assume-role-policy-document).
aws cloudwatchomni operations:
| Intent | Command |
|---|---|
| Check whether a Domain already exists | aws cloudwatchomni list-domains |
| Create the Domain | aws cloudwatchomni create-domain |
| Read the Domain back | aws cloudwatchomni get-domain |
| Check whether a Space already exists | aws cloudwatchomni list-spaces |
| Create the Space | aws cloudwatchomni create-space |
| Read the Space back | aws cloudwatchomni get-space |
| Delete the Space | aws cloudwatchomni delete-space |
| Delete the Domain | aws cloudwatchomni delete-domain |
aws iam operations:
| Intent | Command |
|---|---|
| Create the space access role | aws iam create-role |
| Attach a managed policy | aws iam attach-role-policy |
| Detach a managed policy | aws iam detach-role-policy |
| Delete the role | aws iam delete-role |
aws sso-admin operations:
| Intent | Command |
|---|---|
| Find the customer’s Identity Center instances | aws sso-admin list-instances |
aws sts operations:
| Intent | Command |
|---|---|
| Confirm the calling identity | aws sts get-caller-identity |
Report the caller identity before starting:
aws___call_aws → aws sts get-caller-identityAsk these in one message and wait for answers. Do not ask piecemeal, and do not default silently.
create-space requires this as well as the
space access role, and the agent cannot create it. Ask for an existing ARN, or
offer to look for a reusable one — see Step 3.ENCRYPT_DECRYPT key in the
caller’s account and Region, and its key policy must allow
cloudwatch.amazonaws.com to perform kms:Decrypt and kms:GenerateDataKey.Confirm the choices back in one line, then execute.
If the customer does not know their Identity Center instance ARN, look it up for them rather than sending them away to find it:
aws___call_aws → aws sso-admin list-instancesEach entry carries the instance ARN and its identity store ID. The public reference for this operation is sso-admin list-instances (opens in a new tab).
Constraints:
One Domain per account. A second create-domain in the same account
conflicts, so check before creating and tell the customer the limit.
aws___call_aws → aws cloudwatchomni list-domainsIf a Domain comes back, do NOT create another. Report the existing Domain to the
customer — its name and ID from the summary, and its endpoint URL from
get-domain --domain-id <id> (summaries do not carry it) — and confirm they want to use it. On
confirmation, take its domainId and continue from Step 3.
Constraints:
create-domain.For an IAM-only Domain there is no extra configuration:
aws___call_aws → aws cloudwatchomni create-domain --name <domain-name> --identity-providers IAMFor an Identity Center Domain, identityProviderConfiguration becomes
required and carries the instance ARN. The Domain must be created in the
instance’s primary Region, which the service verifies:
aws___call_aws → aws cloudwatchomni create-domain --name <domain-name> --identity-providers IAM IDC --identity-provider-configuration '{"identityCenterConfiguration": {"identityCenterInstanceArn": "<idc-instance-arn>"}}'If the customer does not have the Identity Center instance ARN to hand, do not leave
<idc-instance-arn> as a placeholder for them to fill — offer to look it up with
list-instances on sso-admin (see Step 1),
which returns each instance’s ARN and identity store ID.
Capture domainId, domainArn, and domainEndpointUrl from the domain object
in the response. domainId is the input to create-space in Step 4.
Constraints:
identityProviders accepts one or two values from IAM and IDC, and they must
be distinct. You MUST NOT send a repeated value.domainEndpointUrl and give it to the customer. It is how they
reach Omni, and it is not derivable from the domain name alone.list-instances on sso-admin rather than leaving
<idc-instance-arn> for them to resolve.Only if the customer chose “create a new role” in Step 1. If they supplied an existing role ARN, do NOT modify it — pass it straight through to Step 4 and, if the create fails, use the Troubleshooting table rather than editing their role.
This is the role the service assumes to operate on the Space.
Constraints:
list-spaces check from Step 4
before creating a role. If a Space already exists in the target Region, no role
is needed, and creating one leaves an unused IAM role behind.Call create-role with AssumeRolePolicyDocument set to this trust policy. The
aws:SourceAccount and aws:SourceArn conditions are confused-deputy
protection — keep both.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "cloudwatch.amazonaws.com"
},
"Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetContext"],
"Condition": {
"StringEquals": {
"aws:SourceAccount": "<account-id>"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:cloudwatch:*:<account-id>:space/*"
}
}
}
]
}aws___call_aws → aws iam create-role --role-name <role-name> --assume-role-policy-document <trust-policy-json>Constraints:
sts actions are required. sts:TagSession and sts:SetContext carry
the session context the Space needs; a role trusted for sts:AssumeRole alone
is accepted at create time and fails later when the Space is used.aws:SourceArn pattern MUST stay wildcarded across Region and space ID
(space/*). The space ID does not exist yet when the role is created, so a
narrower pattern cannot match.create-space
rejects a dataAccessRoleArn from another account.Attach both with the IAM attach-role-policy operation. Do not write an inline
policy instead, and do not substitute a *FullAccess policy — the Space’s
permissions are defined by these two managed policies and nothing else. If the role
was supplied by the customer rather than created here, ask before attaching anything
to it.
aws___call_aws → aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/CloudWatchOmniSpaceAccessPolicy
aws___call_aws → aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/CloudWatchOmniAgentObservabilityPolicyConstraints:
*FullAccess policy or hand-write a broad inline
policy. The Space’s permissions are defined by these managed policies.create-space requires a second role ARN, agentCoreEvaluationRoleArn, which is
not the space access role and is not created here. Resolve it now rather than at the
call:
AgentCoreEvaluationRole or AgentCoreEvalRole, or whose trust policy principal
is bedrock-agentcore.amazonaws.com. An account already running evaluations
usually has one — show the matches and let the customer pick.aws___call_aws → aws iam list-rolesauto_create_execution_role=True) — then use the ARN it returns.Constraints:
create-space. Omitting either is
rejected by the client before the request is sent.When the customer asks what this role is — typically because create-space just
asked them for it mid-setup — answer with all of the following, because each one
changes what they do next: it is a second, separate role from the space access
role and is not created as part of Space setup; offer to reuse one first by
listing roles and surfacing matches on the name prefixes or the
bedrock-agentcore.amazonaws.com trust principal, so they can pick; otherwise they
create it through the AgentCore Evaluations console’s new-service-role option or the
AgentCore CLI/SDK auto-create option and bring back the ARN; both role ARNs must
be in hand before create-space, since omitting either is rejected by the client
before any request is sent; you will not hand-write its IAM policy (the console
and CLI build the authoritative one); and you will not attach the Space’s managed
policies to it, because it is a different role with a different trust principal.
One Space per account per Region. A second Space in the same Region conflicts; a Space in a different Region is a different Space. Check before creating, scope the check to the target Region, and tell the customer the limit.
aws___call_aws → aws cloudwatchomni list-spaces --domain-id <domain-id>If a Space already exists in the target Region, do NOT create another. Report it to the customer — its name, ID, and Region — and confirm they want to use it.
Constraints:
create-space.
create-spacetakes TWO required role ARNs, not one. Alongside--data-access-role-arnit requires--agent-core-evaluation-role-arn. Both must be in the caller’s account, and omitting either is rejected by the client before the request is sent. If you do not already have both from Step 1 and Step 3, resolve the AgentCore evaluation role there before calling.
aws___call_aws → aws cloudwatchomni create-space --name <space-name> --domain-id <domain-id> \
--data-access-role-arn arn:aws:iam::<account-id>:role/<role-name> \
--agent-core-evaluation-role-arn arn:aws:iam::<account-id>:role/<agentcore-eval-role-name>--domain-id accepts the Domain’s ID, its name, or its ARN.
For customer managed encryption, add encryptionConfiguration:
"encryptionConfiguration": {"encryptionStrategy": "CUSTOMER_MANAGED", "kmsKeyArn": "<kms-key-arn>"}Omit encryptionConfiguration entirely for service-owned encryption; that is
equivalent to "encryptionStrategy": "AWS_OWNED".
The customer managed key’s key policy must allow the service to use it. This statement belongs on the key, not on the space access role:
{
"Effect": "Allow",
"Principal": {
"Service": "cloudwatch.amazonaws.com"
},
"Action": ["kms:Decrypt", "kms:GenerateDataKey"],
"Resource": "*"
}Capture spaceId, spaceArn, region, and status from the space object in
the response.
A successful create does not mean the role works.
create-spacechecks only thatdataAccessRoleArnis present and in the caller’s account — it never tries to assume it. A role with a wrong trust policy produces a Space that is created successfully and fails the moment anything uses it.
Constraints:
encryptionConfiguration; it is the only create-time encryption
input. create-space has no top-level kmsKeyArn member (the CLI has no
--kms-key-arn flag). There is no top-level kmsKeyArn on read either:
get-space always reports encryptionConfiguration, and list-spaces summaries
carry no encryption fields at all.ENCRYPT_DECRYPT key in the caller’s
account and Region.create-space does not verify that the role can actually be assumed. It checks
only that the ARN is present and in the caller’s account, so a role with a wrong
trust policy creates a Space that fails in use. You MUST NOT treat a successful
create as proof the role is correct — verify per the next section.Read both resources back rather than trusting the create responses:
aws___call_aws → aws cloudwatchomni get-domain --domain-id <domain-id>
aws___call_aws → aws cloudwatchomni get-space --space-id <space-id>Confirm the Domain reports the expected endpoint URL, and the Space reports
ACTIVE in the expected Region with the role ARN you passed.
Then report to the customer, in one line: the domain endpoint URL, the space ID, and the Region.
Constraints:
ACTIVE.Before any delete step, in every answer from this section: say plainly that deleting a Space destroys its telemetry and the deletion cannot be undone, and ask the customer to confirm before you run anything. State this first, ahead of the ordering and the API details — an answer that gets the order right but never warns the customer is wrong.
Confirm with the customer before running anything in this section. Deleting a Space is destructive and its telemetry goes with it.
Order matters. Delete the Space first.
Confirm first, then delete the Space. Deleting a Space is destructive: the Space’s telemetry goes with it and cannot be recovered. Get the customer’s explicit confirmation before running this call.
aws___call_aws → aws cloudwatchomni delete-space --space-id <space-id>Delete the Domain. This fails while any Space still exists under it:
aws___call_aws → aws cloudwatchomni delete-domain --domain-id <domain-id>The space access role — branch on who created it:
detach-role-policy, then delete-role. A role with policies attached
cannot be deleted.Constraints:
delete-domain returns a conflict
naming the number of remaining Spaces, and the fix is always to delete those
first — never to retry the Domain delete.get-space after delete-space to confirm removal. Once
delete returns success the Space is gone, and a follow-up read can fail in ways
that look like the delete did not work.Rule: When a call returns an error, surface the error code and message verbatim, then map to the mitigation below. Do NOT invent error text, do NOT paraphrase what the service returned, and do NOT synthesize a mitigation for an error that is not listed.
| Error signal | Cause | Mitigation to surface |
|---|---|---|
ConflictException on create-domain | A Domain already exists in this account, or an org Domain covers it | Read it with list-domains, confirm with the customer, and continue from Step 3. Do NOT retry the create |
ConflictException on create-space | A Space already exists in this account and Region | Read it with list-spaces and compare Regions. A different Region is a different Space |
ValidationException naming the Region on create-domain | Identity Center Domain not being created in the instance’s primary Region | Ask the customer for the instance’s primary Region and create the Domain there. list-instances on sso-admin shows the instances they can reach |
ValidationException on create-space naming the Domain’s Region | Space Region differs from the Domain’s, and the Domain uses Identity Center | Create the Space in the Domain’s Region. If the customer needs Identity Center across several Regions, the org-scoped Domain path supports that |
ValidationException that dataAccessRoleArn must be in the caller’s account | Role ARN belongs to a different account | Ask the customer for a role in this account; do NOT attempt a cross-account role |
ParamValidationError naming a missing required parameter on create-space | agentCoreEvaluationRoleArn (or dataAccessRoleArn) was omitted — both are required and the client rejects the call before sending | Resolve the AgentCore evaluation role as described in Step 4 and pass both ARNs |
ValidationException on encryptionConfiguration | encryptionStrategy and kmsKeyArn inside encryptionConfiguration do not agree | Send the key ARN nested inside encryptionConfiguration only with encryptionStrategy CUSTOMER_MANAGED; omit the block entirely for AWS_OWNED |
AccessDeniedException naming iam:PassRole | Caller may create the Space but cannot pass the role | Ask the customer to add iam:PassRole on that specific role ARN to their calling identity |
AccessDeniedException on the operation itself | Caller lacks cloudwatch:CreateDomain or cloudwatch:CreateSpace | Ask the customer to add the specific action to their calling role |
ResourceNotFoundException on create-space | domainId typo, or the Domain is in another account or Region | Re-read the Domain with list-domains and use the exact domainId |
ConflictException on delete-domain naming remaining Spaces | Spaces still exist under the Domain | Delete every Space first, then retry. Do NOT retry the Domain delete on its own |
If the error does not match a row above, quote it verbatim, say it is unmapped, and ask the customer how to proceed.
An authorization error after a successful create-space almost always means the
space access role’s trust policy is wrong, not that access is missing.
create-space validates only that dataAccessRoleArn is non-blank and in the
caller’s account. It never attempts to assume the role, so a role missing
sts:TagSession or sts:SetContext, naming the wrong service principal, or carrying
condition keys that do not match is accepted at create time and fails in use. Note
the asymmetry: the organization path does verify its role by performing a real
sts:AssumeRole at create time, so a reader who did that flow first will reasonably
expect this one to validate too.
Telling this apart from a missing grant is a matter of timing:
sts actions, and both condition keys.references/cloudwatch-omni/access-grants.md.You MUST check the trust policy before proposing new grants. Creating a grant to fix a trust-policy problem leaves the customer with the same error and two things to unwind.
What to check on the space access role, against the trust policy in Step 3:
cloudwatch.amazonaws.com — not an account or a
user.sts:AssumeRole, sts:TagSession, and
sts:SetContext. A role trusted for sts:AssumeRole alone is the classic case:
accepted at create time, denied the first time the Space assumes it.aws:SourceAccount equal
to the account, and aws:SourceArn matched with ArnLike against
arn:aws:cloudwatch:*:<account-id>:space/* — Region and space ID wildcarded.
A condition that spells out the space ID cannot have been right when the role was
created, because the ID did not exist yet; a condition pinned to one Region breaks
the moment a Space is created elsewhere.Then read the Space back with get-space and confirm it reports ACTIVE with the
role ARN you expect. A successful create-space response is not evidence the role is
correct — the create never exercised it — so the read-back after the fix is the check.
Do not recreate the Space as the first remedy. The role is what is wrong; a new Space created with the same role fails the same way, and if telemetry has started arriving the delete is destructive. Fix the trust policy on the existing role instead.
create-space succeeded but using the Space fails with an authorization
error. The create does not verify the role is assumable. Check the trust
policy’s principal, all three sts actions, and both condition keys.cloudwatch.amazonaws.com the required KMS actions. Nothing about
the Space or the role is wrong.aws:SourceAccount and aws:SourceArn conditions on the space access
role’s trust policy. They are the confused-deputy protection.aws:SourceArn wildcard covers Region and space ID because neither is known
when the role is created. Do not widen it further — the account ID and the
space/ resource type MUST stay pinned.*FullAccess policies to the space access role. Its permissions
come from the Omni managed policies and nothing else.kms:Decrypt and kms:GenerateDataKey. Do not author or modify the key
policy; if it needs changing, direct the customer to their key administrator.references/cloudwatch-omni/org-domains.md — Domains shared across an AWS Organizationreferences/cloudwatch-omni/instrumentation/collector.md — deploying a collector that exports to
CloudWatch’s OTLP endpoints, for telemetry not yet reaching CloudWatchreferences/cloudwatch-omni/data-forwarding-and-centralization.md — forwarding telemetry that is
already in CloudWatch log groups into the Dataset