Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.37
references/cloudwatch-omni/org-domains.mdMarkdown33 KBView on GitHub
An org-scoped Domain is created once from the organization’s management account and shared by every member account. Spaces are then created in member accounts under it. It is a separate operation family from the account-scoped Domain, with a different required role, a different deletion operation, and a prerequisite that lives in AWS Organizations rather than in Omni.
Always state these, in any answer drawn from this file:
- Trusted access for
observabilityadmin.amazonaws.commust be enabled in AWS Organizations first, and because that is an organization-wide change you MUST ask the customer before enabling it — even if they already said to proceed.create-domain-for-organizationis management account only; a registered delegated administrator can do everything else but not that. An account-scoped Domain on the management account blocks an org Domain and must be deleted first, which is the customer’s decision.- A Domain grant is
ADMINonly and reaches every Space in the Domain, including future ones. PreferIDC_GROUP; do not useIAM_ROOTunless asked.- Teardown order: every Space first, then the Domain, from its home Region, using
delete-domain-for-organization(delete-domainrejects an org Domain). Deleting the Domain removes its grants, so do not revoke them first.
| The customer wants | Path |
|---|---|
| A Domain shared across an AWS Organization | Org-scoped — this file |
| A Domain for a single account | Account-scoped — see references/cloudwatch-omni/spaces-and-domains.md |
The two are mutually exclusive on the same account. A Domain of either scope already on the management account blocks creating the other.
Who may call what differs across the org operations. Establish the caller’s account up front and match it against this table before calling anything:
| Operation | Eligible caller |
|---|---|
enable-aws-service-access on organizations | Management account only |
create-domain-for-organization | Management account only |
Every other *-for-organization operation | Management account, or a registered delegated administrator |
create-space | The member account that will own the Space, or the management account using credentials vended for that account |
Constraints:
create-domain-for-organization. A delegated administrator can
do everything else but neither of those.create-domain or delete-domain on an org Domain. Those are
the account-scoped operations and the service rejects the mismatch.d- followed by up to 25 lowercase alphanumerics, a
domain ARN, and a domain endpoint URL derived from the domain name.An org Domain ARN uses the resource type organization-domain, where an
account-scoped Domain uses domain. That difference is how you tell them apart in
a listing.
Quotas and scope rules to state up front:
ADMIN.
There is no read-only or scoped variant at the Domain level; narrower access is
granted per Space.Constraints:
ADMIN-only. A customer who expects it to be
limited to one account or one Space has misunderstood the blast radius.domainAccessRoleArn. See
Step 3.Region rules:
Constraints:
Every operation runs through the aws___call_aws tool as an aws <service> <operation> CLI command, with the operation’s inputs passed as CLI flags
(--kebab-key). Policy documents appear below as standalone JSON blocks — pass each
as the corresponding flag value (--assume-role-policy-document).
aws organizations operations:
| Intent | Command |
|---|---|
| Check whether trusted access is enabled | aws organizations list-aws-service-access-for-organization |
| Enable trusted access | aws organizations enable-aws-service-access |
aws cloudwatchomni operations:
| Intent | Command |
|---|---|
| Discover which Domains exist, and of which scope | aws cloudwatchomni list-domains |
| Create the Domain | aws cloudwatchomni create-domain-for-organization |
| Read the Domain back | aws cloudwatchomni get-domain-for-organization |
| Change the Domain’s name or providers | aws cloudwatchomni update-domain-for-organization |
| Delete the Domain | aws cloudwatchomni delete-domain-for-organization |
| List every Space across the organization | aws cloudwatchomni list-spaces-for-organization |
| Vend credentials for a member account | aws cloudwatchomni get-space-credentials-for-organization |
| Create a Space | aws cloudwatchomni create-space — requires BOTH --data-access-role-arn and --agent-core-evaluation-role-arn; see references/cloudwatch-omni/spaces-and-domains.md |
| Grant a principal Domain administration | aws cloudwatchomni create-domain-access-grant-for-organization |
| List Domain grants | aws cloudwatchomni list-domain-access-grants-for-organization |
| Read one Domain grant | aws cloudwatchomni get-domain-access-grant-for-organization |
| Revoke one Domain grant | aws cloudwatchomni delete-domain-access-grant-for-organization |
aws iam operations:
| Intent | Command |
|---|---|
| Create the domain access role | aws iam create-role |
| Attach a managed policy | aws iam attach-role-policy |
| Detach a managed policy | aws iam detach-role-policy |
| Delete the role | aws iam delete-role |
aws sso-admin operations:
| Intent | Command |
|---|---|
| Find the customer’s Identity Center instances | aws sso-admin list-instances |
aws sts operations:
| Intent | Command |
|---|---|
| Confirm the calling identity and account | aws sts get-caller-identity |
Report the caller identity before starting, and confirm the account is the management account:
aws___call_aws → aws sts get-caller-identityAsk these in one message and wait for answers. Do not ask piecemeal, and do not default silently.
ADMIN to in Step 6.Confirm the choices back in one line, then execute.
If the customer does not know their Identity Center instance ARN, look it up for them rather than sending them away to find it:
aws___call_aws → aws sso-admin list-instancesEach entry carries the instance ARN and its identity store ID. The public reference for this operation is sso-admin list-instances (opens in a new tab).
Constraints:
Omni’s organization operations depend on CloudWatch Observability Admin having
trusted access in AWS Organizations. The service principal is
observabilityadmin.amazonaws.com.
aws___call_aws → aws organizations list-aws-service-access-for-organizationThe response carries EnabledServicePrincipals, a list of entries each with a
ServicePrincipal and a DateEnabled. Look for observabilityadmin.amazonaws.com
among them. This call works from the management account or a registered delegated
administrator. The public reference is
organizations list-aws-service-access-for-organization (opens in a new tab).
Ask the customer before enabling — this changes an organization-wide setting.
Present the enable-aws-service-access command and get the customer’s explicit
confirmation before running it; do not enable trusted access as an implied step of
the overall setup, because it affects every account in the organization.
aws___call_aws → aws organizations enable-aws-service-access --service-principal observabilityadmin.amazonaws.comThis call is management account only and returns no body. Re-run the check to confirm the principal now appears.
Constraints:
observabilityadmin.amazonaws.com. Do NOT use
telemetry-pipelines.observabilityadmin.amazonaws.com — that is a different
principal for a different purpose and enabling it does not grant trusted access.This is the role the service assumes to operate on Spaces across the organization.
It MUST live in the management account, and unlike the account-scoped Space
role it is verified at create time — the service performs a real
sts:AssumeRole against it while handling create-domain-for-organization, and
the create fails if that does not succeed.
The ARN must match arn: followed by the partition, then
:iam::<management-account-id>:role/<role-name>, and be 20–2048 characters.
Call create-role with AssumeRolePolicyDocument set to this trust policy. The
aws:SourceAccount and aws:SourceArn conditions are confused-deputy
protection — keep both. Note the resource type is organization-domain, not
domain.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "cloudwatch.amazonaws.com"
},
"Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetContext"],
"Condition": {
"StringEquals": {
"aws:SourceAccount": "<management-account-id>"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:cloudwatch:*:<management-account-id>:organization-domain/*"
}
}
}
]
}aws___call_aws → aws iam create-role --role-name <role-name> --assume-role-policy-document <trust-policy-json>aws___call_aws → aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/CloudWatchOmniDomainAccessPolicyConstraints:
aws:SourceArn pattern MUST use the organization-domain resource type and
MUST stay wildcarded across Region and domain ID. The domain ID does not exist
yet when the role is created, so a narrower pattern cannot match.*FullAccess policy or hand-write a broad inline
policy. The Domain’s permissions are defined by the managed policy above.create-domain-for-organization
provisions one. The role ARN is a required input.list-domains reports the Domains visible to the caller, of both scopes, and each
entry carries its ARN. Use the ARN’s resource type to tell them apart —
organization-domain for an org Domain, domain for an account-scoped one. This
is the check to run when the customer has no domain ID yet.
aws___call_aws → aws cloudwatchomni list-domainsRead the result before creating anything:
organization-domain entry exists — the org Domain is already set up.
Report its name and ID from the summary and its endpoint URL from
get-domain-for-organization (summaries do not carry it), confirm the customer wants to use it, and
continue from Step 5 with its domainId.domain entry exists — an account-scoped Domain occupies the
management account, and it blocks the org Domain. Tell the customer it must be deleted before an org Domain can
be created, and that deleting it is their decision. Then stop. Do not create
anything, and do not describe the situation merely as blocking.Constraints:
create-domain-for-organization.For an IAM-only Domain:
aws___call_aws → aws cloudwatchomni create-domain-for-organization \
--name <domain-name> --identity-providers IAM \
--domain-access-role-arn arn:aws:iam::<management-account-id>:role/<role-name>For an Identity Center Domain, identityProviderConfiguration becomes
required and carries the instance ARN:
aws___call_aws → aws cloudwatchomni create-domain-for-organization \
--name <domain-name> --identity-providers IAM IDC \
--domain-access-role-arn arn:aws:iam::<management-account-id>:role/<role-name> \
--identity-provider-configuration '{"identityCenterConfiguration": {"identityCenterInstanceArn": "<idc-instance-arn>"}}'Capture domainId, domainArn, domainEndpointUrl, organizationId, and
ownerAccountId from the organizationDomain object in the response (the same
object get-domain-for-organization and update-domain-for-organization return).
Constraints:
domainEndpointUrl and give it to the customer. Every member
account reaches Omni through it.A Space lives in a member account. There are two ways to create one, and the choice was made in Step 1.
The Space itself — its name, its space access role, its encryption — works exactly
as it does under an account-scoped Domain. See
references/cloudwatch-omni/spaces-and-domains.md, picking up at its Space steps. The space
access role is still required and still belongs in the account that will own the
Space.
The member account calls create-space directly, passing the shared domainId.
Nothing about the call differs from the account-scoped path except that the Domain
was not created by that account.
The management account, or a registered delegated administrator, vends credentials for the target account and uses them to create the Space:
aws___call_aws → aws cloudwatchomni get-space-credentials-for-organization \
--context '{"domainId": "<domain-id>", "targetAccountId": "<target-account-id>"}' \
--credential-type SPACE_OPERATION--credential-type is required, and SPACE_OPERATION is its only value — always
pass it, and do not treat it as a choice to put to the customer. Omitting it is rejected
by the client before the request is sent.
The response carries temporary credentials for the target account. Use them to call
create-space against that account. Once a Space exists, credentials can be vended
for it directly by passing spaceId in the context instead of the
domainId/targetAccountId pair.
Enumerate what exists across the organization at any point:
aws___call_aws → aws cloudwatchomni list-spaces-for-organizationConstraints:
create-domain first. The org Domain
already exists and a second Domain in the member account is neither needed nor
permitted alongside it.create-space call itself.
Calling create-space with the management account’s own credentials creates a
Space in the management account, not the target account.A Domain with no grants has no administrators. Grant the principals the customer named in Step 1.
A grant applies to the entire Domain. It conveys administration of every Space in the Domain, including Spaces created later. There is no way to scope a Domain-level grant to one account or one Space — that is what per-Space grants are for.
aws___call_aws → aws cloudwatchomni create-domain-access-grant-for-organization \
--domain-id <domain-id> \
--name <grant-name> \
--principal '{"principalType": "<principal-type>", "principalId": "<principal-id>"}' \
--permission ADMINname is required: 1–64 characters matching ^[a-zA-Z0-9_-]+$. permission takes
the single value ADMIN. principalType is one of IDC_USER, IDC_GROUP,
IAM_USER, IAM_ROLE, or IAM_ROOT.
Capture the grant ID from the accessGrant object; it is what
delete-domain-access-grant-for-organization takes if a single grant needs
revoking later.
Review the grants with:
aws___call_aws → aws cloudwatchomni list-domain-access-grants-for-organization --domain-id <domain-id>Constraints:
permission is a single value and it must be ADMIN. The account-scoped
permission values — READ, READ_WRITE_DELETE, SPACE_ADMIN, and CUSTOM — are
not valid at the Domain level.IDC_GROUP over IDC_USER.
Group membership changes without touching grants.IAM_ROOT unless the customer explicitly asks. It is the
broadest principal available and rarely what they mean.Read the Domain back rather than trusting the create response, and confirm the Spaces and grants landed:
aws___call_aws → aws cloudwatchomni get-domain-for-organization --domain-id <domain-id>
aws___call_aws → aws cloudwatchomni list-spaces-for-organization
aws___call_aws → aws cloudwatchomni list-domain-access-grants-for-organization --domain-id <domain-id>Then report to the customer, in one line: the domain endpoint URL, the domain ID, the organization ID, how many Spaces exist, and how many administrators were granted.
Constraints:
ADMIN grant exists before declaring setup
complete. A Domain nobody administers is not a working setup.Order matters, and it spans accounts.
Every Space is deleted. The Domain cannot be deleted while any Space is
associated with it. Use list-spaces-for-organization to enumerate what
remains. A Space is deleted by its owning account, or by the management account
using credentials vended through get-space-credentials-for-organization.
Delete the Domain, from its home Region — the Region it was created in:
aws___call_aws → aws cloudwatchomni delete-domain-for-organization --domain-id <domain-id>Use delete-domain-for-organization, not delete-domain — the account-scoped
operation rejects an org Domain and tells you to use this one instead.
Deleting the Domain removes its access grants. Do not revoke them first as
hygiene; it is unnecessary work. delete-domain-access-grant-for-organization is
for revoking one principal’s access while the Domain stays in use.
The domain access role — branch on who created it:
detach-role-policy, then delete-role. A role with policies attached cannot
be deleted.Trusted access can be left enabled. If the customer wants it off, that is an organization-wide change made through CloudWatch’s own telemetry configuration controls rather than by disabling the Organizations integration directly.
Constraints:
delete-domain-for-organization. delete-domain rejects an org
Domain and tells you to use this operation instead.Rule: When a call returns an error, surface the error code and message verbatim, then map to the mitigation below. Do NOT invent error text, do NOT paraphrase what the service returned, and do NOT synthesize a mitigation for an error that is not listed.
| Error signal | Cause | Mitigation to surface |
|---|---|---|
| Authorization error stating trusted access is disabled | CloudWatch Observability Admin trusted access is not enabled for the organization | Run Step 2 — check with list-aws-service-access-for-organization and enable observabilityadmin.amazonaws.com. Nothing else works until it is on |
AccessDeniedException on enable-aws-service-access | Caller is not in the management account | Only the management account can enable trusted access. A delegated administrator can read the setting but not change it |
| Authorization error stating only the management account can create an organization domain | Caller is a member account or a delegated administrator | Domain creation must run from the management account. A delegated administrator can perform every other org operation |
ParamValidationError naming a missing required parameter on get-space-credentials-for-organization | credentialType was omitted — it is required, and the client rejects the call before sending it | Resend with --credential-type SPACE_OPERATION |
ValidationException that the management account ID could not be resolved | Caller is not in an organization, or trusted access is off | Confirm the account belongs to an organization and that trusted access is enabled |
ValidationException that domainAccessRoleArn must be a role in the management account | Role lives in a member account | Create the role in the management account. A member-account role cannot be used |
| An error stating the data access role could not be assumed, naming the trust policy | The create-time assume check failed | Fix the role’s trust policy so the CloudWatch service can assume it, and confirm the aws:SourceArn uses the organization-domain resource type. Do NOT retry unchanged — it fails identically |
ConflictException naming a management account that already has a Domain | The management account already has a Domain. This fires whether that Domain is org-scoped or account-scoped — the two are mutually exclusive on one account | Run list-domains and read the ARN resource type. If it is organization-domain, use it. If it is domain, the account-scoped Domain must be deleted before an org Domain can be created. Do NOT retry the create |
ValidationException naming the Identity Center primary Region | Domain not being created in the instance’s primary Region | Create the Domain in the primary Region the message names. list-instances on sso-admin shows the instances the caller can reach |
ValidationException that Identity Center is not replicated to a Region | A Space was requested in a Region where the Identity Center instance is not replicated | Either create the Space in a Region where Identity Center is replicated, or have the customer replicate Identity Center to that Region |
| Error stating Spaces for organization domains must be created in the same Region as the organization domain | The target Region is not available for Spaces under this Domain | Create the Space in the Domain’s Region, or ask the customer which Regions are enabled for Spaces in their organization |
| Error stating cross-account access requires management account or registered delegated administrator | Caller is an ordinary member account | Run the operation from the management account, or register the account as a delegated administrator |
| A Space appears in the management account instead of the target account | create-space was called with the management account’s own credentials rather than the vended ones | Delete that Space, re-vend credentials with get-space-credentials-for-organization, and call create-space with those |
ResourceNotFoundException that the Domain is not an organization-scoped domain | delete-domain-for-organization was called on an account-scoped Domain | Use delete-domain for that Domain |
ValidationException that an organization-scoped domain cannot be deleted via DeleteDomain | Wrong deletion operation | Use delete-domain-for-organization |
| Error stating the Domain must be deleted from its home Region | Delete attempted from another Region | Retry from the Region named in the message |
ConflictException stating Spaces are still associated with the Domain | Spaces still exist under the Domain | Enumerate them with list-spaces-for-organization and delete each, then retry |
If the error does not match a row above, quote it verbatim, say it is unmapped, and ask the customer how to proceed.
ADMIN over the whole Domain is the only shape. Narrower access is granted per
Space.aws:SourceAccount and
aws:SourceArn conditions on its trust policy, keep the organization-domain
resource type pinned, and keep the role out of any workflow that does not need
it.IDC_GROUP over IDC_USER for Domain administrators so that offboarding
a person does not require finding and revoking their grant.IAM_ROOT as a grant principal. Root has no session identity to attribute
actions to.get-space-credentials-for-organization are temporary and act
in another account. Do not persist or log them, and do not reuse them beyond the
operation they were vended for.list-domain-access-grants-for-organization periodically.references/cloudwatch-omni/spaces-and-domains.md — account-scoped Domains, and the Space steps
used under an org Domainreferences/cloudwatch-omni/instrumentation/collector.md — deploying a collector that exports to
CloudWatch’s OTLP endpoints, for telemetry not yet reaching CloudWatchreferences/cloudwatch-omni/data-forwarding-and-centralization.md — forwarding telemetry that is
already in CloudWatch log groups into the Dataset