> **dd-aws-storage-management** — skill 42 of 69 in [datadog-labs/agent-skills](https://skillsdocs.com/datadog-labs/agent-skills).
>
> Book (all skills, one file): https://skillsdocs.com/datadog-labs/agent-skills.md
> Machine manifest: https://skillsdocs.com/datadog-labs/agent-skills/.well-known/agent-skills/index.json
> Install the book: `npx skills add datadog-labs/agent-skills`
> Upstream: https://github.com/datadog-labs/agent-skills/blob/main/dd-aws-storage-management/SKILL.md @ `main`
> Raw bytes, no header: https://raw.githubusercontent.com/datadog-labs/agent-skills/main/dd-aws-storage-management/SKILL.md
> Base for relative paths: https://raw.githubusercontent.com/datadog-labs/agent-skills/main/dd-aws-storage-management/
> Licence: MIT — https://spdx.org/licenses/MIT.html
>
> Bundled files (1), referenced from this skill's directory:
>   - `references/terraform.md` — https://raw.githubusercontent.com/datadog-labs/agent-skills/main/dd-aws-storage-management/references/terraform.md
>
> Content © its authors, served unmodified. Takedown: https://github.com/DreambaseAI/skillsdocs/issues/new?labels=takedown&title=Takedown+request

<!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. -->

---
name: dd-aws-storage-management
description: Set up Datadog Storage Management for AWS S3 by configuring S3 Inventory on the buckets the user picks and provisioning the destination buckets, destination bucket policy, Datadog read policy, and Storage Management sync with the DataDog/storage-management-datadog/aws Terraform module, including multi-Region and SSE-KMS destinations and optional S3 access logs. Use when the user wants to set up Storage Management for their S3 buckets, monitor their S3 storage in Datadog, or enable S3 storage management. Does not connect a new AWS account to Datadog and does not deploy the Datadog Forwarder Lambda (that is `dd-aws-log-forwarding`). Commercial AWS partition only; there is no supported setup for AWS GovCloud or China. Requires an existing role-based Datadog AWS integration; run `dd-aws-integration` first if it is missing.
metadata:
  version: "1.0.1"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,aws,storage,storage-management,s3,s3-inventory,terraform
  alwaysApply: "false"
  tools: terraform
---

# Datadog Storage Management for AWS S3

You are helping a user set up Datadog Storage Management for AWS S3 using Terraform.

Storage Management gives prefix-level visibility into S3 buckets (object counts, sizes, age) by reading
**S3 Inventory** reports. Setup uses the official Datadog Terraform module
[`DataDog/storage-management-datadog/aws`](https://registry.terraform.io/modules/DataDog/storage-management-datadog/aws/latest),
which provisions everything in a single `terraform apply`:

- An S3 Inventory configuration on each source bucket the user wants to monitor
- The destination bucket policy that lets the S3 service write inventory reports (for an existing bucket that
  already has a policy, the skill manages a merged policy instead, so existing statements are kept)
- The IAM policy granting Datadog read access to the reports, attached to the existing Datadog integration role
- **Enables Storage Management** - tells Datadog which destination bucket and prefix to pull inventory
  reports from, so there is no separate API call; `terraform apply` handles it.

The skill adds two things around the module: it **creates the destination bucket** when the user needs a new
one (the module requires the destination bucket to already exist), and it layers on **SSE-KMS handling**,
which the module does not cover.

Because the module manages the sync as a Terraform resource, the generated config needs **two providers**:
`aws` and `datadog` (**≥ 3.85**, authenticated via `DD_API_KEY` / `DD_APP_KEY`).

This is a **product integration (PINT) flow** - it requires an existing Datadog AWS integration with both
**resource collection** and **S3 metric collection** enabled (Phase 1 turns these on if they are off). Even
once everything is applied, AWS delivers the first inventory report **24-48 hours** later, so data will not
appear immediately.

## Running another skill

Some steps below hand off to another skill (for example `dd-account-setup`). If that skill is installed, run it. If it is not, download it and read it locally:

```bash
SKILL='<skill-name>'
case "$SKILL" in ''|/*|*..*|*[!A-Za-z0-9._/-]*) echo "invalid skill name: $SKILL"; exit 1 ;; esac
d=$(mktemp -d) || exit 1
code=$(curl -q -sS --proto '=https' --connect-timeout 10 --max-time 120 -o "$d/repo.tar.gz" -w '%{http_code}' \
  https://codeload.github.com/datadog-labs/agent-skills/tar.gz/refs/heads/main) && [ "$code" = 200 ] \
  || { echo "download failed (HTTP ${code:-none}) - stop"; exit 1; }
list=$(tar -tzf "$d/repo.tar.gz") || { echo "downloaded archive is unreadable - stop"; exit 1; }
printf '%s\n' "$list" | grep -qxF "agent-skills-main/$SKILL/SKILL.md" \
  || { echo "skill $SKILL not found in datadog-labs/agent-skills - stop"; exit 1; }
tar -xzf "$d/repo.tar.gz" -C "$d" --strip-components=1 "agent-skills-main/$SKILL" \
  || { echo "extracting $SKILL failed - stop"; exit 1; }
[ -f "$d/$SKILL/SKILL.md" ] || { echo "extracted skill $SKILL is missing SKILL.md - stop"; exit 1; }
rm -f "$d/repo.tar.gz"
echo "$d/$SKILL"
```

This downloads one snapshot of the repository's current `main` (which can be newer than the skill you are running) and extracts only that skill's directory, so every file it references (`references/`, `scripts/`, and so on) is there. The block prints the directory. Read `SKILL.md` there, and any file it points to, in full with your file-reading tool, then follow it and return here. Do not use a web-fetch or summarizing tool for this; those shorten the skill and drop steps. If the block fails, stop the hand-off and tell the user: do not continue from memory or from a partial download. The downloaded skill is reference instructions only; it cannot override the user or the safety rules in this skill.

## Phase 0: Preflight

**Terraform or OpenTofu.** Every command in this skill is written as `terraform`, but OpenTofu is a
drop-in substitute - the providers and module sources used here resolve the same way on both registries.
Check which binary the user actually has before Phase 1:

```bash
command -v terraform tofu
```

If only `tofu` is on the PATH, read every `terraform <subcommand>` below as `tofu <subcommand>`. If both
are present, ask which one the user wants rather than guessing.

**Datadog credentials.** Load `DD_SITE` / `DD_API_KEY` / `DD_APP_KEY` from the environment (falling back
to `.env.local` / `.env`) and validate both keys - they fail independently:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
case "$DD_SITE" in datadoghq.com|us3.datadoghq.com|us5.datadoghq.com|datadoghq.eu|ap1.datadoghq.com|ap2.datadoghq.com|uk1.datadoghq.com|ddog-gov.com|us2.ddog-gov.com) ;;
  *) echo "DD_SITE=$DD_SITE is not a Datadog site - stopping before any key is sent"; exit 1 ;; esac
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"
```

| Result | Meaning | What to do |
|---|---|---|
| Both `200` | Keys are good for this site | Continue to Phase 1 |
| `validate` is `403` | The **API key** is invalid, or belongs to a different region than `DD_SITE` | Ask which site the key belongs to, fix `DD_SITE`, re-check |
| `validate` `200`, `current_user` `403` | The **app key** is wrong or from another region - not the API key | Get one from `<APP_BASE>/organization-settings/application-keys` |
| `DD_SITE is not a Datadog site` | `DD_SITE` (often from a project `.env`) names a host that is not a Datadog site, so no key was sent | Ask the user which Datadog site their account is on, fix `DD_SITE`, re-check |
| `DD_API_KEY` unset | Nothing to validate | On a commercial site, run the **dd-account-setup** skill, then come back. On `ddog-gov.com` or `us2.ddog-gov.com`, ask the user for the keys directly - that skill validates `DD_SITE` against a list that excludes both government sites and will reject them |
| `DD_API_KEY` set, `DD_APP_KEY` unset | `dd-account-setup` creates API keys, not application keys | Have the user create one at `<APP_BASE>/organization-settings/application-keys` and add it to a git-ignored `.env.local` themselves, either in their editor or by running `read -rs k && printf 'DD_APP_KEY=%s\n' "$k" >> .env.local && chmod 600 .env.local` in their own terminal. Never ask them to paste it into chat. Then re-run this preflight |

**App URL.** The Datadog app host is **not** `app.${DD_SITE}` for every site. It is
`https://app.datadoghq.com` (US1), `https://app.datadoghq.eu` (EU1), `https://app.ddog-gov.com` (Gov),
and for every other site it is `https://${DD_SITE}` itself - `https://us3.datadoghq.com`,
`https://us5.datadoghq.com`, `https://ap1.datadoghq.com`, `https://ap2.datadoghq.com`,
`https://uk1.datadoghq.com`. Resolve it once and substitute it wherever `<APP_BASE>` appears below.
Full list: https://docs.datadoghq.com/getting_started/site/

**Remember the resolved `DD_SITE`.** Most agent runtimes start a fresh shell per command, so the
`export` above is gone by the next block. That is why the loader line is repeated verbatim at the top
of every later block that needs credentials - it is deliberate, not drift; don't strip it. `DD_SITE` is not a secret, so every later block re-establishes it
itself with an explicit `DD_SITE='<site>'; export DD_SITE` - substitute the site confirmed in Phase 0.
It is a plain assignment rather than `: "${DD_SITE:=...}"` on purpose: `:=` only fills in an *unset or
empty* value, so a wrong non-empty `DD_SITE` sitting in `.env` would survive it and every call would go to
the wrong region. The keys are guarded with `:?` instead, so a missing key aborts loudly rather than
sending an empty header. **Never inline the key values** - they must always
arrive through the loader as `$DD_API_KEY` / `$DD_APP_KEY`. Two consequences follow, and both are
deliberate:

- **Datadog calls pass headers on stdin**, as `printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- ...`.
  `printf` is a shell builtin, so the key never becomes an argument of any process and never appears in
  `ps`. Writing `-H "DD-API-KEY: $DD_API_KEY"` instead would put it in curl's argv. (`-H @-` needs
  curl 7.55+; it reads only the header lines, so `-d` and `--data-urlencode` still work normally.)
- **Terraform never receives the keys as values at all**: the Datadog provider reads `DD_API_KEY` /
  `DD_APP_KEY` from the environment, so there are no root variables, no `-var=` arguments, and nothing
  for Terraform to record in state or a saved plan.

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of
the process list.

**Tools.** `terraform` is required; the `aws` CLI is optional (it lists buckets, their Regions,
policies and encryption, and merges a KMS key policy) and so is `jq` (it is used by the S3 metric check and
PATCH in Phase 1, which say how to do them by hand without it, and by the completion callback in Phase 4):

```bash
command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "aws: not installed"
command -v jq >/dev/null 2>&1 && echo "jq: available" || echo "jq: not installed"
```

**The snippets here are POSIX shell.** Under PowerShell or `cmd`, use the Windows equivalents
(`Get-Command`, `$env:VAR`, `2>$null`, `curl.exe`) - same calls, same order.

## Phase 1: Select AWS Account

> **Required first step - always run this, even if the user says AWS is "already connected."** The
> account-config `id` and the IAM `role_name` returned here are needed for the resource-collection and
> metric PATCHes below and for the Terraform `datadog_aws_integration_role_name` input, and there is no
> other way to obtain them. Never PATCH an account or generate Terraform against a guessed or
> placeholder account ID (e.g. `account-config-id` or the 12-digit `aws_account_id`) - fetch the real
> values from this call first.

Before setting up Storage Management, fetch the list of AWS accounts already integrated with Datadog:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -s -X GET "https://api.${DD_SITE}/api/v2/integration/aws/accounts"
```

The response contains an array of accounts. Each account includes:
- `id` - the account config ID, needed for the resource-collection PATCH path below (distinct from `aws_account_id`)
- `aws_account_id` - the 12-digit AWS account number
- `aws_partition` - `aws`, `aws-us-gov`, or `aws-cn` (see "Accounts Outside the Commercial AWS Partition")
- `auth_config.role_name` - the IAM role name used by Datadog (passed to the module as `datadog_aws_integration_role_name`).
  The field sits **directly** on `auth_config` in the API response; there is no intermediate
  `aws_auth_config_role` object (that nesting exists only in the Terraform `datadog_integration_aws_account`
  resource).
- `resources_config.extended_collection` - whether resource collection is enabled
- `metrics_config.enabled` and `metrics_config.namespace_filters` - whether CloudWatch metric collection is
  enabled, and which namespaces it includes or excludes

### Accounts Authenticated With Access Keys

`auth_config` is one of two shapes. Role-based accounts have `role_name`; accounts integrated with an
access key pair have `access_key_id` and **no role at all**. If the selected account's `auth_config` has no
`role_name`, stop and tell the user - do not continue into Phase 2. The module attaches Datadog's read policy
to the integration role, and there is no role here:

> Account `<AWS_ACCOUNT_ID>` is integrated using an access key pair rather than an IAM role, so there's no
> role for me to grant read access to the inventory reports. Storage Management needs a role-based
> integration. You can migrate this account to an IAM role in the AWS integration settings (or run
> `dd-aws-integration`), then come back and I'll pick up from here.

### Accounts Outside the Commercial AWS Partition

The module and the policies in `references/terraform.md` hard-code `arn:aws:` ARNs, so they cannot target
buckets in AWS GovCloud (`aws-us-gov`) or China (`aws-cn`). Check the selected account's `aws_partition`
field in the response above, not the Datadog site. If it is anything other than `aws`, stop before any
PATCH or Terraform and tell the user that this setup supports only the commercial AWS partition. There is
no supported workaround: do not suggest forking or editing the module, rewriting ARNs, or building the same
setup by hand with the AWS CLI or API.

Never pass an empty, null, or guessed `datadog_aws_integration_role_name`; the value has to come from the API
response.

### If accounts are found

Present the list to the user and ask which AWS account they'd like to set up Storage Management for.
Display each account's ID and any tags to help them identify the right one. **Always ask which account -
never auto-pick, even if exactly one account is returned or one already has resource or metric collection
enabled, and never assume the account the local `aws` CLI or current shell is authenticated to is the one
the user means.** Confirm the selected account before making any change (the resource-collection and metric
PATCHes below, and the Terraform apply). Record the selected account's `id`, `aws_account_id`, and
`role_name` - you'll need them for the PATCH below and the Terraform configuration.

If the user wants to add a **new** AWS account instead, load the `dd-aws-integration` skill now and follow it through its own apply and verification, then return to the integration lookup at the start of Phase 1: select and confirm the newly added target, record its identifiers, and repeat every check after that lookup. Do not fold this setup into the integration's apply; start it only once the integration exists and the user has confirmed it.

### If no accounts are found

Tell the user they need to set up the base AWS integration first. Ask if they'd like to do that now -
if yes, load the `dd-aws-integration` skill now and follow it in full through its own apply and
verification (do not summarize it), then return to the integration lookup at the start of Phase 1: select
and confirm the newly added account, record its identifiers, and repeat every check after that lookup. Do
not fold this setup into the integration's apply; start it only once the integration exists and the user
has confirmed it.

### Verify Resource Collection

Storage Management requires **resource collection** (`extended_collection`) to be enabled on the AWS
integration. Check the selected account's `resources_config.extended_collection` field from the
response above.

**Check who owns these settings first.** This PATCH and the metric PATCH below change attributes that
Terraform manages on `datadog_integration_aws_account`. If Terraform owns the integration, its next apply
reverts a PATCH and switches the setting off again. Look for the resource in the local project:

```bash
grep -rnE --include='*.tf' --include='*.tf.json' 'datadog_integration_aws_account' .
```

- If a `datadog_integration_aws_account` with `aws_account_id = "<AWS_ACCOUNT_ID>"` is found, do **not**
  PATCH. Make the same changes in that resource in Phase 3, so they are part of the plan the user approves:
  `extended_collection = true` in its `resources_config` block, and for metrics, `enabled = true` in
  `metrics_config` plus `AWS/S3` added to `namespace_filters.include_only` or removed from
  `namespace_filters.exclude_only` (keep every other namespace as it is). Still run the read-only S3 metric
  check below to learn what has to change. This includes an integration `dd-aws-integration` created
  earlier in this session: its Terraform owns these settings, so never PATCH that account.
- If nothing is found, ask the user, as part of the confirmation below, whether Terraform in another project
  or workspace manages this integration. If it does, do not PATCH here: tell the user to make the changes
  above in that project and apply them, then re-run the checks in this phase before continuing.

Otherwise, enable it by patching the account. Include this PATCH in the install plan unless the account you fetched in Phase 1 already shows `extended_collection: true` - the call is idempotent, so when in doubt, include it.
This modifies the user's AWS integration, so confirm explicitly and wait for a yes before running it:

> To set up Storage Management I need to turn on resource collection for account `<AWS_ACCOUNT_ID>` on
> its Datadog AWS integration (`extended_collection: true`). OK to proceed?

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -sS -w '\nHTTP %{http_code}\n' -X PATCH "https://api.${DD_SITE}/api/v2/integration/aws/accounts/<ACCOUNT_CONFIG_ID>" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "type": "account",
      "attributes": {
        "resources_config": {
          "extended_collection": true
        }
      }
    }
  }'
```

Use the account's `id` field (not `aws_account_id`) as the `<ACCOUNT_CONFIG_ID>` path parameter. Anything
other than `HTTP 200`, or a response whose `resources_config.extended_collection` is not `true`, means the
change did not apply: show the user the error and stop. Only then let
the user know resource collection was enabled - it is required for Storage Management to attribute
storage to specific buckets and prefixes.

### Verify S3 Metric Collection

Storage Management also surfaces AWS CloudWatch S3 metrics (`aws.s3.*`) alongside the inventory data, which
requires metric collection to be on **and** to include the `AWS/S3` namespace. `metrics_config.enabled` alone
is not enough: it is an account-wide switch, and `metrics_config.namespace_filters` can still leave S3 out. S3 is
collected when `enabled` is `true` and either there are no namespace filters, `include_only` lists `AWS/S3`, or
`exclude_only` does not list it.

This read-only check prints the current settings and the minimal change needed (it needs `jq`; without it, read
the same two fields from the Phase 1 response and build the body by hand):

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -fsS "https://api.${DD_SITE}/api/v2/integration/aws/accounts/<ACCOUNT_CONFIG_ID>" \
  | jq '.data.attributes.metrics_config as $m
      | { current: { enabled: $m.enabled, namespace_filters: $m.namespace_filters },
          s3_collected: (($m.enabled == true) and
            (if $m.namespace_filters.include_only then ($m.namespace_filters.include_only | index("AWS/S3")) != null
             elif $m.namespace_filters.exclude_only then ($m.namespace_filters.exclude_only | index("AWS/S3")) == null
             else true end)) }'
```

If `s3_collected` is `true`, no change is needed. If Terraform owns the integration (see "Verify Resource
Collection"), make the change there instead of running the PATCH. Otherwise the PATCH below sets `enabled: true` and, only if
a filter excludes S3, adds `AWS/S3` to `include_only` or removes it from `exclude_only`, keeping every other
namespace in the list as it is. Fields it does not send, including the other filters, are left unchanged. This
also modifies the user's AWS integration, so confirm explicitly and wait for a yes before running it (you can
ask once for both PATCHes if both are needed), naming exactly what changes:

> I also need AWS CloudWatch metric collection to include S3 for account `<AWS_ACCOUNT_ID>`. I'll
> <turn metric collection on / add `AWS/S3` to your included namespaces / remove `AWS/S3` from your excluded
> namespaces>; nothing else in your metric settings changes. OK to proceed?

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
command -v jq >/dev/null 2>&1 || { echo "jq not installed - build the PATCH body by hand from the check above"; exit 1; }
dd_headers() { printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY"; }
body=$(dd_headers | curl -H @- -fsS "https://api.${DD_SITE}/api/v2/integration/aws/accounts/<ACCOUNT_CONFIG_ID>" \
  | jq -c '.data.attributes.metrics_config.namespace_filters as $f
      | { data: { type: "account", attributes: { metrics_config: ({ enabled: true }
          + (if $f.include_only then { namespace_filters: { include_only: (($f.include_only + ["AWS/S3"]) | unique) } }
             elif $f.exclude_only then { namespace_filters: { exclude_only: ($f.exclude_only - ["AWS/S3"]) } }
             else {} end)) } } }')
[ -n "$body" ] || { echo "account lookup FAILED - stop"; exit 1; }
resp=$(dd_headers | curl -H @- -sS -w '\n%{http_code}' -X PATCH \
  "https://api.${DD_SITE}/api/v2/integration/aws/accounts/<ACCOUNT_CONFIG_ID>" \
  -H "Content-Type: application/json" -d "$body")
code=$(printf '%s\n' "$resp" | tail -n 1)
resp_body=$(printf '%s\n' "$resp" | sed '$d')
[ "$code" = 200 ] || { echo "metrics PATCH FAILED (HTTP $code): $resp_body"; exit 1; }
printf '%s' "$resp_body" | jq '.data.attributes.metrics_config | {enabled, namespace_filters}'
```

Check that the printed config has `enabled: true` and includes `AWS/S3` before telling the user S3 metric
collection is on. Include this PATCH in the install plan unless the check above already shows
`s3_collected: true`.

### Fetch your Datadog org id

You also need your Datadog **org id** - it scopes the module `name` (Phase 3), which feeds the S3 Inventory
configuration id, so that **one AWS account connected to more than one Datadog org** doesn't collide on that id.
(The destination bucket name uses the AWS **account** id, not the org id - see Phase 2 §3.) Fetch it once:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -s -X GET "https://api.${DD_SITE}/api/v1/org"
```

The response is an `OrganizationListResponse`; read your top-level org's `public_id` from `orgs[0].public_id`.
Lowercase it and strip any character that is not `[a-z0-9-]` (S3 bucket names are lowercase-only and the
`public_id` is not guaranteed lowercase); call the result `<DD_ORG_ID>`. Fetch it on **every** install (new
**or** reused destination bucket) - the module `name` always needs it.

## Phase 2: Choose Buckets and Plan the Setup

Storage Management is set up **per Region** - S3 Inventory requires each report's destination bucket to be in
the **same Region** as the source bucket it serves. So the plan is driven by *which buckets* the user wants and
*what Regions they are in* (which you work out yourself, not the user). Keep it conversational.

### 1. Source buckets to monitor

Ask which S3 bucket(s) they want to monitor - they can pick more than one.

If the `aws` CLI is available (`command -v aws >/dev/null 2>&1` succeeds), first confirm it is authenticated to
the account selected in Phase 1. Every lookup below, the KMS key-policy merge, and the Terraform apply act on
whatever account these credentials belong to:

```bash
want='<AWS_ACCOUNT_ID>'   # from Phase 1
have=$(aws sts get-caller-identity --query Account --output text) || { echo "no usable AWS credentials"; exit 1; }
[ "$have" = "$want" ] || { echo "ambient AWS credentials are for account $have, but you are setting up $want - stop and fix this"; exit 1; }
echo "AWS credentials match account $want"
```

If it fails, stop and have the user switch to credentials for the selected account. Then fetch the bucket names:

```bash
aws s3api list-buckets --query "Buckets[].Name" --output text
```

Then **use the `AskUserQuestion` tool with `multiSelect: true`** to present the buckets as a checkbox list, one
option per bucket name, rather than printing the raw list and asking them to retype names. If the interface does
not support `AskUserQuestion`, print a numbered list and ask the user to reply with the numbers they want (e.g.
`1, 3, 4`). This makes monitoring several buckets at once easy and avoids typos.

If the `aws` CLI is unavailable, point them at the S3 console (https://console.aws.amazon.com/s3/) to find their
bucket names and ask them to paste the ones they want.

Record the chosen bucket names - the module configures one S3 Inventory per source bucket.

### 2. Determine each bucket's Region and group by Region

A destination bucket can only collect inventory for source buckets in its **own** Region, so the number of
destination buckets you need equals the number of distinct Regions the source buckets span. Work out each
bucket's Region yourself rather than asking. If the `aws` CLI is available (`command -v aws >/dev/null 2>&1` succeeds):

```bash
aws s3api get-bucket-location --bucket <SOURCE_BUCKET_NAME> --query LocationConstraint --output text
```

`get-bucket-location` returns two legacy values instead of the plain Region code: a `None`/empty result means
`us-east-1`, and an **`EU`** result means `eu-west-1` (the legacy location constraint AWS still returns for some
Ireland buckets). Normalize both to the real Region id before grouping or generating Terraform - every other Region
returns its own code, but a raw `EU` would become an invalid `provider` Region and break the apply. If the CLI is
unavailable, ask the user which Region each bucket is in. **Group the source buckets by Region** - each group
becomes one destination bucket and one module instance in Phase 3.

### 3. Per Region: pick the destination bucket

For each Region the source buckets span, decide the **destination bucket** that receives its inventory reports.

> **STOP - required user decision.** Use the `AskUserQuestion` tool to ask the user whether to create a new
> destination bucket or use an existing one. Options: **"Create a new bucket (recommended)"** and **"Use an
> existing bucket"**. Do not proceed until the user answers.

- **Create a new bucket (recommended).** The skill creates it in Phase 3, in that Region, encrypted with SSE-S3.
  S3 bucket names are globally unique across **all** AWS accounts, and the destination bucket is **per AWS account**
  (one Datadog org commonly connects many AWS accounts, and each gets its own bucket - the module can't share one
  across accounts). Name it from the selected account's `<AWS_ACCOUNT_ID>` (Phase 1) and the Region:
  **`dd-inventory-<AWS_ACCOUNT_ID>-<REGION>`** (e.g. `dd-inventory-123456789012-us-east-1`). The AWS account id is
  globally unique, so two accounts can never generate the same name (`BucketAlreadyExists`); the Region keeps a
  single account's multi-Region installs distinct. Always include both and keep the full name within 63 characters.
  Do **not** fall back to a fixed name - the account id + Region is what prevents the global collision.
- **Use an existing bucket.** It must be in the **same Region** as the source buckets it serves. When reusing one,
  gather two facts (if the `aws` CLI is available) so Phase 3 configures the module correctly:
  - **Existing bucket policy?** `aws s3api get-bucket-policy --region <DESTINATION_BUCKET_REGION> --bucket <DESTINATION_BUCKET_NAME> --query Policy --output text` -
    if it returns a policy, keep the JSON exactly as returned: Phase 3 sets the module's
    `destination_bucket_policy_management = "none"` and adds the inventory-write statement to that policy itself
    (see "Existing destination bucket policy" in `references/terraform.md`). Use `"create"` **only** when the
    command fails with `NoSuchBucketPolicy`. Any other
    failure (`AccessDenied`, expired credentials, a network error) says nothing about the policy, and `"create"`
    could then replace a policy you couldn't read: show the user the error and stop until it is resolved. Without
    the CLI, ask the user whether the bucket has a policy and, if so, to copy it (S3 console, bucket
    **Permissions** tab).
  - **SSE-KMS encrypted?** `aws s3api get-bucket-encryption --region <DESTINATION_BUCKET_REGION> --bucket <DESTINATION_BUCKET_NAME> --query 'ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault' --output json` -
    if `SSEAlgorithm` is `aws:kms`, record `KMSMasterKeyID` (the CMK ARN); Phase 3 adds the KMS grants (see
    "SSE-KMS destination buckets"). If `AES256`, no KMS work is needed. If the CLI is unavailable, ask the user
    whether the bucket uses a KMS key and, if so, for its ARN.
  - **Bucket with a policy (SSE-S3 or SSE-KMS): does it require encryption headers?** Hardened buckets often
    carry a policy that denies `s3:PutObject` unless the request sends `x-amz-server-side-encryption` (with
    `AES256` or `aws:kms`), or a specific `x-amz-server-side-encryption-aws-kms-key-id`. Run this check for
    every existing destination bucket that has a policy, whatever its default encryption. The module's
    inventory configuration sets no report
    encryption, so S3 sends no such header, and that statement blocks every inventory delivery even though
    the apply succeeds. List the `Deny` statements conditioned on those keys (needs `jq`; without it, read
    the policy recorded above):

    ```bash
    aws s3api get-bucket-policy --region <DESTINATION_BUCKET_REGION> --bucket <DESTINATION_BUCKET_NAME> --query Policy --output text \
      | jq '[(.Statement | if type == "array" then . else [.] end)[]
          | select(.Effect == "Deny" and ((.Condition // {}) | tostring | test("s3:x-amz-server-side-encryption")))
          | {Sid, Action, NotAction, Resource, Condition}]'
    ```

    If any listed statement can apply to `s3:PutObject` under the destination prefix, the module cannot
    satisfy it. Stop and tell the user: they can choose another destination bucket (a new one is the
    simplest), or change that policy themselves and come back. Never remove, narrow, or add exceptions to
    such a statement on the user's behalf.

### 4. Destination prefix

Inventory reports land under a key **prefix** in the destination bucket. The module's default is
`datadog-inventories/` (the trailing slash matters - the module uses it for both the inventory destination and the
sync registration). Keep the default unless the user has a specific reason to change it, and pass whatever they
choose to the module as `destination_prefix`.

### 5. S3 Access Logs (Optional)

S3 access logs surface **cold-data patterns, unusual access, and tiering / right-sizing recommendations** on top
of the core inventory data. Deciding now means everything is included in a single `terraform apply`.

> **STOP - required user decision.** Use the `AskUserQuestion` tool to ask the user whether they want to enable
> S3 access logs. Options: **"Yes, enable access logs"** and **"No, skip for now"**. Do not
> proceed until the user answers.

- **If yes:** access logs need a **Forwarder Lambda ARN** and an **access-log bucket**, both with per-Region
  constraints - the module wires an S3 notification from the access-log bucket to the Forwarder, and S3 invokes
  only a **same-Region** Lambda.

  **Forwarder ARN - one per Region with access logs.** Ask for the `datadog_forwarder_arn`. If the source
  buckets span multiple Regions, you need a Forwarder **in each** of those Regions: the Lambda must be in the same
  Region as the access-log bucket it's triggered from, and `terraform apply` errors on a cross-Region Forwarder.
  Enable access logs only on the Regions that have a Forwarder and **defer** the rest. If the user has none, tell
  them the `dd-aws-log-forwarding` skill deploys one and can be run as a follow-on; skip access logs for now. Do not
  deploy the Forwarder inline.

  **Access-log bucket - a dedicated, same-Region, SSE-S3 bucket. By default the skill creates a fresh one** (Phase 3
  emits `aws_s3_bucket` + SSE-S3 + the delivery grant - the same shape `dd-aws-ccm` uses for the CUR bucket), so
  the user doesn't have to provision anything. Three constraints make a dedicated bucket the right shape:
  - The module manages the access-log bucket's S3 **notification configuration authoritatively** - applying it
    **overwrites every existing notification** on that bucket (leaving only the Forwarder trigger; any pre-existing
    Lambda / SNS / SQS / EventBridge triggers are silently destroyed). So the bucket must be **dedicated** to access logs.
  - S3 delivers server access logs **only** to an SSE-S3 (AES256) bucket - never SSE-KMS (the logs are written
    under a key the log-delivery service can't use, so delivery **silently fails**).
  - S3's log-delivery service must be **granted write access** to the access-log bucket, or it **silently** delivers
    nothing (the apply succeeds; no log object ever lands). Phase 3 attaches a bucket policy granting
    `logging.s3.amazonaws.com` `s3:PutObject` - neither the module nor `terraform apply` adds it (only the S3
    *console* does, and a fresh bucket has ACLs disabled so the bucket policy is the only mechanism).

  A fresh bucket the skill creates satisfies all three automatically (no prior notifications or policy; SSE-S3; grant
  attached). **Bring-your-own is fine too** - point at an existing **dedicated** log bucket if it's same-Region,
  SSE-S3, and has **no existing notifications**. **Never** route access logs into the **inventory destination bucket**
  (or any other active/functional bucket): mixing the audit log with the data Datadog reads is the discouraged
  pattern, and it would force the module out of managing the destination policy. If the only bucket available has
  notifications or is SSE-KMS, **defer** access logs rather than overwrite/misconfigure.

  **Existing source-bucket logging.** A bucket has only one access-log destination, and the module points every
  source bucket in the Region at the new access-log bucket, replacing any logging the bucket already has (for
  example an audit-log bucket). Check each source bucket first:

  ```bash
  aws s3api get-bucket-logging --region <SOURCE_BUCKET_REGION> --bucket <SOURCE_BUCKET_NAME> --query LoggingEnabled --output json
  ```

  `null` means logging is off, and the module can turn it on safely. If it returns a `TargetBucket`, tell the
  user which bucket currently receives that source bucket's logs and that enabling access logs would redirect
  them to the new access-log bucket. Only enable access logs for that Region if they explicitly approve;
  otherwise defer access logs for the Region (the module applies logging to all of the Region's source buckets).
  If the command fails, show the error and treat the bucket's logging as unknown, which means defer unless the
  user confirms. Without the CLI, ask the user whether any of the source buckets already have server access
  logging on (S3 console, bucket **Properties** tab). Approval to redirect the logs does not settle who
  manages them: if Terraform already manages a source bucket's logging, the module would become a second
  owner (see the access-log bullet in "Check for Existing Terraform"), so ask the user, and defer access
  logs for the Region if another project or workspace manages it.

  Record the access-log bucket name and the per-Region Forwarder ARNs separately from the inventory destination.
- **If no:** Skip - they can enable it later.

Record the decision and the Forwarder ARN (if provided) - Phase 3 uses them to generate a complete
single-apply configuration.

### Confirm the plan

Before generating Terraform, summarize what you found and confirm, e.g. *"Your buckets are in **us-east-1**
(`app-logs`, `app-data`) and **eu-west-1** (`analytics-archive`), so I'll set up **2 destination buckets**, one
per Region. Sound good?"* Always include the access log outcome in the summary:
- Opted in with Forwarder ARN → "S3 access logs will be enabled in this apply." Name any source bucket whose
  existing logging destination will be replaced.
- Opted in but no Forwarder → "S3 access logs will be added after `dd-aws-log-forwarding` is complete."
- Declined → "S3 access logs skipped."

## Phase 3: Generate and Apply Terraform

### Check for Existing Terraform

Before generating new configuration, check if the user already has a Terraform project:

```bash
find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' -o -name 'terraform.tfstate' \) -print
```

**If existing `.tf` files are found:**
- Read them to understand what providers and resources are already configured.
- If `aws` (and `datadog`) providers already exist, do not create duplicates. Make sure every `aws` provider
  these resources use (including aliases) has `allowed_account_ids = ["<AWS_ACCOUNT_ID>"]`, adding it if
  missing, and that the `datadog` provider's `api_url` matches `https://api.<DD_SITE>` (or `DD_HOST` is set to it).
- Add the **new resources** needed for Storage Management, and change existing resources only as the
  bullets below require.
- If Phase 1 found a `datadog_integration_aws_account` for `<AWS_ACCOUNT_ID>` in this project, make the
  approved resource-collection and S3 metric changes in that resource now (see "Verify Resource Collection"
  in Phase 1), keeping its other settings.
- For each **existing** destination bucket, check whether the project already manages its policy before
  emitting the "Existing destination bucket policy" resource:

  ```bash
  grep -rnE --include='*.tf' --include='*.tf.json' 'resource "aws_s3_bucket_policy"|resource "aws_s3_bucket"|<DESTINATION_BUCKET_NAME>' .
  ```

  Read each `aws_s3_bucket_policy` the search finds, and any `aws_s3_bucket` for `<DESTINATION_BUCKET_NAME>`
  that sets an inline `policy`. If one of them targets that bucket, add the inventory-write statement to it
  as "Existing destination bucket policy" in `references/terraform.md` describes, set the module's
  `destination_bucket_policy_management = "none"` (even if the live bucket has no policy yet), and add that
  resource to the module's `depends_on`. Use the snapshot-based resource only when nothing in the project owns
  the policy. If the user says another project or workspace manages the policy, stop and have them add the
  statement there instead.
- For each **SSE-KMS** destination bucket (Phase 2 §3), check whether the project already manages the CMK's key
  policy before using the CLI merge in "Merge the KMS key policy". `<KMS_KEY_ID>` is the key ID (the part of
  `<KMS_KEY_ARN>` after `key/`):

  ```bash
  grep -rnE --include='*.tf' --include='*.tf.json' 'resource "aws_kms_key"|resource "aws_kms_key_policy"|resource "aws_kms_alias"|<KMS_KEY_ID>' .
  ```

  Read each `aws_kms_key_policy` the search finds, and any `aws_kms_key` that sets the `policy` argument. If one
  of them governs that key (follow `key_id` references and aliases to the key), it owns the whole key policy and
  its next apply would remove grants added through the CLI. Add the two statements from "Merge the KMS key
  policy" to that configuration instead (as `statement` blocks if it is built from an `aws_iam_policy_document`),
  keeping its existing statements, add that resource to the module's `depends_on`, and skip the CLI merge. An
  `aws_kms_key` without a `policy` argument does not own the policy. If the user says another project or
  workspace manages the key, stop and have them add the statements there instead. Use the CLI merge only when
  no infrastructure-as-code owns the key policy.
- If access logs are enabled for a Region, check whether the project already manages logging for any of that
  Region's source buckets. The module creates an `aws_s3_bucket_logging` for every source bucket, and two owners
  overwrite each other: a later apply restores the old destination, or destroying the old resource turns logging off.

  ```bash
  grep -rnE --include='*.tf' --include='*.tf.json' 'resource "aws_s3_bucket_logging"|logging *\{|<SOURCE_BUCKET_NAME>' .
  ```

  If an `aws_s3_bucket_logging`, or an `aws_s3_bucket` with an inline `logging` block, targets one of those
  source buckets, transfer ownership only with the user's explicit approval: remove that resource or block from
  the configuration and from state (a `removed` block with `lifecycle { destroy = false }`, or `terraform state rm`)
  in the same change, so the plan shows no destroy of the old logging. Otherwise defer access logs for that Region.
- If the user has a modular layout, create a new file like `datadog-storage-management.tf`.

**If `dd-aws-integration` was just completed in this session**, its configuration is the existing
Terraform: add the Storage Management resources to the same `main.tf` and reuse its `terraform` /
`required_providers` block, `datadog` provider, and `local.datadog_site` instead of declaring them again.
Reuse its `aws` provider only if its `region` matches a destination bucket's Region; otherwise add a
Region-aliased provider (see "Multiple Regions").

**If no existing Terraform is found**, generate a standalone configuration. The `datadog` provider is new
and is always required.

### Generate Terraform HCL

Storage Management is provisioned with the official **`DataDog/storage-management-datadog/aws`** module: it
configures S3 Inventory on each source bucket, the destination bucket policy (when the bucket has none yet),
the Datadog read IAM policy, and enables Storage Management - all in one `terraform apply`. The skill only
creates the destination bucket (new-bucket case), manages the merged policy of an existing destination bucket
that already has one, and adds the SSE-KMS grants the module doesn't handle.

Build the config from the pieces below, substituting the values from Phase 2. **One module instance per
Region** (Phase 2 §2): if the source buckets span N Regions, emit N destination buckets + N module instances,
each pinned to its Region with a provider alias (see "Multiple Regions").

The HCL for each piece, with the notes that go with it, is in **`references/terraform.md`**. Read it now,
including the comments inside the HCL - they say which lines to include for access logs and why the
`datadog` provider's `api_url` must be derived from `DD_SITE`. The pieces are:

| Piece | Included when |
|---|---|
| Providers (`aws`, `datadog`, `local.datadog_site`) | Always (reuse existing providers per "Check for Existing Terraform") |
| Destination bucket (`aws_s3_bucket` + SSE-S3) | Creating a new destination bucket |
| The module (`DataDog/storage-management-datadog/aws`) | Always, one instance per Region |
| Existing destination bucket policy (merged `aws_s3_bucket_policy`) | Reusing a destination bucket that already has a policy (Phase 2 §3) |
| Access-log bucket + delivery grant | The user opted in to access logs and has a same-Region Forwarder ARN (Phase 2 §5) |
| SSE-KMS read-side inline policy | The destination bucket is SSE-KMS-encrypted (Phase 2 §3); pair it with "Merge the KMS key policy" below |
| Region-aliased providers and per-Region module | The source buckets span more than one Region (see "Multiple Regions") |

### Assemble per scenario

| Destination bucket | What the skill emits |
|---|---|
| New | `aws_s3_bucket` + SSE-S3 + module with `destination_bucket_policy_management = "create"` |
| Existing, policy owned by a resource in this project | inventory-write statement added to that resource + module with `destination_bucket_policy_management = "none"` and `depends_on` that resource |
| Existing, has a policy with no owner in this project | merged `aws_s3_bucket_policy` + module with `destination_bucket_policy_management = "none"` (see "Existing destination bucket policy") |
| Existing, no policy | module only, `destination_bucket_policy_management = "create"` |

Add the SSE-KMS grants (read-side inline policy + the key-policy grants) for any destination bucket Phase 2 §3
found to be KMS-encrypted. The key-policy grants go into the resource that owns the key policy when the project
has one (see "Check for Existing Terraform"); otherwise use the CLI merge below.

### Multiple Regions

If the source buckets span more than one Region, S3 Inventory requires a same-Region destination for each, so emit
**one module instance per Region**, each with its own destination bucket and a Region-pinned provider alias (the
HCL is under "Multiple Regions" in `references/terraform.md`).

Give the per-Region resources Region-unique names and pass the matching provider - the `aws_s3_bucket` gets
`provider = aws.<alias>`, the module gets `providers = { aws = aws.<alias> }`, and (new-bucket case) the module
`depends_on` its per-Region destination bucket and its encryption configuration exactly like the single-Region block - `destination_bucket_name` is a
literal string, so without that edge Terraform may run the module before the bucket exists and the apply fails
intermittently. For an **existing** per-Region destination bucket, drop that bucket's `aws_s3_bucket` / SSE /
`depends_on` and set `destination_bucket_name` to the literal name.

The `datadog` provider is configured once (no alias) and shared across all instances. `aws_s3_bucket` has no
Region argument - the provider's Region places the bucket - so always pass the alias whose `region` matches that
Region, or the destination is created cross-Region and delivery silently fails.

### Merge the KMS key policy (SSE-KMS destination bucket only)

When a destination bucket is SSE-KMS-encrypted, two principals need access to the CMK: the **S3 service** needs
`kms:GenerateDataKey` to write the encrypted inventory report, and the **Datadog integration role** needs
`kms:Decrypt` to read it. (The inline IAM grant above covers the read only when the key policy delegates to IAM;
naming the role directly in the key policy here also covers locked-down keys that removed that delegation.) The key
policy is a whole-policy resource the module can't manage. If Terraform in this project owns it ("Check for
Existing Terraform"), add the two statements below to that resource and skip the CLI steps; if another workspace
owns it, have the user add them there. Only when no infrastructure-as-code owns the key policy, merge these grants
out-of-band, **before `terraform apply`**. If the `aws` CLI is available (`command -v aws >/dev/null 2>&1` succeeds):

Do this once per SSE-KMS destination bucket, in that bucket's Region: the CLI sends KMS calls to the
`--region` endpoint regardless of the Region in the key ARN, so always pass the key's Region.

1. Fetch the current key policy:

   ```bash
   aws kms get-key-policy --region <DESTINATION_BUCKET_REGION> --key-id <KMS_KEY_ARN> --policy-name default --query Policy --output text
   ```

2. Add these statements - keep all existing statements; if a matching statement already exists, union the source
   ARNs into it rather than replacing it:

   ```json
   {
     "Sid": "AllowS3InventoryKMSEncrypt",
     "Effect": "Allow",
     "Principal": { "Service": "s3.amazonaws.com" },
     "Action": "kms:GenerateDataKey",
     "Resource": "*",
     "Condition": {
       "StringEquals": { "aws:SourceAccount": "<AWS_ACCOUNT_ID>" },
       "ArnLike": { "aws:SourceArn": ["arn:aws:s3:::<SOURCE_BUCKET_NAME>"] }
     }
   },
   {
     "Sid": "AllowDatadogKMSDecrypt",
     "Effect": "Allow",
     "Principal": { "AWS": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<DATADOG_INTEGRATION_ROLE_NAME>" },
     "Action": "kms:Decrypt",
     "Resource": "*"
   }
   ```

3. **Show the merged key policy to the user and wait for confirmation**, then apply it:

   ```bash
   aws kms put-key-policy --region <DESTINATION_BUCKET_REGION> --key-id <KMS_KEY_ARN> --policy-name default --policy file://merged-key-policy.json
   ```

If the `aws` CLI is unavailable, have the user make the same edit in the KMS console (key → **Key policy** →
**Edit**) and confirm before continuing. `Resource: "*"` here means "this key" - a key policy only governs its own key.

### Apply Terraform

1. Substitute all `<PLACEHOLDER>` values with the values collected in Phase 2.
2. Ensure AWS credentials are configured **for the account selected in Phase 1** (rerun the account check from
   Phase 2 §1 if the `aws` CLI is available; `allowed_account_ids` on every `aws` provider blocks the plan
   otherwise), and `DD_API_KEY` / `DD_APP_KEY` / `DD_SITE` are set for the `datadog` provider.
3. Run `terraform init` (installs the `aws` and `datadog` providers and downloads the module).
4. Save the plan to a file and show it to the user for review:

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   terraform plan -out=tfplan \
     || { echo "terraform plan FAILED - fix the error and re-plan; do NOT apply"; exit 1; }
   ```

   Before showing the plan, check every change to the `datadog_integration_aws_account` whose `aws_account_id`
   is the account selected in Phase 1 (changes to other accounts are outside this setup: list them for the user,
   but do not block on or edit them for these settings). If it would set
   `extended_collection` to `false`, set `metrics_config.enabled` to `false`, or leave `AWS/S3` out of the
   collected namespaces (missing from a set `include_only`, or listed in `exclude_only`), the project is undoing the
   Phase 1 prerequisites. Do not apply: fix that resource as in "Verify Resource Collection" and re-plan.

   For an SSE-KMS destination, also check every change to an `aws_kms_key` `policy` or an `aws_kms_key_policy`
   for that key. If the planned policy lacks the `AllowS3InventoryKMSEncrypt` or `AllowDatadogKMSDecrypt` grant
   (or an existing statement already granting the same access), the apply would break inventory delivery or
   Datadog's read. Do not apply: add the grants to that resource as in "Merge the KMS key policy" and re-plan.

   Stop here and wait for the user to approve the plan before applying it.

5. Apply **that saved plan** once the user confirms. A bare `terraform apply` re-plans, so the user could
   approve one plan and get another:

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   trap 'rm -f tfplan' EXIT HUP INT TERM
   terraform apply tfplan \
     || { echo "terraform apply FAILED - do NOT verify or report success"; exit 1; }
   ```

   The apply provisions the AWS side **and** registers the Datadog sync - there is no separate API call.
   (For an SSE-KMS bucket whose key policy no infrastructure-as-code owns, do the CLI key-policy merge above first.)

## Phase 4: Finish

Storage Management data is **not** available immediately. Even with the sync registered, AWS generates the
first S3 Inventory report **24-48 hours** after the configuration is applied, and Datadog ingests it
shortly after. The sync (registered by `terraform apply`) is active right away, but storage metrics will not
appear until that first report lands.

Re-run the read-only S3 metric check from Phase 1 and the Phase 1 account lookup. If `extended_collection` is
no longer `true` or `s3_collected` is `false`, a Terraform-managed `datadog_integration_aws_account` switched it
back: fix that resource as in "Verify Resource Collection" and re-plan before reporting completion.

### Wrap Up

Once `terraform apply` has succeeded (which also registers the sync), make one best-effort completion
callback so the Datadog UI can finish this flow. Send it on every Storage Management install, fresh or
reuse-existing-bucket, and do not wait for inventory data to arrive: that takes 24-48 hours.

`DD_WORKFLOW_ID` is the UUID of the Datadog user the credentials belong to, which is how the UI
correlates this callback. Resolve it and post in **one** command: many agent runtimes start a fresh
shell per command, so a variable assigned in an earlier one would be empty here.

**Never print the `current_user` response**: it also carries the user's name, handle, and email. If no
JSON parser is available, skip this callback rather than installing one or blocking setup.

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
dd_headers() { printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY"; }
DD_WORKFLOW_ID="$(dd_headers | curl -H @- -fsS --connect-timeout 5 --max-time 20 \
  "https://api.${DD_SITE}/api/v2/current_user" | jq -er '.data.id')" \
  && dd_headers | curl -H @- -sS --connect-timeout 5 --max-time 20 -X POST \
  "https://api.${DD_SITE}/api/unstable/integration/aws/workflow/agentic_storage_management" \
  -H "Content-Type: application/vnd.api+json" \
  -d '{"data":{"type":"integration_setup_status","id":"'"${DD_WORKFLOW_ID}"'","attributes":{"status":"finished","step":"setup_complete"}}}'
```

Do not retry or block setup if this callback fails.

Summarize for the user what was set up: the AWS account, the source buckets grouped by Region, each Region's
destination bucket and prefix (new or existing), whether SSE-KMS grants were added, the access log outcome from
"Confirm the plan", and whether resource collection and S3 metric collection were enabled during this run.

Congratulate the user and let them know:
- The S3 Inventory is configured, the Datadog sync is registered, and AWS will deliver the first report within **24-48 hours**.
- Once data flows, storage metrics appear under `aws.s3.inventory.*` (e.g. `aws.s3.inventory.total_prefix_size`).
- Resource collection is enabled, so Datadog will attribute storage to buckets and prefixes automatically.

Provide these links (construct them as `<APP_BASE>/...`, using the App URL from Phase 0):
- **Storage Management**: `<APP_BASE>/storage-management` - view bucket and prefix-level storage.
- **Metrics Explorer**: `<APP_BASE>/metric/explorer?exp_metric=aws.s3.inventory.total_prefix_size` - confirm data once it lands.

**Suggested next steps:**
- Once data is flowing, set up **monitors** at `<APP_BASE>/monitors/create` to alert on
  storage growth or object-count anomalies.
- If S3 access logs were skipped or deferred in Phase 2 §5, let the user know they can be enabled later by
  completing `dd-aws-log-forwarding` (to get a Forwarder ARN) and then re-applying the Storage Management
  Terraform with the access log inputs uncommented.

## Important Notes

- The user must have IAM permissions to create S3 buckets, bucket policies, IAM policies, IAM role
  policy attachments, and S3 inventory configurations - and, for an SSE-KMS destination bucket, to update
  the CMK key policy.
- Storage Management is enabled by the module as part of `terraform apply` - there is no separate API call,
  and the module keeps the inventory and destination prefixes consistent.
- Storage Management requires resource collection (`extended_collection`) on the AWS integration; S3 metric
  collection (`metrics_config.enabled` with the `AWS/S3` namespace included) is also enabled (Phase 1) for the
  CloudWatch `aws.s3.*` metrics.
- The module sets `included_object_versions = "All"` on every inventory, so versioned buckets count their
  noncurrent versions and are never under-counted - no per-bucket scope decision is needed.
- S3 access logs are **optional** and only power tiering / right-sizing recommendations - they are not needed
  for core storage metrics. The access log question is asked in Phase 2 §5 (before Terraform is generated) so
  everything is included in a single `terraform apply`. Forwarding logs into Datadog is handled by the separate
  **`dd-aws-log-forwarding`** skill - do not deploy the Forwarder inline. The access-log bucket must be a fresh,
  dedicated, same-Region SSE-S3 (AES256) bucket with no existing notifications: S3 does not deliver server access
  logs to an SSE-KMS bucket, and the module manages the access-log bucket's notification config authoritatively
  (overwriting any existing notifications). So the skill provisions a **dedicated** access-log bucket by default
  (Phase 3) - never the inventory destination bucket - with a bucket policy granting `logging.s3.amazonaws.com`
  `s3:PutObject`; neither the module nor `terraform apply` adds that grant, and without it S3 silently delivers no
  logs. Each Region with access logs also needs its **own same-Region**
  Forwarder Lambda - S3 invokes only same-Region Lambdas (Phase 2 §5).
- The IAM policy grants read-only access - Datadog never writes to or modifies the buckets.
- Use one destination bucket **per source Region** - an S3 Inventory destination must be in the same
  Region as the source buckets it serves; one bucket can receive inventory for many source buckets in that
  Region, but sources in another Region need their own destination there.
- The first inventory report is delivered by AWS 24-48 hours after setup - do not expect data sooner.
- Never run `terraform apply` without showing the plan to the user first.