> **dd-aws-log-forwarding** — skill 41 of 69 in [datadog-labs/agent-skills](https://skillsdocs.com/datadog-labs/agent-skills).
>
> Book (all skills, one file): https://skillsdocs.com/datadog-labs/agent-skills.md
> Machine manifest: https://skillsdocs.com/datadog-labs/agent-skills/.well-known/agent-skills/index.json
> Install the book: `npx skills add datadog-labs/agent-skills`
> Upstream: https://github.com/datadog-labs/agent-skills/blob/main/dd-aws-log-forwarding/SKILL.md @ `main`
> Raw bytes, no header: https://raw.githubusercontent.com/datadog-labs/agent-skills/main/dd-aws-log-forwarding/SKILL.md
> Base for relative paths: https://raw.githubusercontent.com/datadog-labs/agent-skills/main/dd-aws-log-forwarding/
> Licence: MIT — https://spdx.org/licenses/MIT.html
>
> Content © its authors, served unmodified. Takedown: https://github.com/DreambaseAI/skillsdocs/issues/new?labels=takedown&title=Takedown+request

<!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. -->

---
name: dd-aws-log-forwarding
description: Set up AWS log forwarding to Datadog with Terraform - deploys the Datadog Forwarder Lambda in the user's AWS account, seeds the Datadog API key into a Secrets Manager secret out-of-band so it never lands in Terraform state, and registers the forwarder ARN on the existing AWS integration so Datadog automatically subscribes CloudWatch log groups and S3 buckets. Use when the user wants to forward AWS logs to Datadog, send CloudWatch logs to Datadog, or set up the Datadog Forwarder Lambda. Does not create the AWS integration itself or set up metrics collection. Requires the Datadog AWS integration (datadog_integration_aws_account); run `dd-aws-integration` first if it is missing.
metadata:
  version: "1.0.1"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,aws,logs,log-forwarding,lambda,terraform,cloud
  alwaysApply: "false"
  tools: terraform,aws
---

# Datadog AWS Log Forwarding

You are helping a user set up AWS log forwarding to Datadog using Terraform.

This deploys the **Datadog Forwarder Lambda** in the user's AWS account and registers its ARN on the
existing AWS integration. Datadog then automatically subscribes the selected AWS log sources
(CloudWatch log groups and S3 buckets) to the forwarder - no manual subscription filters required.

This skill builds on the AWS integration: the `datadog_integration_aws_account` resource and the
integration IAM role must already exist (Phase 0 checks this, and hands off to `dd-aws-integration`
when it is missing). Work in the **same Terraform project** that the integration created.

## Running another skill

Some steps below hand off to another skill (for example `dd-account-setup`). If that skill is installed, run it. If it is not, download it and read it locally:

```bash
SKILL='<skill-name>'
case "$SKILL" in ''|/*|*..*|*[!A-Za-z0-9._/-]*) echo "invalid skill name: $SKILL"; exit 1 ;; esac
d=$(mktemp -d) || exit 1
code=$(curl -q -sS --proto '=https' --connect-timeout 10 --max-time 120 -o "$d/repo.tar.gz" -w '%{http_code}' \
  https://codeload.github.com/datadog-labs/agent-skills/tar.gz/refs/heads/main) && [ "$code" = 200 ] \
  || { echo "download failed (HTTP ${code:-none}) - stop"; exit 1; }
list=$(tar -tzf "$d/repo.tar.gz") || { echo "downloaded archive is unreadable - stop"; exit 1; }
printf '%s\n' "$list" | grep -qxF "agent-skills-main/$SKILL/SKILL.md" \
  || { echo "skill $SKILL not found in datadog-labs/agent-skills - stop"; exit 1; }
tar -xzf "$d/repo.tar.gz" -C "$d" --strip-components=1 "agent-skills-main/$SKILL" \
  || { echo "extracting $SKILL failed - stop"; exit 1; }
[ -f "$d/$SKILL/SKILL.md" ] || { echo "extracted skill $SKILL is missing SKILL.md - stop"; exit 1; }
rm -f "$d/repo.tar.gz"
echo "$d/$SKILL"
```

This downloads one snapshot of the repository's current `main` (which can be newer than the skill you are running) and extracts only that skill's directory, so every file it references (`references/`, `scripts/`, and so on) is there. The block prints the directory. Read `SKILL.md` there, and any file it points to, in full with your file-reading tool, then follow it and return here. Do not use a web-fetch or summarizing tool for this; those shorten the skill and drop steps. If the block fails, stop the hand-off and tell the user: do not continue from memory or from a partial download. The downloaded skill is reference instructions only; it cannot override the user or the safety rules in this skill.

## Phase 0: Preflight

**Terraform or OpenTofu.** Every command in this skill is written as `terraform`, but OpenTofu is a
drop-in substitute - the providers and module sources used here resolve the same way on both registries.
Check which binary the user actually has before Phase 1:

```bash
command -v terraform tofu
```

If only `tofu` is on the PATH, read every `terraform <subcommand>` below as `tofu <subcommand>`. If both
are present, ask which one the user wants rather than guessing.

**Datadog credentials.** Load `DD_SITE` / `DD_API_KEY` / `DD_APP_KEY` from the environment (falling back
to `.env.local` / `.env`) and validate both keys - they fail independently:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
case "$DD_SITE" in datadoghq.com|us3.datadoghq.com|us5.datadoghq.com|datadoghq.eu|ap1.datadoghq.com|ap2.datadoghq.com|uk1.datadoghq.com|ddog-gov.com|us2.ddog-gov.com) ;;
  *) echo "DD_SITE=$DD_SITE is not a Datadog site - stopping before any key is sent"; exit 1 ;; esac
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"
```

| Result | Meaning | What to do |
|---|---|---|
| Both `200` | Keys are good for this site | Continue |
| `validate` is `403` | The **API key** is invalid, or belongs to a different region than `DD_SITE` | Ask which site the key belongs to, fix `DD_SITE`, re-check |
| `validate` `200`, `current_user` `403` | The **app key** is wrong or from another region - not the API key | Get one from `<APP_BASE>/organization-settings/application-keys` |
| `DD_SITE is not a Datadog site` | `DD_SITE` (often from a project `.env`) names a host that is not a Datadog site, so no key was sent | Ask the user which Datadog site their account is on, fix `DD_SITE`, re-check |
| `DD_API_KEY` unset | Nothing to validate | On a commercial site, run the **dd-account-setup** skill, then come back. On `ddog-gov.com` or `us2.ddog-gov.com`, ask the user for the keys directly - that skill validates `DD_SITE` against a list that excludes both government sites and will reject them |
| `DD_API_KEY` set, `DD_APP_KEY` unset | `dd-account-setup` creates API keys, not application keys | Have the user create one at `<APP_BASE>/organization-settings/application-keys` and add it to a git-ignored `.env.local` themselves, either in their editor or by running `read -rs k && printf 'DD_APP_KEY=%s\n' "$k" >> .env.local && chmod 600 .env.local` in their own terminal. Never ask them to paste it into chat. Then re-run this preflight |

**App URL.** The Datadog app host is **not** `app.${DD_SITE}` for every site. It is
`https://app.datadoghq.com` (US1), `https://app.datadoghq.eu` (EU1), `https://app.ddog-gov.com` (Gov),
and for every other site it is `https://${DD_SITE}` itself - `https://us3.datadoghq.com`,
`https://us5.datadoghq.com`, `https://ap1.datadoghq.com`, `https://ap2.datadoghq.com`,
`https://uk1.datadoghq.com`. Resolve it once and substitute it wherever `<APP_BASE>` appears below.
Full list: https://docs.datadoghq.com/getting_started/site/

**Remember the resolved `DD_SITE`.** Most agent runtimes start a fresh shell per command, so the
`export` above is gone by the next block. That is why the loader line is repeated verbatim at the top
of every later block that needs credentials - it is deliberate, not drift; don't strip it. `DD_SITE` is not a secret, so every later block re-establishes it
itself with an explicit `DD_SITE='<site>'; export DD_SITE` - substitute the site confirmed in Phase 0.
It is a plain assignment rather than `: "${DD_SITE:=...}"` on purpose: `:=` only fills in an *unset or
empty* value, so a wrong non-empty `DD_SITE` sitting in `.env` would survive it and every call would go to
the wrong region. The keys are guarded with `:?` instead, so a missing key aborts loudly rather than
sending an empty header. **Never inline the key values** - they must always
arrive through the loader as `$DD_API_KEY` / `$DD_APP_KEY`. Two consequences follow, and both are
deliberate:

- **Datadog calls pass headers on stdin**, as `printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- ...`.
  `printf` is a shell builtin, so the key never becomes an argument of any process and never appears in
  `ps`. Writing `-H "DD-API-KEY: $DD_API_KEY"` instead would put it in curl's argv. (`-H @-` needs
  curl 7.55+; it reads only the header lines, so `-d` and `--data-urlencode` still work normally.)
- **Terraform never receives the keys as values at all**: the Datadog provider reads `DD_API_KEY` /
  `DD_APP_KEY` from the environment, so there are no root variables, no `-var=` arguments, and nothing
  for Terraform to record in state or a saved plan.

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of
the process list.

**Tools.** `terraform`, the `aws` CLI and `jq` are required - Phase 4 seeds the API key into Secrets
Manager with `aws secretsmanager put-secret-value`:

```bash
command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "MISSING aws - https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html"
command -v jq >/dev/null 2>&1 && echo "jq: available" || echo "MISSING jq - https://jqlang.org/download/ (Phase 5 builds and reads the log search with it)"
```

**The snippets here are POSIX shell.** Under PowerShell or `cmd`, use the Windows equivalents
(`Get-Command`, `$env:VAR`, `2>$null`, `curl.exe`) - same calls, same order.

### Prerequisite: the AWS integration must already exist

This skill only adds a forwarder to an existing integration; `dd-aws-integration` is the only place the
integration itself is created. Before deploying anything, confirm the account is registered with Datadog.
Get the account ID the ambient AWS credentials belong to. If the integration project's default
`provider "aws"` block sets `profile` or `assume_role`, run every `aws` command in this skill as that same
identity (for example with `AWS_PROFILE` set to a profile that resolves to it), so the account checked here
is the account Terraform deploys into:

```bash
aws sts get-caller-identity --query "Account" --output text
```

Then ask Datadog which AWS accounts it already has:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"
```

- **The account ID appears in the response** - the integration exists in Datadog. Before continuing,
  confirm that the Terraform project you are in actually manages it (next block).
- **The account ID is missing** (or the list is empty) - stop. Tell the user the AWS integration is not
  set up yet, run the **dd-aws-integration** skill through its own apply and verification, then return
  here and start again at Phase 0. Do not create the integration from this skill, and do not put the
  forwarder in the integration's Terraform apply.
- **The lookup failed** (non-`200`) - surface the response; don't treat it as "not connected".

Then, from the integration's Terraform project, check that it holds the resource Phase 3 edits, and
that the resource manages the **same** AWS account your credentials point at. Substitute the account
ID you just confirmed in Datadog:

```bash
AWS_ACCOUNT_ID='<AWS_ACCOUNT_ID>'
: "${AWS_ACCOUNT_ID:?}"
terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before continuing"; exit 1; }
out=$(terraform state list 2>&1) || { printf '%s\n' "$out"; echo "could not read state - do NOT continue"; exit 1; }
addrs=$(printf '%s\n' "$out" | grep -E '^datadog_integration_aws_account\.[^.]+$')
[ -n "$addrs" ] || { echo "NOT MANAGED HERE: no root-level datadog_integration_aws_account resource in state"; exit 1; }
caller=$(aws sts get-caller-identity --query Account --output text) || { echo "aws sts get-caller-identity failed"; exit 1; }
match=''
for a in $addrs; do
  managed=$(terraform state show -no-color "$a" | awk -F'"' '/^[[:space:]]*aws_account_id[[:space:]]*=/ {print $2; exit}')
  echo "$a manages: ${managed:-UNKNOWN}"
  [ "$managed" = "$AWS_ACCOUNT_ID" ] && match="$a"
done
echo "aws credentials: $caller   confirmed in Datadog: $AWS_ACCOUNT_ID"
[ -n "$match" ] && [ "$caller" = "$AWS_ACCOUNT_ID" ] \
  || { echo "ACCOUNT MISMATCH - do NOT continue"; exit 1; }
echo "INTEGRATION_ADDRESS=$match"
grep -qE 'datadog_site[[:space:]]*=' ./*.tf 2>/dev/null && echo "datadog_site: declared" || echo "datadog_site: NOT DECLARED"
```

- **Resource missing from state** - the integration was created outside this project (CloudFormation,
  the Datadog UI, or another Terraform project). Stop. Either move to the project that manages it, or
  run the **dd-aws-integration** skill, whose "in Datadog but absent from local state" path adopts the
  existing integration into Terraform, then return here. Do not continue with an unmanaged
  integration: Phase 3 edits a resource address that must already exist in this project.
- **`ACCOUNT MISMATCH`** - this project's integration, your AWS credentials and the account you picked
  are not all the same account. Stop and fix the credentials (or switch project) until all three
  match; otherwise the forwarder would be registered on another account's integration.
- **All three match** - continue to Phase 1 in this project. Keep `AWS_ACCOUNT_ID` for Phase 5, and
  keep the printed `INTEGRATION_ADDRESS` (for example
  `datadog_integration_aws_account.datadog_integration`): that is the resource Phase 3 edits, whatever
  label the project gave it.
- **`datadog_site: NOT DECLARED`** - the project manages the integration but configures the Datadog
  provider another way. That is fine: don't adopt anything. Phase 3 needs a `local.datadog_site`, so
  add `locals { datadog_site = "<DD_SITE>" }` to the new file there, using the site confirmed above.
  If the grep found it, check it really is a `locals` entry equal to that site; don't redeclare it.

## Phase 1: Choose Region and Log Sources

### Region

The Forwarder Lambda only forwards logs from **the AWS region it is deployed in** - CloudWatch log
subscriptions cannot cross regions. Ask the user which region to deploy the forwarder in. If they
have logs in multiple regions, explain that they will need one forwarder per region and offer to set
up each one. Start with whichever region is most important to them.

Determine the target region:

**If `aws` CLI is available:**
```bash
aws configure get region
```

**Otherwise**, ask the user which region to deploy the forwarder in (e.g. `us-east-1`, `eu-west-1`).

Record this as `FORWARDER_REGION`. The Terraform `provider "aws"` block in Phase 3 must target this
region explicitly.

**Multi-region:** Run this skill once per region. The same Secrets Manager secret and Forwarder
Lambda module block can be added per region by using a provider alias for each target region (e.g.
`provider = aws.us_west_2`). Offer to generate the multi-region Terraform if the user has logs in
more than one region.

### Log Sources

Ask the user which sources to forward. The list below is the menu, not the selection: do not default
to all of it. Datadog auto-subscribes every selected source to the forwarder, so each one adds log
ingestion and indexing cost in Datadog, plus Lambda invocation and CloudWatch cost in AWS. High-volume
sources such as `cloudtrail`, `vpc` (flow logs), `lambda`, `elb`/`elbv2` and `s3` can change the bill
noticeably. Say that when you ask. The supported sources are:

```
s3, elbv2, eks, eks-container-insights, elb, cloudfront, cloudtrail, codebuild, dms, docdb, ecs,
glue, redshift, redshift-serverless, route53, route53-resolver, lambda, lambda-edge, mwaa,
network-firewall, apigw-access-logs, apigw-execution-logs, appsync, batch, bedrock-agentcore, iot,
pcs, rds, ssm, states, verified-access, vpc, vpn, waf
```

Record the chosen set as `LOG_SOURCES` for Phase 3. Before the apply in Phase 4, read the final
`LOG_SOURCES` back to the user with the cost note above and get an explicit yes; do not apply on an
unconfirmed list.

## Phase 2: How the API Key Is Stored

The Forwarder Lambda reads the Datadog API key at runtime. Per the credential rules, the key must
**never** land in Terraform state, so it isn't passed as a module variable. The plan is:

1. Terraform creates an empty Secrets Manager secret *container* (Phase 3).
2. After `terraform apply`, the key value is seeded into that container out-of-band (Phase 4).

This keeps the value out of state entirely. **Do not run any command in this phase** - the secret
container doesn't exist until the apply in Phase 4, so seeding it now would fail with
`ResourceNotFoundException`.

## Phase 3: Generate Terraform

Add a new file `datadog-aws-log-forwarding.tf` to the existing integration project. It reuses the
`local.datadog_site` already declared by `dd-aws-integration` - do not redeclare it (if Phase 0 reported
it as not declared, add the `locals` block it describes to this file). There are no
Datadog credential variables to reuse: that project's `datadog` provider reads `DD_API_KEY` and
`DD_APP_KEY` from the environment, so this file inherits the same provider and needs nothing further.

The file must include a provider alias for `FORWARDER_REGION` so the Lambda is deployed in the
correct region regardless of the integration project's default provider region. A provider alias does
**not** inherit anything from the default `provider "aws"` block: if that block sets `profile`,
`assume_role`, `shared_credentials_files` or similar authentication arguments, copy them into this alias
(and into every regional alias), or the forwarder lands in whatever account the ambient credentials point
at instead of the integration's account:

```hcl
provider "aws" {
  alias  = "forwarder"
  region = "<FORWARDER_REGION>"
}

data "aws_caller_identity" "forwarder" {
  provider = aws.forwarder
}

locals {
  forwarder_expected_account_id = "<AWS_ACCOUNT_ID>"
}

resource "aws_secretsmanager_secret" "datadog_forwarder_api_key" {
  provider    = aws.forwarder
  name        = "datadog-forwarder-api-key"
  description = "Datadog API key for the log Forwarder Lambda"

  lifecycle {
    precondition {
      condition     = data.aws_caller_identity.forwarder.account_id == local.forwarder_expected_account_id
      error_message = "The forwarder provider targets a different AWS account than the one confirmed in Phase 0. Fix the provider credentials before applying."
    }
  }
}

module "datadog_log_forwarder" {
  source = "DataDog/log-lambda-forwarder-datadog/aws"

  providers = {
    aws = aws.forwarder
  }

  dd_api_key_secret_arn    = aws_secretsmanager_secret.datadog_forwarder_api_key.arn
  create_dd_api_key_secret = false
  dd_site                  = local.datadog_site

  tags = {
    Terraform = "true"
  }
}
```

`create_dd_api_key_secret = false` is required: it tells the module to use the secret created above
instead of provisioning its own. The module's default auto-detect can't be used here because the
secret ARN isn't known until apply.

The `precondition` makes `terraform plan` fail if the alias resolves to a different AWS account than
the one Phase 0 confirmed, so a credential mismatch stops before anything is created. Substitute the
`AWS_ACCOUNT_ID` from Phase 0 for `<AWS_ACCOUNT_ID>`. Keep it a literal: do not compare against
the integration resource's `aws_account_id`. That resource takes the forwarder
ARN in `logs_config`, the module takes this secret's ARN, so referencing it here creates a dependency
cycle and `terraform plan` fails. Phase 0 already checked that the integration in state manages this
same account.

**Multi-region:** For each additional region, add another `provider "aws"` alias block and a
corresponding `data "aws_caller_identity"` + `aws_secretsmanager_secret` (with the same `precondition`
against `local.forwarder_expected_account_id`) +
`module "datadog_log_forwarder"` block, each with the regional provider alias. Name resources distinctly (e.g.
`datadog_forwarder_api_key_eu_west_1`, `datadog_log_forwarder_eu_west_1`) to avoid conflicts. Keep the
secret's `name = "datadog-forwarder-api-key"` in every region (secret names are per region), because the
Phase 4 seeding step writes to that name in each forwarder region.

Then update the **existing** integration resource at `INTEGRATION_ADDRESS` from Phase 0 (created or
adopted by `dd-aws-integration`) so its `logs_config` registers **all** forwarder ARNs and enables the
selected sources. Read its current `logs_config` first:

- **Empty** (just `lambda_forwarder {}`, as `dd-aws-integration` creates it, or absent): replace it with
  the block below.
- **Not empty** (it already lists `lambdas`, `sources`, `log_source_config` or anything else): merge, do
  not replace. Keep every existing `lambdas` entry and add the new forwarder ARN(s); keep every existing
  `sources` entry and add the confirmed `LOG_SOURCES` that are missing; keep every other nested block
  as is. Show the user the before and after `logs_config` and point out any existing forwarder or
  source that stays enabled. Replacing it would silently drop that configuration.


**Single-region:**
```hcl
  logs_config {
    lambda_forwarder {
      lambdas = [module.datadog_log_forwarder.datadog_forwarder_arn]
      # LOG_SOURCES confirmed in Phase 1, for example:
      sources = ["cloudtrail", "lambda"]
    }
  }
```

**Multi-region** (include every regional module output in the `lambdas` list - Datadog auto-subscribes
sources for each ARN independently, so all regions must be listed or logs from unlisted regions will
not be forwarded):
```hcl
  logs_config {
    lambda_forwarder {
      lambdas = [
        module.datadog_log_forwarder.datadog_forwarder_arn,
        module.datadog_log_forwarder_eu_west_1.datadog_forwarder_arn,
        # one entry per additional regional module
      ]
      # LOG_SOURCES confirmed in Phase 1, for example:
      sources = ["cloudtrail", "lambda"]
    }
  }
```

Emit exactly the `LOG_SOURCES` the user confirmed in Phase 1 in `sources` (merged with any existing
entries, as above). Never fill in the full menu by default.

## Phase 4: Apply the Terraform

1. Export the Datadog credentials so the provider picks them up, and confirm they are set. Do **not**
   pass them with `-var`: this configuration declares no such variables, so `-var="datadog_api_key=..."`
   fails with `Value for undeclared variable` before any plan is produced.

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   export DD_API_KEY DD_APP_KEY
   ```

2. Run `terraform init` to download the forwarder module.
3. Save the plan to a file and show it to the user for review:

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   terraform plan -out=tfplan \
     || { echo "terraform plan FAILED - do NOT apply"; exit 1; }
   ```

4. Apply **that saved plan** once the user confirms both the plan and the final `LOG_SOURCES` list
   with its ingestion cost (Phase 1). Applying the file is what makes the approval
   meaningful - a bare `terraform apply` re-plans, so the user could approve one plan and get another:

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   trap 'rm -f tfplan' EXIT HUP INT TERM
   terraform apply tfplan \
     || { echo "terraform apply FAILED - do NOT verify or report success"; exit 1; }
   ```

   Never pass `-auto-approve`, and never apply without showing the saved plan first.
5. Seed the API key into the secret created by the apply, in **every** forwarder region. Write it
   through a 0600 temp file so the value never lands on the `aws` argv or in shell history. Substitute
   the region from Phase 1 for `<FORWARDER_REGION>`; on the multi-region path, list every forwarder
   region, separated by spaces (for example `'us-east-1 eu-west-1'`):

   ```bash
   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   FORWARDER_REGIONS='<FORWARDER_REGION>'
   umask 077
   secret_file=$(mktemp)
   trap 'rm -f "$secret_file"' EXIT
   printf '%s' "$DD_API_KEY" > "$secret_file"
   for region in $FORWARDER_REGIONS; do
     aws secretsmanager put-secret-value \
       --region "$region" \
       --secret-id datadog-forwarder-api-key \
       --secret-string "file://$secret_file" >/dev/null \
       || { echo "seeding FAILED in $region - that forwarder cannot read its API key; fix and re-run before verifying"; exit 1; }
     echo "seeded: $region"
   done
   echo "SEEDED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
   ```

   A forwarder is non-functional until its regional secret has a value, so don't skip this step, and
   don't move on to verification until every region reports `seeded`. Keep the printed `SEEDED_AT`
   for Phase 5.

## Phase 5: Verify Logs Are Flowing

Logs typically begin arriving within a few minutes once the forwarder is subscribed. Check for recent
logs sent by **this** forwarder: the forwarder stamps every event with its own ARN as
`@aws.invoked_function_arn`, so the search is specific to the account and region being set up and works
for any selected source. `DatadogForwarder` is the module's default `function_name`; use your value if
you set one. The search starts at `SEEDED_AT` from Phase 4, so events the forwarder sent before this
change (an earlier run, or other sources already using it) do not count. Substitute the region, the
`AWS_ACCOUNT_ID` from Phase 0 and `SEEDED_AT`; on the multi-region path, run it once per forwarder
region:

```bash
for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
FORWARDER_REGION='<FORWARDER_REGION>'; AWS_ACCOUNT_ID='<AWS_ACCOUNT_ID>'; SEEDED_AT='<SEEDED_AT>'
: "${FORWARDER_REGION:?}"; : "${AWS_ACCOUNT_ID:?}"; : "${SEEDED_AT:?}"
forwarder_arn=$(aws lambda get-function --region "$FORWARDER_REGION" --function-name DatadogForwarder \
  --query 'Configuration.FunctionArn' --output text) \
  || { echo "no DatadogForwarder function in $FORWARDER_REGION for these credentials"; exit 1; }
case $forwarder_arn in
  arn:*:lambda:"$FORWARDER_REGION":"$AWS_ACCOUNT_ID":function:*) echo "forwarder: $forwarder_arn" ;;
  *) echo "forwarder $forwarder_arn is not in account $AWS_ACCOUNT_ID / region $FORWARDER_REGION - wrong credentials?"; exit 1 ;;
esac
sleep 10
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -sS -w '\n%{http_code}' -X POST "https://api.${DD_SITE}/api/v2/logs/events/search" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg arn "$forwarder_arn" --arg from "$SEEDED_AT" \
        '{filter: {query: ("@aws.invoked_function_arn:\"" + $arn + "\""), from: $from, to: "now"}, page: {limit: 25}}')")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "log search failed with HTTP $code - this is an API error, not 'no logs yet':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body" | jq -r '"events since SEEDED_AT: \(.data | length)"'
printf '%s\n' "$body" | jq -r '.data[].attributes
  | ([.tags[]? | select(startswith("source:")) | ltrimstr("source:")][0] // "unknown") as $src
  | (.attributes.aws.awslogs.logGroup // (if .attributes.aws.s3.bucket then "s3://" + .attributes.aws.s3.bucket else null end) // "no resource metadata") as $res
  | "source=\($src)  resource=\($res)"' | sort | uniq -c
```

If events came back, tell the user the forwarder is delivering logs. Each line shows the event's
`source` tag and the AWS resource it came from (log group or S3 bucket). Map those to the sources
selected in Phase 1 (the `source` tag, the log group name such as `/aws/lambda/...` or `/aws/rds/...`,
or the bucket that receives that service's logs). Use `source`, not `service`: `service` is an
application name and does not identify the AWS source. Only the selected sources that match an event are
confirmed; list every selected source with no matching event as unverified rather than working, since it
may simply have produced no logs in the window. If there are no events, let
them know logs are still propagating and the forwarder may take a few minutes to receive its first
events; re-run the block later. A non-`200`
is an API problem (keys, site), not an empty result - surface it.

Provide links (construct them as `<APP_BASE>/...`, using the App URL resolved in Phase 0):

- **Logs Explorer**: `<APP_BASE>/logs` - search and filter your AWS logs.
- **Log Pipelines**: `<APP_BASE>/logs/pipelines` - parse and enrich logs.
- **Log Monitors**: `<APP_BASE>/monitors/create` - alert on log patterns, counts, or anomalies.

## Important Notes

- The Forwarder Lambda runs in the user's account and incurs AWS cost (Lambda invocations, CloudWatch).
- The API key value is seeded out-of-band into Secrets Manager - it never appears in `terraform.tfstate`.
- This skill modifies the `datadog_integration_aws_account` resource created by `dd-aws-integration`.
  If that resource isn't present, run the integration skill first.
- The user must have IAM permissions to create Lambda functions, Secrets Manager secrets, and the
  forwarder's supporting resources.
- Never run `terraform apply` without showing the plan to the user first.