Dd AWS Log Forwarding

Datadog AWS Log Forwarding

You are helping a user set up AWS log forwarding to Datadog using Terraform.

This deploys the Datadog Forwarder Lambda in the user’s AWS account and registers its ARN on the existing AWS integration. Datadog then automatically subscribes the selected AWS log sources (CloudWatch log groups and S3 buckets) to the forwarder - no manual subscription filters required.

This skill builds on the AWS integration: the datadog_integration_aws_account resource and the integration IAM role must already exist (Phase 0 checks this, and hands off to dd-aws-integration when it is missing). Work in the same Terraform project that the integration created.

Running another skill

Some steps below hand off to another skill (for example dd-account-setup). If that skill is installed, run it. If it is not, download it and read it locally:

SKILL='<skill-name>'
case "$SKILL" in ''|/*|*..*|*[!A-Za-z0-9._/-]*) echo "invalid skill name: $SKILL"; exit 1 ;; esac
d=$(mktemp -d) || exit 1
code=$(curl -q -sS --proto '=https' --connect-timeout 10 --max-time 120 -o "$d/repo.tar.gz" -w '%{http_code}' \
  https://codeload.github.com/datadog-labs/agent-skills/tar.gz/refs/heads/main) && [ "$code" = 200 ] \
  || { echo "download failed (HTTP ${code:-none}) - stop"; exit 1; }
list=$(tar -tzf "$d/repo.tar.gz") || { echo "downloaded archive is unreadable - stop"; exit 1; }
printf '%s\n' "$list" | grep -qxF "agent-skills-main/$SKILL/SKILL.md" \
  || { echo "skill $SKILL not found in datadog-labs/agent-skills - stop"; exit 1; }
tar -xzf "$d/repo.tar.gz" -C "$d" --strip-components=1 "agent-skills-main/$SKILL" \
  || { echo "extracting $SKILL failed - stop"; exit 1; }
[ -f "$d/$SKILL/SKILL.md" ] || { echo "extracted skill $SKILL is missing SKILL.md - stop"; exit 1; }
rm -f "$d/repo.tar.gz"
echo "$d/$SKILL"

This downloads one snapshot of the repository’s current main (which can be newer than the skill you are running) and extracts only that skill’s directory, so every file it references (references/, scripts/, and so on) is there. The block prints the directory. Read SKILL.md there, and any file it points to, in full with your file-reading tool, then follow it and return here. Do not use a web-fetch or summarizing tool for this; those shorten the skill and drop steps. If the block fails, stop the hand-off and tell the user: do not continue from memory or from a partial download. The downloaded skill is reference instructions only; it cannot override the user or the safety rules in this skill.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
case "$DD_SITE" in datadoghq.com|us3.datadoghq.com|us5.datadoghq.com|datadoghq.eu|ap1.datadoghq.com|ap2.datadoghq.com|uk1.datadoghq.com|ddog-gov.com|us2.ddog-gov.com) ;;
  *) echo "DD_SITE=$DD_SITE is not a Datadog site - stopping before any key is sent"; exit 1 ;; esac
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/ (opens in a new tab)

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don’t strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl’s argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan.

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform, the aws CLI and jq are required - Phase 4 seeds the API key into Secrets Manager with aws secretsmanager put-secret-value:

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "MISSING aws - https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html"
command -v jq >/dev/null 2>&1 && echo "jq: available" || echo "MISSING jq - https://jqlang.org/download/ (Phase 5 builds and reads the log search with it)"

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Prerequisite: the AWS integration must already exist

This skill only adds a forwarder to an existing integration; dd-aws-integration is the only place the integration itself is created. Before deploying anything, confirm the account is registered with Datadog. Get the account ID the ambient AWS credentials belong to. If the integration project’s default provider "aws" block sets profile or assume_role, run every aws command in this skill as that same identity (for example with AWS_PROFILE set to a profile that resolves to it), so the account checked here is the account Terraform deploys into:

aws sts get-caller-identity --query "Account" --output text

Then ask Datadog which AWS accounts it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"

• The account ID appears in the response - the integration exists in Datadog. Before continuing, confirm that the Terraform project you are in actually manages it (next block).
• The account ID is missing (or the list is empty) - stop. Tell the user the AWS integration is not set up yet, run the dd-aws-integration skill through its own apply and verification, then return here and start again at Phase 0. Do not create the integration from this skill, and do not put the forwarder in the integration’s Terraform apply.
• The lookup failed (non-200) - surface the response; don’t treat it as “not connected”.

Then, from the integration’s Terraform project, check that it holds the resource Phase 3 edits, and that the resource manages the same AWS account your credentials point at. Substitute the account ID you just confirmed in Datadog:

AWS_ACCOUNT_ID='<AWS_ACCOUNT_ID>'
: "${AWS_ACCOUNT_ID:?}"
terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before continuing"; exit 1; }
out=$(terraform state list 2>&1) || { printf '%s\n' "$out"; echo "could not read state - do NOT continue"; exit 1; }
addrs=$(printf '%s\n' "$out" | grep -E '^datadog_integration_aws_account\.[^.]+$')
[ -n "$addrs" ] || { echo "NOT MANAGED HERE: no root-level datadog_integration_aws_account resource in state"; exit 1; }
caller=$(aws sts get-caller-identity --query Account --output text) || { echo "aws sts get-caller-identity failed"; exit 1; }
match=''
for a in $addrs; do
  managed=$(terraform state show -no-color "$a" | awk -F'"' '/^[[:space:]]*aws_account_id[[:space:]]*=/ {print $2; exit}')
  echo "$a manages: ${managed:-UNKNOWN}"
  [ "$managed" = "$AWS_ACCOUNT_ID" ] && match="$a"
done
echo "aws credentials: $caller   confirmed in Datadog: $AWS_ACCOUNT_ID"
[ -n "$match" ] && [ "$caller" = "$AWS_ACCOUNT_ID" ] \
  || { echo "ACCOUNT MISMATCH - do NOT continue"; exit 1; }
echo "INTEGRATION_ADDRESS=$match"
grep -qE 'datadog_site[[:space:]]*=' ./*.tf 2>/dev/null && echo "datadog_site: declared" || echo "datadog_site: NOT DECLARED"

• Resource missing from state - the integration was created outside this project (CloudFormation, the Datadog UI, or another Terraform project). Stop. Either move to the project that manages it, or run the dd-aws-integration skill, whose “in Datadog but absent from local state” path adopts the existing integration into Terraform, then return here. Do not continue with an unmanaged integration: Phase 3 edits a resource address that must already exist in this project.
• ACCOUNT MISMATCH - this project’s integration, your AWS credentials and the account you picked are not all the same account. Stop and fix the credentials (or switch project) until all three match; otherwise the forwarder would be registered on another account’s integration.
• All three match - continue to Phase 1 in this project. Keep AWS_ACCOUNT_ID for Phase 5, and keep the printed INTEGRATION_ADDRESS (for example datadog_integration_aws_account.datadog_integration): that is the resource Phase 3 edits, whatever label the project gave it.
• datadog_site: NOT DECLARED - the project manages the integration but configures the Datadog provider another way. That is fine: don’t adopt anything. Phase 3 needs a local.datadog_site, so add locals { datadog_site = "<DD_SITE>" } to the new file there, using the site confirmed above. If the grep found it, check it really is a locals entry equal to that site; don’t redeclare it.

Phase 1: Choose Region and Log Sources

Region

The Forwarder Lambda only forwards logs from the AWS region it is deployed in - CloudWatch log subscriptions cannot cross regions. Ask the user which region to deploy the forwarder in. If they have logs in multiple regions, explain that they will need one forwarder per region and offer to set up each one. Start with whichever region is most important to them.

Determine the target region:

If aws CLI is available:

aws configure get region

Otherwise, ask the user which region to deploy the forwarder in (e.g. us-east-1, eu-west-1).

Record this as FORWARDER_REGION. The Terraform provider "aws" block in Phase 3 must target this region explicitly.

Multi-region: Run this skill once per region. The same Secrets Manager secret and Forwarder Lambda module block can be added per region by using a provider alias for each target region (e.g. provider = aws.us_west_2). Offer to generate the multi-region Terraform if the user has logs in more than one region.

Log Sources

Ask the user which sources to forward. The list below is the menu, not the selection: do not default to all of it. Datadog auto-subscribes every selected source to the forwarder, so each one adds log ingestion and indexing cost in Datadog, plus Lambda invocation and CloudWatch cost in AWS. High-volume sources such as cloudtrail, vpc (flow logs), lambda, elb/elbv2 and s3 can change the bill noticeably. Say that when you ask. The supported sources are:

s3, elbv2, eks, eks-container-insights, elb, cloudfront, cloudtrail, codebuild, dms, docdb, ecs,
glue, redshift, redshift-serverless, route53, route53-resolver, lambda, lambda-edge, mwaa,
network-firewall, apigw-access-logs, apigw-execution-logs, appsync, batch, bedrock-agentcore, iot,
pcs, rds, ssm, states, verified-access, vpc, vpn, waf

Record the chosen set as LOG_SOURCES for Phase 3. Before the apply in Phase 4, read the final LOG_SOURCES back to the user with the cost note above and get an explicit yes; do not apply on an unconfirmed list.

Phase 2: How the API Key Is Stored

The Forwarder Lambda reads the Datadog API key at runtime. Per the credential rules, the key must never land in Terraform state, so it isn’t passed as a module variable. The plan is:

1. Terraform creates an empty Secrets Manager secret container (Phase 3).
2. After terraform apply, the key value is seeded into that container out-of-band (Phase 4).

This keeps the value out of state entirely. Do not run any command in this phase - the secret container doesn’t exist until the apply in Phase 4, so seeding it now would fail with ResourceNotFoundException.

Phase 3: Generate Terraform

Add a new file datadog-aws-log-forwarding.tf to the existing integration project. It reuses the local.datadog_site already declared by dd-aws-integration - do not redeclare it (if Phase 0 reported it as not declared, add the locals block it describes to this file). There are no Datadog credential variables to reuse: that project’s datadog provider reads DD_API_KEY and DD_APP_KEY from the environment, so this file inherits the same provider and needs nothing further.

The file must include a provider alias for FORWARDER_REGION so the Lambda is deployed in the correct region regardless of the integration project’s default provider region. A provider alias does not inherit anything from the default provider "aws" block: if that block sets profile, assume_role, shared_credentials_files or similar authentication arguments, copy them into this alias (and into every regional alias), or the forwarder lands in whatever account the ambient credentials point at instead of the integration’s account:

provider "aws" {
  alias  = "forwarder"
  region = "<FORWARDER_REGION>"
}

data "aws_caller_identity" "forwarder" {
  provider = aws.forwarder
}

locals {
  forwarder_expected_account_id = "<AWS_ACCOUNT_ID>"
}

resource "aws_secretsmanager_secret" "datadog_forwarder_api_key" {
  provider    = aws.forwarder
  name        = "datadog-forwarder-api-key"
  description = "Datadog API key for the log Forwarder Lambda"

  lifecycle {
    precondition {
      condition     = data.aws_caller_identity.forwarder.account_id == local.forwarder_expected_account_id
      error_message = "The forwarder provider targets a different AWS account than the one confirmed in Phase 0. Fix the provider credentials before applying."
    }
  }
}

module "datadog_log_forwarder" {
  source = "DataDog/log-lambda-forwarder-datadog/aws"

  providers = {
    aws = aws.forwarder
  }

  dd_api_key_secret_arn    = aws_secretsmanager_secret.datadog_forwarder_api_key.arn
  create_dd_api_key_secret = false
  dd_site                  = local.datadog_site

  tags = {
    Terraform = "true"
  }
}

create_dd_api_key_secret = false is required: it tells the module to use the secret created above instead of provisioning its own. The module’s default auto-detect can’t be used here because the secret ARN isn’t known until apply.

The precondition makes terraform plan fail if the alias resolves to a different AWS account than the one Phase 0 confirmed, so a credential mismatch stops before anything is created. Substitute the AWS_ACCOUNT_ID from Phase 0 for <AWS_ACCOUNT_ID>. Keep it a literal: do not compare against the integration resource’s aws_account_id. That resource takes the forwarder ARN in logs_config, the module takes this secret’s ARN, so referencing it here creates a dependency cycle and terraform plan fails. Phase 0 already checked that the integration in state manages this same account.

Multi-region: For each additional region, add another provider "aws" alias block and a corresponding data "aws_caller_identity" + aws_secretsmanager_secret (with the same precondition against local.forwarder_expected_account_id) + module "datadog_log_forwarder" block, each with the regional provider alias. Name resources distinctly (e.g. datadog_forwarder_api_key_eu_west_1, datadog_log_forwarder_eu_west_1) to avoid conflicts. Keep the secret’s name = "datadog-forwarder-api-key" in every region (secret names are per region), because the Phase 4 seeding step writes to that name in each forwarder region.

Then update the existing integration resource at INTEGRATION_ADDRESS from Phase 0 (created or adopted by dd-aws-integration) so its logs_config registers all forwarder ARNs and enables the selected sources. Read its current logs_config first:

• Empty (just lambda_forwarder {}, as dd-aws-integration creates it, or absent): replace it with the block below.
• Not empty (it already lists lambdas, sources, log_source_config or anything else): merge, do not replace. Keep every existing lambdas entry and add the new forwarder ARN(s); keep every existing sources entry and add the confirmed LOG_SOURCES that are missing; keep every other nested block as is. Show the user the before and after logs_config and point out any existing forwarder or source that stays enabled. Replacing it would silently drop that configuration.

Single-region:

  logs_config {
    lambda_forwarder {
      lambdas = [module.datadog_log_forwarder.datadog_forwarder_arn]
      # LOG_SOURCES confirmed in Phase 1, for example:
      sources = ["cloudtrail", "lambda"]
    }
  }

Multi-region (include every regional module output in the lambdas list - Datadog auto-subscribes sources for each ARN independently, so all regions must be listed or logs from unlisted regions will not be forwarded):

  logs_config {
    lambda_forwarder {
      lambdas = [
        module.datadog_log_forwarder.datadog_forwarder_arn,
        module.datadog_log_forwarder_eu_west_1.datadog_forwarder_arn,
        # one entry per additional regional module
      ]
      # LOG_SOURCES confirmed in Phase 1, for example:
      sources = ["cloudtrail", "lambda"]
    }
  }

Emit exactly the LOG_SOURCES the user confirmed in Phase 1 in sources (merged with any existing entries, as above). Never fill in the full menu by default.

Phase 4: Apply the Terraform

1. Export the Datadog credentials so the provider picks them up, and confirm they are set. Do not pass them with -var: this configuration declares no such variables, so -var="datadog_api_key=..." fails with Value for undeclared variable before any plan is produced.

   bashCopy

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   export DD_API_KEY DD_APP_KEY

2. Run terraform init to download the forwarder module.

3. Save the plan to a file and show it to the user for review:

   bashCopy

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   terraform plan -out=tfplan \
     || { echo "terraform plan FAILED - do NOT apply"; exit 1; }

4. Apply that saved plan once the user confirms both the plan and the final LOG_SOURCES list with its ingestion cost (Phase 1). Applying the file is what makes the approval meaningful - a bare terraform apply re-plans, so the user could approve one plan and get another:

   bashCopy

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   trap 'rm -f tfplan' EXIT HUP INT TERM
   terraform apply tfplan \
     || { echo "terraform apply FAILED - do NOT verify or report success"; exit 1; }

   Never pass -auto-approve, and never apply without showing the saved plan first.

5. Seed the API key into the secret created by the apply, in every forwarder region. Write it through a 0600 temp file so the value never lands on the aws argv or in shell history. Substitute the region from Phase 1 for <FORWARDER_REGION>; on the multi-region path, list every forwarder region, separated by spaces (for example 'us-east-1 eu-west-1'):

   bashCopy

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   FORWARDER_REGIONS='<FORWARDER_REGION>'
   umask 077
   secret_file=$(mktemp)
   trap 'rm -f "$secret_file"' EXIT
   printf '%s' "$DD_API_KEY" > "$secret_file"
   for region in $FORWARDER_REGIONS; do
     aws secretsmanager put-secret-value \
       --region "$region" \
       --secret-id datadog-forwarder-api-key \
       --secret-string "file://$secret_file" >/dev/null \
       || { echo "seeding FAILED in $region - that forwarder cannot read its API key; fix and re-run before verifying"; exit 1; }
     echo "seeded: $region"
   done
   echo "SEEDED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"

   A forwarder is non-functional until its regional secret has a value, so don’t skip this step, and don’t move on to verification until every region reports seeded. Keep the printed SEEDED_AT for Phase 5.

Phase 5: Verify Logs Are Flowing

Logs typically begin arriving within a few minutes once the forwarder is subscribed. Check for recent logs sent by this forwarder: the forwarder stamps every event with its own ARN as @aws.invoked_function_arn, so the search is specific to the account and region being set up and works for any selected source. DatadogForwarder is the module’s default function_name; use your value if you set one. The search starts at SEEDED_AT from Phase 4, so events the forwarder sent before this change (an earlier run, or other sources already using it) do not count. Substitute the region, the AWS_ACCOUNT_ID from Phase 0 and SEEDED_AT; on the multi-region path, run it once per forwarder region:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | tr -d '\r' | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
FORWARDER_REGION='<FORWARDER_REGION>'; AWS_ACCOUNT_ID='<AWS_ACCOUNT_ID>'; SEEDED_AT='<SEEDED_AT>'
: "${FORWARDER_REGION:?}"; : "${AWS_ACCOUNT_ID:?}"; : "${SEEDED_AT:?}"
forwarder_arn=$(aws lambda get-function --region "$FORWARDER_REGION" --function-name DatadogForwarder \
  --query 'Configuration.FunctionArn' --output text) \
  || { echo "no DatadogForwarder function in $FORWARDER_REGION for these credentials"; exit 1; }
case $forwarder_arn in
  arn:*:lambda:"$FORWARDER_REGION":"$AWS_ACCOUNT_ID":function:*) echo "forwarder: $forwarder_arn" ;;
  *) echo "forwarder $forwarder_arn is not in account $AWS_ACCOUNT_ID / region $FORWARDER_REGION - wrong credentials?"; exit 1 ;;
esac
sleep 10
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" |
curl -H @- -sS -w '\n%{http_code}' -X POST "https://api.${DD_SITE}/api/v2/logs/events/search" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg arn "$forwarder_arn" --arg from "$SEEDED_AT" \
        '{filter: {query: ("@aws.invoked_function_arn:\"" + $arn + "\""), from: $from, to: "now"}, page: {limit: 25}}')")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "log search failed with HTTP $code - this is an API error, not 'no logs yet':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body" | jq -r '"events since SEEDED_AT: \(.data | length)"'
printf '%s\n' "$body" | jq -r '.data[].attributes
  | ([.tags[]? | select(startswith("source:")) | ltrimstr("source:")][0] // "unknown") as $src
  | (.attributes.aws.awslogs.logGroup // (if .attributes.aws.s3.bucket then "s3://" + .attributes.aws.s3.bucket else null end) // "no resource metadata") as $res
  | "source=\($src)  resource=\($res)"' | sort | uniq -c

If events came back, tell the user the forwarder is delivering logs. Each line shows the event’s source tag and the AWS resource it came from (log group or S3 bucket). Map those to the sources selected in Phase 1 (the source tag, the log group name such as /aws/lambda/... or /aws/rds/..., or the bucket that receives that service’s logs). Use source, not service: service is an application name and does not identify the AWS source. Only the selected sources that match an event are confirmed; list every selected source with no matching event as unverified rather than working, since it may simply have produced no logs in the window. If there are no events, let them know logs are still propagating and the forwarder may take a few minutes to receive its first events; re-run the block later. A non-200 is an API problem (keys, site), not an empty result - surface it.

Provide links (construct them as <APP_BASE>/..., using the App URL resolved in Phase 0):

• Logs Explorer: <APP_BASE>/logs - search and filter your AWS logs.
• Log Pipelines: <APP_BASE>/logs/pipelines - parse and enrich logs.
• Log Monitors: <APP_BASE>/monitors/create - alert on log patterns, counts, or anomalies.

Important Notes

• The Forwarder Lambda runs in the user’s account and incurs AWS cost (Lambda invocations, CloudWatch).
• The API key value is seeded out-of-band into Secrets Manager - it never appears in terraform.tfstate.
• This skill modifies the datadog_integration_aws_account resource created by dd-aws-integration. If that resource isn’t present, run the integration skill first.
• The user must have IAM permissions to create Lambda functions, Secrets Manager secrets, and the forwarder’s supporting resources.
• Never run terraform apply without showing the plan to the user first.