Subchapter 37.12
references/cloudwatch/application-signals-onboarding.mdMarkdown20 KBView on GitHub
Enable AWS Application Signals for a service that is not yet instrumented, by using ADOT (AWS Distro for OpenTelemetry) auto-instrumentation SDKs and making minimal, reviewable changes to the customer’s infrastructure-as-code, Dockerfiles, CI/CD workflows, and deployment manifests. This is the side of observability (turning an un-instrumented service into one that reports to Application Signals via ADOT). For querying, alarms, dashboards, or trace analysis on an already-instrumented service, use the other references.
Never modify application source code (.py, .js, .ts, .java, .cs). Only edit IaC, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests. Make the minimum changes needed and preserve existing configuration. Present changes for the user to review; do not run terraform apply, cdk deploy, or kubectl apply automatically.
Onboarding has two tiers. Apply the second only when it is supported for the platform + language.
| Tier | What it adds | Supported on |
|---|---|---|
| 1. Application Signals enablement (always) | ADOT auto-instrumentation: CloudWatch Observability add-on (EKS), CloudWatch Agent, IAM, the inject annotation / init container / SDK install | All platforms (EC2, ECS, EKS, Lambda) and all languages (Python, Node.js, Java, .NET) |
| 2. ServiceEvents extras (when supported) | Git & deployment metadata env vars (CI/CD propagation) + OTLP endpoints + Dynamic Instrumentation | EC2, ECS, EKS with Python, Node.js, Java only |
Minimum component versions for ServiceEvents (Tier 2). The base Application Signals (Tier 1) works on any recent version. ServiceEvents requires:
| Component | Minimum for ServiceEvents | Notes | Latest version links |
|---|---|---|---|
| CloudWatch Agent | 1.300070.0 (recommended — includes on-prem credential bugfix) or 1.300069.0 | Use latest by default; flag to the user if they are on an older version | — |
| CloudWatch Observability EKS add-on | v6.3.0 | Use latest by default; flag if the customer’s IaC pins an older version | — |
| ADOT Python SDK / ECS init container | 0.18.0 | pip: aws-opentelemetry-distro==0.18.0; ECR: adot-autoinstrumentation-python:v0.18.0 | releases (opens in a new tab) · ECR (opens in a new tab) |
| ADOT Node.js SDK / ECS init container | 0.12.0 | npm: @aws/aws-distro-opentelemetry-node-autoinstrumentation@0.12.0; ECR: adot-autoinstrumentation-node:v0.12.0 | releases (opens in a new tab) · ECR (opens in a new tab) |
| ADOT Java agent / ECS init container | 2.28.2 | jar: aws-opentelemetry-agent-2.28.2.jar; ECR: adot-autoinstrumentation-java:v2.28.2 | releases (opens in a new tab) · ECR (opens in a new tab) |
| ADOT .NET / ECS init container | ServiceEvents not supported on .NET | releases (opens in a new tab) · ECR (opens in a new tab) |
Tier 2 is NOT supported on Lambda or .NET. For a Lambda service, or a .NET service on any platform, do Tier 1 only — the service still gets Application Signals, just without the ServiceEvents metadata/OTLP/DI env vars. Do not add OTEL_AWS_SERVICE_EVENTS_*, OTEL_AWS_OTLP_*, or OTEL_AWS_DYNAMIC_INSTRUMENTATION_* env vars for Lambda or .NET.
Detect from the IaC and app code, and confirm with the user if ambiguous:
kind: Deployment), Helm charts, kubectl in scripts, Terraform aws_eks_*, the amazon-cloudwatch-observability add-on.containerDefinitions, Terraform aws_ecs_*.aws_lambda_function.aws_instance.requirements.txt/pyproject.toml/*.py → Python; package.json/*.ts/*.js → Node.js (nodejs); pom.xml/build.gradle/*.java → Java; *.csproj/*.sln/*.cs → .NET (dotnet).The amazon-cloudwatch-observability add-on injects ADOT auto-instrumentation via init containers and runs the CloudWatch Agent.
Prefer the EKS add-on (aws_eks_addon / CfnAddon) — do NOT introduce helm_release to replace an existing add-on (the add-on provides functionality the Helm chart alone does not, e.g. automatic OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT injection). If the user’s IaC already uses helm_release for this chart, work with their existing setup.
Check whether the add-on is already enabled. Present the user with these options and proceed based on their response:
You run it — offer to run the AWS CLI command yourself (requires CLI/credentials access and the cluster name + region from the IaC):
aws eks describe-addon --cluster-name <cluster-name> --addon-name amazon-cloudwatch-observability --region <region>A successful response means it exists; ResourceNotFoundException means it does not.
User runs it — ask the user to run the command above themselves or check the EKS console → Add-ons tab (opens in a new tab), and share the result.
User says it’s not enabled — proceed to add the add-on (see below).
User says it’s already enabled — proceed to the import step (see “Add-on already exists” below).
Add-on does NOT exist: add the aws_eks_addon / CfnAddon resource:
resource "aws_eks_addon" "cloudwatch_observability" {
cluster_name = ...
addon_name = "amazon-cloudwatch-observability"
# addon_version omitted = uses the latest default version (recommended).
# ServiceEvents requires v6.3.0+. If the customer's IaC pins an older version, flag it.
}Add-on already exists (Terraform): still add the resource above, and add a terraform import step to the CI/CD workflow before terraform apply so apply uses UpdateAddon instead of CreateAddon. Use || true so reruns don’t fail:
# Import existing CW Observability add-on into Terraform state (first run only; can be removed after).
# Add only this import line, BEFORE the workflow's existing `terraform apply` step, and mention that it can be removed after the first run as a comment.
terraform import -var="region=..." -var="cluster_name=..." \
aws_eks_addon.cloudwatch_observability <cluster-name>:amazon-cloudwatch-observability || trueAdd-on already exists (CDK): do NOT add it to CDK; no change needed.
Do NOT introduce helm_release, kubernetes, or helm provider resources for this purpose.
The CloudWatch Agent needs CloudWatchAgentServerPolicy and AWSXRayDaemonWriteAccess to send metrics, logs, and traces. When ServiceEvents Dynamic Instrumentation applies (Tier 2), also add a custom policy with application-signals:ListInstrumentationConfigurations and application-signals:ReportInstrumentationConfigurationStatus on Resource: "*".
Attach to the role the CloudWatch Agent uses, per platform:
EKS — terraform-aws-modules/eks/aws module (most common): add to iam_role_additional_policies:
resource "aws_iam_policy" "application_signals_di" {
name = "${var.cluster_name}-${var.region}-application-signals-di"
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"application-signals:ListInstrumentationConfigurations",
"application-signals:ReportInstrumentationConfigurationStatus"
]
# Resource = "*" is the recommended scope for Dynamic Instrumentation: these
# application-signals actions do not support resource-level permissions.
Resource = "*"
}]
})
}
eks_managed_node_groups = {
main = {
iam_role_additional_policies = {
CloudWatchAgentServerPolicy = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
AWSXRayDaemonWriteAccess = "arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess"
ApplicationSignalsDI = aws_iam_policy.application_signals_di.arn
}
}
}For raw aws_iam_role / ECS / EC2, attach the same three policies via aws_iam_role_policy_attachment. Use the exact managed-policy name AWSXRayDaemonWriteAccess (not AWSXRayWriteOnlyAccess). Omit the application_signals_di policy entirely for Lambda/.NET (Tier 1 only).
Note: the per-language guide (Step 4) may mention CloudWatchAgentServerPolicy but omit AWSXRayDaemonWriteAccess, or use a raw attachment pattern that doesn’t match the module’s iam_role_additional_policies syntax. Match the actual IaC pattern; prefer this step’s guidance if they conflict.
Read the guide for the detected combination and apply its instrumentation changes (the inject annotation on EKS, the ADOT init container on ECS, the SDK/agent install on EC2, the Lambda layer on Lambda):
references/appsignals-guides/<platform>-<language>.mdValid platforms: ec2, ecs, eks, lambda. Valid languages: python, nodejs, java, dotnet. Example: Python on EKS → references/cloudwatch/appsignals-guides/eks-python.md.
If the running agent has the AWS MCP server / Application Signals tooling available, get_enablement_guide can be used as a fallback for the same parameters.
For EC2/ECS/EKS with Python/Node.js/Java, add the ServiceEvents environment variables.
5a and 5b are the core of Tier 2 — apply them by default, do NOT ask the user whether to include them. They are part of what “enable ServiceEvents / onboard to Application Signals” means. They differ in how strict they are:
Only 5c and 5d are optional opt-in questions.
Add these 5 env vars, using the exact names below. See application-signals-cicd-metadata.md for the full propagation-chain procedure and per-provider patterns.
| Variable (exact name) | When set |
|---|---|
OTEL_AWS_SERVICE_EVENTS_GIT_REPO_URL | build-time (bake into image) |
OTEL_AWS_SERVICE_EVENTS_GIT_COMMIT_SHA | build-time (bake into image) |
OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_URL | deploy-time (runtime env var) |
OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_ID | deploy-time (runtime env var) |
OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_TIMESTAMP | deploy-time (runtime env var) |
ServiceEvents adds two OTLP endpoint env vars — OTEL_AWS_OTLP_LOGS_ENDPOINT and OTEL_AWS_OTLP_METRICS_ENDPOINT. These are in addition to (not replacements for) the base Application Signals exporter env vars the per-platform guide already sets (OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT). On ECS/EC2 a fully onboarded service ends up with all of them. All point at the CloudWatch Agent’s OTLP receiver on port 4316 (NOT the OpenTelemetry SDK default 4318). Where the two ServiceEvents vars are set depends on the platform:
| Variable | EKS | ECS / EC2 |
|---|---|---|
OTEL_AWS_OTLP_LOGS_ENDPOINT | Auto-injected by the CloudWatch Observability add-on — do NOT set as a pod env var | Set manually: http://localhost:4316/v1/logs (ECS sidecar / EC2), or the CloudWatch Agent host/IP on port 4316 (ECS daemon) |
OTEL_AWS_OTLP_METRICS_ENDPOINT | Auto-injected — do NOT set | Set manually: http://localhost:4316/v1/metrics (ECS sidecar / EC2), or the CloudWatch Agent host/IP on port 4316 (ECS daemon) |
EKS: do NOT manually set the OTLP endpoint env vars on the pod — the amazon-cloudwatch-observability add-on injects them into instrumented pods with the correct values. On EKS, Step 5b typically adds nothing to the Deployment manifest; the Step 5a metadata env vars are still set as usual.
Steps 5c and 5d are the only parts of onboarding to ask the user about — two separate, optional ServiceEvents features, both off by default and both Tier 2 (EC2/ECS/EKS × Python/Node.js/Java). (5a and 5b above are not opt-in questions — they are applied by default; see the Step 5 intro.) Ask the user about 5c and 5d each as its own distinct question before moving to Review — they are independent (the user may want neither, either, or both). Fold whatever the user opts into the same place as the other Step 5 env vars (k8s Deployment env, ECS container env, or EC2 process/userdata env), so Step 6 reviews the complete set.
Ask the user whether they want per-function (FunctionCall) telemetry for their own application code. It emits nothing by default — but not because a toggle is off: OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED is already true by default. What suppresses output is the empty OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE allowlist. The two work as a pair — with the flag on but no allowlist, the SDK installs the hooks and instruments nothing. So opting in means setting one env var (do NOT set the enable flag — it is already on):
| Variable | Value |
|---|---|
OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE | The only way to opt code in. Empty = nothing instrumented (there is no implicit default scope). On Node.js, a list entry of exactly * or ** is dropped (with a warning) as too broad — but partial wildcards (**/src/**, *.js) are fine. |
The match syntax differs per SDK — set it to the customer’s own application code, not third-party libraries:
| SDK | Form | Example |
|---|---|---|
| Java | Java package prefix (dot-separated; no wildcard needed) | com.example.simplesample, com.amazon.indico |
| Python | dotted module path + .* | indico.*, myapp.* |
| Node.js | path glob (minimatch) matched against the file’s absolute resolved path (NOT a module name) | **/indico/src/** — i.e. **/<app-dir>/src/** for code under <app-dir>/src/ |
Determining the value — inspect the customer’s source layout. PACKAGES_INCLUDE is the one onboarding value that depends on how the customer’s code is organized, so read the repo to derive it (reading source to determine config is allowed; the never-modify rule is about editing source, not looking at it). Per SDK:
src/main/java/<group>/<artifact>/…) or the package/namespace declarations and groupId in pom.xml/build.gradle. Use the top-level package that covers the customer’s own classes, e.g. com.amazon.indico.__init__.py, or the name/packages in pyproject.toml/setup.py) and append .*, e.g. myapp.*.src/, or main/exports in package.json) and build a path glob **/<app-dir>/src/**. Remember it matches the absolute runtime path, so anchor on a suffix that survives the build/deploy (the **/ prefix), not the repo-relative path.If the layout is ambiguous or spans multiple top-level packages, confirm the intended scope with the user rather than guessing — too broad an allowlist adds overhead and noise; too narrow misses functions. Prefer the customer’s own application packages over dependencies unless the user explicitly wants a dependency instrumented.
Node.js — the leading **/ is required, not optional. The SDK matches the pattern against the fully-resolved absolute path (e.g. /app/indico/src/handlers/order.js), which begins with deploy-specific prefixes the customer doesn’t control (/app, the WORKDIR, etc.). minimatch’s matchBase only helps for slash-free patterns; any pattern containing a / (like …/src/**) is anchored to the whole absolute path, so indico/src/** matches nothing. Lead with **/ to absorb the prefix (**/indico/src/**), or — less portably — hardcode the absolute path (/app/indico/src/**). Usually you want the customer’s own application code.
Ask the user — as a separate question from 5c — whether they want Dynamic Instrumentation. It shares OTEL_AWS_OTLP_LOGS_ENDPOINT with ServiceEvents. To enable, set OTEL_AWS_DYNAMIC_INSTRUMENTATION_ENABLED=true — on EKS either as a pod env var on the Deployment OR via the add-on’s autoInstrumentationConfiguration (configuration_values); on ECS/EC2 as a container/process env var. Leave it off (omit, or set false) unless the user wants it.
| Variable | EKS | ECS / EC2 |
|---|---|---|
OTEL_AWS_DYNAMIC_INSTRUMENTATION_ENABLED | Opt in by EITHER setting it true as a pod env var OR via the add-on’s autoInstrumentationConfiguration | Set true to opt in |
OTEL_AWS_DYNAMIC_INSTRUMENTATION_API_URL | Auto-injected — do NOT set | Only needed on ECS daemon (CloudWatch Agent host/IP on port 2000); default localhost:2000 works on ECS sidecar / EC2 |
Summarize all changes grouped by file, state the platform + language, list the env vars that will reach the app at runtime (including any optional 5c / 5d features the user opted into), and note build-time vs deploy-time. Explicitly call out anything that was NOT set — in particular any 5a git/deployment metadata vars skipped because their value couldn’t be sourced (which ones, and why, e.g. “no CI/CD provider detected to supply GIT_COMMIT_SHA“), so the user knows the metadata is partial and can wire it manually if they want full deployment correlation. Present for the user to review and commit. Do not deploy automatically.
AWSXRayDaemonWriteAccess.