Instrumentation

Before you start: confirm a Space exists in the target Region (aws cloudwatchomni list-spaces --region <region>); if none, stop and run ../spaces-and-domains.md first — instrumentation started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see.

Never modify application source code (.py, .js, .ts, .java, .cs). Only edit infrastructure-as-code, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests. Make the minimum change needed and preserve existing configuration. Present changes for the user to review; do not run terraform apply, cdk deploy, kubectl apply, helm install/helm upgrade, kubectl annotate, kubectl delete, or aws ecs update-service automatically. That applies to the collector step too: some of it is imperative commands rather than a diff, and those still belong to the user.

Scope

The default job is instrumentation only — getting the ADOT SDK loaded into the running workload and emitting telemetry. Standing up a destination is a separate, optional second step.

Where telemetry goes is deliberately not configured here. The SDK falls back to its spec defaults (http://localhost:4318 for OTLP/HTTP, http://localhost:4317 for OTLP/gRPC). Which of the two applies depends on the SDK: the Java agent defaults to OTLP/gRPC on 4317, while the Python, Node.js and .NET distros default to OTLP/HTTP on 4318. Instrumentation is complete without a destination, and you must never invent an endpoint.

When the user does need one, deploy a collector. Use collector.md and its platform guides to put a collector on EC2, ECS, or EKS and wire the app to it — the CloudWatch Agent by default, with upstream otelcol-contrib as the backup. The collector exports straight to CloudWatch’s own per-signal OTLP endpoints. ../data-forwarding-and-centralization.md is not an alternative here — it accepts no telemetry over OTLP, it forwards what is already in CloudWatch log groups into a Dataset.

Do not start down this path unprompted — instrumentation alone is a complete answer to “instrument my app”.

Do NOT add these

These belong to CloudWatch Application Signals and must not appear in an Omni instrumentation change. If the user explicitly wants Application Signals instead, stop and route to the aws-observability skill.

• OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT
• OTEL_AWS_SERVICE_EVENTS_* (git/deployment metadata, PACKAGES_INCLUDE)
• OTEL_AWS_OTLP_LOGS_ENDPOINT, OTEL_AWS_OTLP_METRICS_ENDPOINT
• OTEL_AWS_DYNAMIC_INSTRUMENTATION_*
• The amazon-cloudwatch-observability EKS add-on, and the instrumentation.opentelemetry.io/inject-* annotation that depends on it
• Any endpoint on the CloudWatch Agent’s port 4316 (a CloudWatch-specific port, not an OTel default)

One exception, on Lambda only: set OTEL_AWS_APPLICATION_SIGNALS_ENABLED=false explicitly. This is required, not stylistic — the layer’s otel-instrument wrapper defaults the variable to true when it is unset, so omitting it silently ENABLES Application Signals. See the Lambda guides.

A benign ServiceEvents WARN on ADOT Java 2.30.0. Even with none of these variables set, the Java agent logs OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED=true but OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE ... around four times during startup, interleaved with [SERVICE_EVENTS] INFO lines such as ServiceEventsInstrumentation initialized at startup that read as though it were enabled, and then ServiceEventsSpanProcessor not registered (serviceevents disabled). It defaults the flag internally and self-disables; nothing leaks. Expect the warning and leave it alone — do not “fix” it by setting OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED=false, which is one of the variables above.

IAM follows the destination, not the instrumentation

On EC2, ECS, and EKS, instrumentation itself needs no IAM changes — loading the SDK into a workload grants nothing, and the SDK is left pointing at its default local endpoint. Do not add CloudWatchAgentServerPolicy, AWSXRayDaemonWriteAccess, or application-signals:* permissions to those workloads.

That is not the same as saying the workload will never need permissions — it means instrumentation does not. Permissions follow whatever ends up talking to AWS:

• A destination the user already has — the permission belongs with it. Do not add a policy speculatively.
• A collector deployed via collector.md — the collector is the thing calling AWS, so it does get IAM: CloudWatchAgentServerPolicy on the instance profile (EC2), the task role (ECS), or an IRSA role (EKS). The application still gets nothing.

Lambda is different, and does get a policy here. The ADOT layer exports to the execution environment’s X-Ray receiver under active tracing, so the destination is already known and lives inside the function’s own configuration. The Lambda guides therefore grant the execution role xray:PutTraceSegments and xray:PutTelemetryRecords — via AWSXRayDaemonWriteAccess, or the inline grant CDK adds when tracing is enabled. CloudWatchLambdaApplicationSignalsExecutionRolePolicy is still out of scope, since Application Signals is disabled.

Step 1: Determine platform and language

Detect from the IaC and application code, and confirm with the user if ambiguous.

• EKS: k8s Deployment manifests (kind: Deployment), Helm charts, kubectl in scripts, Terraform aws_eks_*
• ECS: ECS task definitions, containerDefinitions, Terraform aws_ecs_*
• EC2: EC2 instances, userdata scripts, launch templates, Terraform aws_instance
• Language: requirements.txt/pyproject.toml/*.py → Python; package.json/*.ts/*.js → Node.js; pom.xml/build.gradle/*.java → Java; *.csproj/*.sln/*.cs → .NET

If the workload is an AI agent built on a framework (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI), stop — use ../omni-agents-instrumentation/omni-agents-instrumentation.md instead. That path instruments at the framework layer.

Step 2: Read the matching guide

Only if the user also needs a destination, continue to collector.md — it is language-agnostic and covers EC2, ECS, and EKS. Lambda is excluded: there is nowhere to run a collector in it, and its layer already exports to the execution environment’s X-Ray receiver.

How the SDK gets in

The mechanism differs by platform, but on ECS and EKS it is the same idea — an init container copies the SDK into a shared volume at startup, so the application image never changes:

For instrumentation alone, no cluster-level components are installed on EKS — no add-on and no operator; injection is written directly into the Deployment manifest. (If the user also asks for a destination, collector.md deploys the CloudWatch Agent by Helm, which necessarily brings its operator. That is expected there, and is not the amazon-cloudwatch-observability add-on.)

Lambda is the odd one out: the execution environment supplies an X-Ray receiver that the layer exports to, so unlike the other three platforms its export path is already wired. That is why the Lambda guides look different and why OTEL_AWS_APPLICATION_SIGNALS_ENABLED=false is set explicitly there.

Verifying without a receiver

Because no endpoint is configured, the SDK exports to localhost:4318 where nothing is listening. OTLP connection errors in the logs after instrumenting are expected and are not a failure — they mean the SDK loaded and is trying to export. Tell the user the errors will clear once a receiver exists.

Do not rely on “the SDK announces itself” as the check — it only works for Java. What a working deployment actually logs, by language:

For Python and .NET, verify by other means: the init container exiting 0, the loader variables present in the running process (printenv, or tr '\0' '\n' < /proc/1/environ — use PID 1 in a single-process container, since slim images often lack pgrep), and ultimately spans at the destination.

For .NET the strongest check is the SDK’s own Managed log above. If you also check paths, expand the variable inside the container — kubectl exec <pod> -- sh -c 'test -f "$CORECLR_PROFILER_PATH"'; unwrapped, it expands in your local shell where it is unset and always reports failure. Note DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE legitimately dangle in the container image.

If a collector was deployed in the same change, these errors should be absent — and on the CloudWatch Agent, all of them. The agent builds traces, metrics and logs pipelines from its opentelemetry section, so a correctly wired app logs no OTLP errors at all — a GET probe of /v1/logs returns 405 (method not allowed on a path that IS registered), not 404. Treat any OTLP error on the agent path as a real problem.

On the otelcol-contrib backup only: the hand-written Option A config in collector.md defines only traces and metrics pipelines, while the SDKs default OTEL_LOGS_EXPORTER=otlp — so the receiver returns 404 on /v1/logs and the app logs that permanently. That is expected there and is not a connectivity failure; traces flow fine alongside it. Either add a logs pipeline or set OTEL_LOGS_EXPORTER=none. Do not set OTEL_LOGS_EXPORTER=none on the CloudWatch Agent path — it would discard a signal the agent delivers for free.

Step 3: Set the service identity

Every guide sets these two, and nothing else about export:

Derive OTEL_SERVICE_NAME from the existing workload name in the IaC (deployment name, ECS service name, or the application name) rather than inventing one. If it cannot be determined, ask.

Custom metrics: what makes a metric queryable in Omni

Auto-instrumentation produces the standard request metrics on its own. A custom metric — a latency histogram, a queue depth, a business counter — is a code change the customer makes in the application, and this reference never edits source; so when the question is “how do I get my own metric to show up in Omni”, hand them the rule and the instrument to use rather than a file diff.

Omni serves metrics from a single OpenTelemetry-native, PromQL-queryable surface. A metric is on that surface when it arrives as an OTLP metric and carries the workload’s OTel resource attributes — service.name above all, which Omni exposes as the label "@resource.service.name" and uses to tie the metric to the service. Two paths put a metric there, and they are the same two the rest of this file wires:

• OTel SDK in the application — create the instrument through the OTel Metrics API (Meter → Histogram for latency, Counter for counts, Gauge for levels) and record to it. The ADOT SDK the guides above load already runs a MeterProvider exporting over OTLP, so a custom instrument rides the same exporter, endpoint, and resource as the auto-instrumented telemetry — no extra configuration in the app.
• An OTel Collector the application exports OTLP to — the SDK sends to the collector, which exports to CloudWatch’s OTLP metrics endpoint (https://monitoring.<region>.amazonaws.com/v1/metrics, collector.md).

Either way the exporter must reach a destination that delivers into the account’s Dataset in a Region with a Space; an SDK left at its localhost default records the metric and delivers it nowhere.

What does not work, and why customers try it: the CloudWatch PutMetricData API and Embedded Metric Format (EMF) log lines are the familiar CloudWatch ways to publish a custom metric — and they produce a CloudWatch metric in a CloudWatch namespace, which is a different surface. CloudWatch metrics are not queryable in Omni PromQL. The one bridge between the surfaces, OTel enrichment, projects AWS-vended service metrics (EC2, Lambda, RDS, …) onto the OTel surface and does not apply to a custom application metric, so EMF or PutMetricData on its own never makes a custom metric visible in Omni. (The IAM action the OTLP metrics endpoint authorizes against is also named cloudwatch:PutMetricData; that is the permission, not the API — a metric sent through the OTLP endpoint stays OTel-native.)

Constraints:

• You MUST NOT present PutMetricData or EMF as a way to get a custom metric into Omni. If the customer already publishes that way, the answer is to emit the metric through the OTel SDK (or a collector) as well — not to look for a setting that makes the CloudWatch metric appear.
• You MUST NOT invent a metric namespace, a CloudWatch-side setting, or an exporter option for this. OTLP metrics have no CloudWatch namespace; the SDK’s standard OTEL_EXPORTER_OTLP_* variables and the resource attributes from Step 3 are the whole configuration.
• For latency, use a Histogram, so percentiles can be computed at query time; a gauge of the last value cannot be turned back into a distribution.
• Metric attributes become queryable labels stored in the Dataset. Keep customer identifiers, tokens, request payloads, and other sensitive or personal data out of metric attributes and resource attributes — use coarse dimensions such as route, method, or status — and keep cardinality bounded, since every distinct attribute value is a separate series. CloudWatch encrypts metric data at rest; the Space’s own encryption (service-owned or a customer managed KMS key) is chosen at create-space (../spaces-and-domains.md).

Step 4: Review

Summarize the changes grouped by file, state the platform and language, and list the environment variables that will reach the application at runtime. Note explicitly that no exporter endpoint was configured and that telemetry will go to the SDK default until a receiver is set up — and point the user to the two paths in Scope. If a collector was deployed, state the endpoint the app now exports to and where the collector forwards it. Present the changes for review; do not deploy.

Component versions

Use the latest published ADOT auto-instrumentation version for the language unless the user pins one. Look the current version up from the source of truth rather than hardcoding from this guide:

Constraints

• Only IaC, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests — never application source code.
• Do not deploy. Stage the changes and let the user review and apply them.
• Do not configure an OTLP endpoint, collector, or agent unless the user asked for a destination. If they did, use collector.md, where the CloudWatch Agent is the primary collector.
• Do not add Application Signals, ServiceEvents, or Dynamic Instrumentation configuration.