Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.21
references/cloudwatch-omni/instrumentation/eks-java.mdMarkdown8 KBView on GitHub
Wire the ADOT Java auto-instrumentation agent into a Deployment on Amazon EKS. An init container copies the agent jar into the pod at startup, so the application image is not rebuilt and the application source is not touched.
Read instrumentation.md first — it defines what is out of scope (OTLP endpoints, Application Signals). Deploying a collector is a separate, optional step — collector-eks.md.
Deploymentkubectl context pointing at the cluster, for verificationDo NOT:
amazon-cloudwatch-observability add-on, or any OpenTelemetry operator — unless you are also deploying a collector (collector-eks.md). Deploying the CloudWatch Agent by Helm necessarily installs its operator, because the agent ships as a custom resource that only the operator reconciles; that is expected there, and is not the add-oninstrumentation.opentelemetry.io/inject-java annotation (it requires an operator)OTEL_EXPORTER_OTLP_* — leave the SDK’s default endpoint in place — unless you are also deploying a collector (collector-eks.md), which sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL deliberatelyCloudWatchAgentServerPolicy or AWSXRayDaemonWriteAccess to the workload, or add Application Signals / ServiceEvents / Dynamic Instrumentation configuration. The IAM half is lifted when you are also deploying a collector (collector-eks.md), which grants CloudWatchAgentServerPolicy to the collector’s own service account — never to the workload. The Application Signals half always standskubectl apply, terraform apply, cdk deploy, helm install/helm upgrade, kubectl annotate, or kubectl delete automatically — including the out-of-band IRSA commands in collector-eks.md. Present every mutating command for the user to run; they act on a live cluster.java files or its build configurationOn IAM: instrumentation on its own needs no permissions, so this change adds none. The workload may later need permission to reach wherever telemetry is sent — that belongs with the destination, not this guide. If you deploy a collector instead (collector-eks.md), the collector is what gets the IAM — the workload still gets none.
Add a shared volume, an init container that copies the agent jar into it, and the environment variables plus volume mount on the application container.
Note that the Java init container copies a single jar, not a directory tree — the copy command differs from the other languages.
# k8s-deployment.yaml
spec:
template:
spec:
volumes:
- name: otel-auto-instrumentation
emptyDir: {}
initContainers:
- name: otel-auto-instrumentation
# Look up the latest tag — see instrumentation.md
image: public.ecr.aws/aws-observability/adot-autoinstrumentation-java:v2.30.0
command: ['cp', '-a', '/javaagent.jar', '/otel-auto-instrumentation-java/javaagent.jar']
volumeMounts:
- name: otel-auto-instrumentation
mountPath: /otel-auto-instrumentation-java
containers:
- name: <my-app>
# ... existing image, ports, probes unchanged ...
env:
# Note the leading space — JAVA_TOOL_OPTIONS is appended to, not replaced
- name: JAVA_TOOL_OPTIONS
value: ' -javaagent:/otel-auto-instrumentation-java/javaagent.jar'
- name: OTEL_SERVICE_NAME
value: <my-app>
volumeMounts:
- name: otel-auto-instrumentation
mountPath: /otel-auto-instrumentation-javaIf the container already sets JAVA_TOOL_OPTIONS, append the -javaagent flag to the existing value rather than overwriting it — losing existing JVM flags (heap settings, GC options, other agents) will change how the application runs:
<EXISTING_JAVA_TOOL_OPTIONS> -javaagent:/otel-auto-instrumentation-java/javaagent.jarCheck the existing container env before writing this value.
Set OTEL_SERVICE_NAME from the existing Deployment name or application name. Optionally add OTEL_RESOURCE_ATTRIBUTES for attributes such as deployment.environment=production.
Preserve the container’s existing env, volumeMounts, probes, and resources — append to them rather than replacing.
When the Deployment is applied through eks.KubernetesManifest / cluster.addManifest, add the same three pieces to the manifest object:
// eks-stack.ts
volumes: [{ name: 'otel-auto-instrumentation', emptyDir: {} }],
initContainers: [{
name: 'otel-auto-instrumentation',
image: 'public.ecr.aws/aws-observability/adot-autoinstrumentation-java:v2.30.0',
command: ['cp', '-a', '/javaagent.jar', '/otel-auto-instrumentation-java/javaagent.jar'],
volumeMounts: [{ name: 'otel-auto-instrumentation', mountPath: '/otel-auto-instrumentation-java' }],
}],
containers: [{
name: 'my-app',
image: imageUri,
env: [
{ name: 'JAVA_TOOL_OPTIONS', value: ' -javaagent:/otel-auto-instrumentation-java/javaagent.jar' },
{ name: 'OTEL_SERVICE_NAME', value: 'my-app' },
],
volumeMounts: [{ name: 'otel-auto-instrumentation', mountPath: '/otel-auto-instrumentation-java' }],
}],Terraform users applying a raw manifest file (kubectl apply -f, envsubst, or kubernetes_manifest) edit the YAML directly — no Terraform resource changes are needed for instrumentation.
After the user applies the change and pods restart:
# The init container should be listed on the pod
kubectl get pod -l app=<my-app> -o jsonpath='{.items[0].spec.initContainers[*].name}'
# The JVM logs the agent being picked up via JAVA_TOOL_OPTIONS on startup
kubectl logs -l app=<my-app> -c <my-app> | grep -iE "javaagent|opentelemetry"The JVM prints a Picked up JAVA_TOOL_OPTIONS: line on startup listing the -javaagent flag — that confirms the agent attached. Until a receiver exists at the default OTLP endpoint, exporter connection errors are expected.
Tell the user:
“I’ve wired the ADOT Java auto-instrumentation agent into your EKS Deployment.
Changes:
emptyDir volume and an otel-auto-instrumentation init container that copies javaagent.jar into the podJAVA_TOOL_OPTIONS (appended to any existing value) and OTEL_SERVICE_NAME to the application container, plus the shared volume mountNot changed: your application image, your application source and build config, and cluster-level components — no add-on or operator was installed, and no IAM changes were needed.
Next steps:
JAVA_TOOL_OPTIONS preserves any JVM flags the container already set.Picked up JAVA_TOOL_OPTIONS: line in the logs.localhost:4317, gRPC — the Java agent’s default, not 4318). Two ways to fix that: deploy an OTel Collector alongside it and export to that (collector-eks.md), or point OTEL_EXPORTER_OTLP_ENDPOINT at an OTLP endpoint you already have.Let me know if you’d like adjustments before you apply.”