Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.13
references/cloudwatch-omni/instrumentation/ec2-java.mdMarkdown10 KBView on GitHub
Download the ADOT Java agent jar onto an EC2 instance and attach it with JAVA_TOOL_OPTIONS, by editing the instance’s UserData (or the systemd unit that starts the app).
Read instrumentation.md first — it defines what is out of scope (OTLP endpoints, Application Signals). Deploying a collector is a separate, optional step — collector-ec2.md.
Java is the least invasive of the four languages here: the agent is a single jar and the JVM attaches it from an environment variable, so nothing is installed into the application’s own dependencies and no image rebuild is needed on either the host or the Docker path.
Do NOT:
OTEL_TRACES_SAMPLER=xray, or any localhost:4316 / localhost:2000 endpoint — these stay forbidden in all casesOTEL_EXPORTER_OTLP_* — unless you are also deploying a collector (collector-ec2.md), which sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL deliberatelyOTEL_AWS_APPLICATION_SIGNALS_*, OTEL_AWS_SERVICE_EVENTS_*, or OTEL_AWS_DYNAMIC_INSTRUMENTATION_*CloudWatchAgentServerPolicy or AWSXRayDaemonWriteAccess to the instance role — unless you are also deploying a collector (collector-ec2.md), which requires CloudWatchAgentServerPolicy on the instance rolecdk deploy / terraform apply, or modify a running instance in place.java files or its build configurationOn IAM: instrumentation on its own needs no permissions, so this change adds none. The instance may later need permission to reach wherever telemetry is sent — that belongs with the destination, not this guide. If you deploy a collector instead (collector-ec2.md), the collector is what gets the IAM — the workload still gets none.
If you cannot determine a value from the IaC, ask the user. Do not guess.
docker run / docker start → Docker. java -jar, ./gradlew bootRun, a catalina.sh/Tomcat script → runs directly on the instance. If it is unclear, ask.<APP_DIR> and the exact startup command.<SERVICE_NAME> — the application or stack name; becomes OTEL_SERVICE_NAME.JAVA_TOOL_OPTIONS is already set anywhere — UserData, the systemd unit, the Dockerfile, or the docker run. If it is, the -javaagent flag must be appended to the existing value, not written over it.yum, Amazon Linux 2023 uses dnf, Ubuntu/Debian uses apt.Terraform heredoc warning: when adding lines to a user_data heredoc, match the exact leading whitespace of the existing lines. <<-EOF only strips indentation when it is consistent; inconsistent indentation leaves spaces before #!/bin/bash and cloud-init fails.
instance.userData.addCommands(
'# Download the ADOT Java agent (latest release)',
// -f: without it an HTTP error body is written into the .jar and curl still exits 0, so the
// failure surfaces later as a JVM that will not start on a corrupt agent jar.
// --retry: this download runs early in UserData; a transient failure under `set -e` aborts every
// later command. See the placement note in collector-ec2.md.
'curl -fsSLo /opt/aws-opentelemetry-agent.jar \\',
' --retry 5 --retry-delay 5 --retry-connrefused \\',
' https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar',
);Place this before the application starts. To pin a version instead of tracking latest, see the source-of-truth links in instrumentation.md.
The startup command itself does not change — the JVM picks the agent up from the environment:
instance.userData.addCommands(
'export JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar',
'export OTEL_SERVICE_NAME=<SERVICE_NAME>',
'',
'# Existing startup command remains unchanged',
'cd <APP_DIR>',
'java -jar <APP_JAR>',
);If JAVA_TOOL_OPTIONS is already set, append rather than replace — dropping existing JVM flags (heap settings, GC options, other agents) will change how the application runs:
export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -javaagent:/opt/aws-opentelemetry-agent.jar"An export in UserData does not reach a process started by a .service unit — ExecStart is a fresh process that does not inherit the UserData shell’s environment. Put the variables on the unit:
# /etc/systemd/system/<SERVICE_NAME>.service (add to the [Service] section)
[Service]
Environment=JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar
Environment=OTEL_SERVICE_NAME=<SERVICE_NAME>ExecStart does not need to change. Then add systemctl daemon-reload and systemctl restart <SERVICE_NAME> to UserData.
If the unit already has a JAVA_TOOL_OPTIONS= line, edit that line — and QUOTE the whole assignment. Do not add a second Environment= for the same variable; the later assignment overrides the earlier one. Quoting is load-bearing, not style: systemd treats space-separated tokens on one Environment= line as separate assignments, so an unquoted append silently drops the -javaagent flag.
# WRONG -- systemd logs `Invalid environment assignment, ignoring: -javaagent:...` and the
# process receives only `-Xmx256m`. The app starts normally, serves traffic, and emits nothing.
Environment=JAVA_TOOL_OPTIONS=-Xmx256m -javaagent:/opt/aws-opentelemetry-agent.jar
# RIGHT -- the process receives both flags
Environment="JAVA_TOOL_OPTIONS=-Xmx256m -javaagent:/opt/aws-opentelemetry-agent.jar"Note also that systemd does no shell expansion here: the export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -javaagent:..." idiom shown above for the shell case does not work in a unit file — it would pass the literal string $JAVA_TOOL_OPTIONS to the JVM. Write the existing flags out explicitly.
Confirm with systemctl show <SERVICE_NAME> --property=Environment, then check the JVM echoed the flag itself, not just the line: journalctl -u <SERVICE_NAME> | grep 'Picked up JAVA_TOOL_OPTIONS.*javaagent'. Grepping only for Picked up JAVA_TOOL_OPTIONS matches the broken unquoted case above too, where the JVM prints the line with the pre-existing flags and no -javaagent.
The agent jar has to be visible inside the container. Bind-mount it from the host — no image rebuild:
instance.userData.addCommands(
'# Download the agent jar on the host',
// -f: without it an HTTP error body is written into the .jar and curl still exits 0, so the
// failure surfaces later as a JVM that will not start on a corrupt agent jar.
// --retry: this download runs early in UserData; a transient failure under `set -e` aborts every
// later command. See the placement note in collector-ec2.md.
'curl -fsSLo /opt/aws-opentelemetry-agent.jar \\',
' --retry 5 --retry-delay 5 --retry-connrefused \\',
' https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar',
'',
'# Mount it into the container and attach it via JAVA_TOOL_OPTIONS',
`docker run -d --name <APP_NAME> \\`,
` -v /opt/aws-opentelemetry-agent.jar:/opt/aws-opentelemetry-agent.jar:ro \\`,
` -e JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar \\`,
` -e OTEL_SERVICE_NAME=<SERVICE_NAME> \\`,
` <IMAGE_URI>`,
);Keep every flag the existing docker run already had — ports, networks, volumes, restart policy. The application image and its Dockerfile are untouched.
If the user prefers the agent baked into the image, the alternative is a Dockerfile line — RUN curl -Lo /opt/aws-opentelemetry-agent.jar <release-url> — plus the same JAVA_TOOL_OPTIONS; mention that it requires rebuilding and republishing the image.
After the user deploys and the instance boots, look for the JVM’s Picked up JAVA_TOOL_OPTIONS: line listing the -javaagent flag — that confirms the agent attached.
journalctl -u <SERVICE_NAME> for a systemd service. /var/log/cloud-init-output.log shows whether the curl succeeded.docker logs <APP_NAME>.Until a receiver exists at the default OTLP endpoint, exporter connection errors are expected — that is the next step, not a failure of instrumentation.
Tell the user:
“I’ve wired the ADOT Java auto-instrumentation agent into your EC2 deployment.
Changes:
aws-opentelemetry-agent.jar to /optJAVA_TOOL_OPTIONS (appended to any existing value) and OTEL_SERVICE_NAME — your startup command is unchangedEnvironment= lines (an export in UserData would not reach it)Dockerfile are unchangedNot changed: your application source and build config, the instance role’s IAM policies, and the instance’s software — no CloudWatch Agent or collector was installed.
Next steps:
JAVA_TOOL_OPTIONS preserves any JVM flags already in use.Picked up JAVA_TOOL_OPTIONS: line.localhost:4317, gRPC — the Java agent’s default, not 4318). Two ways to fix that: deploy an OTel Collector alongside it and export to that (collector-ec2.md), or point OTEL_EXPORTER_OTLP_ENDPOINT at an OTLP endpoint you already have.Let me know if you’d like adjustments before you deploy.”