Skill 46 · Setting Up CloudWatch Observability
Subchapter 46.19
references/cloudwatch-omni/instrumentation/ecs-python.mdMarkdown9 KBView on GitHub
Wire the ADOT Python auto-instrumentation SDK into an ECS task definition (Fargate or EC2 launch type). An init container copies the SDK into a shared volume at task startup, so the application image is not rebuilt and the application source is not touched.
Read instrumentation.md first — it defines what is out of scope (OTLP endpoints, Application Signals). Deploying a collector is a separate, optional step — collector-ecs.md.
Do NOT:
OTEL_EXPORTER_OTLP_* — leave the SDK’s default endpoint in place — unless you are also deploying a collector (collector-ecs.md), which sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL deliberatelyOTEL_AWS_APPLICATION_SIGNALS_*, OTEL_AWS_SERVICE_EVENTS_*, or OTEL_AWS_DYNAMIC_INSTRUMENTATION_*CloudWatchAgentServerPolicy or AWSXRayDaemonWriteAccess to the task role — unless you are also deploying a collector (collector-ecs.md), which requires CloudWatchAgentServerPolicy on the task rolecdk deploy, terraform apply, or aws ecs update-service automatically.py filesOn IAM: instrumentation on its own needs no permissions, so this change adds none. The task may later need permission to reach wherever telemetry is sent — that belongs with the destination, not this guide. If you deploy a collector instead (collector-ecs.md), the collector is what gets the IAM — the workload still gets none.
const taskDefinition = new ecs.FargateTaskDefinition(this, '<Service>TaskDefinition', {
// ... existing configuration unchanged ...
volumes: [
{ name: 'opentelemetry-auto-instrumentation-python' },
],
});A bind mount (a volume with only a name) is all that is needed — no EFS, no host path.
const initContainer = taskDefinition.addContainer('init', {
// Look up the latest tag — see instrumentation.md
image: ecs.ContainerImage.fromRegistry('public.ecr.aws/aws-observability/adot-autoinstrumentation-python:v0.19.0'),
essential: false,
memoryReservationMiB: 64,
cpu: 32,
command: ['cp', '-a', '/autoinstrumentation/.', '/otel-auto-instrumentation-python'],
logging: ecs.LogDrivers.awsLogs({
streamPrefix: 'init-<service>',
logGroup: serviceLogGroup,
}),
});
initContainer.addMountPoints({
sourceVolume: 'opentelemetry-auto-instrumentation-python',
containerPath: '/otel-auto-instrumentation-python',
readOnly: false,
});essential: false matters — the init container exits after the copy, and an essential container exiting would stop the whole task.
const mainContainer = taskDefinition.addContainer('<service>-container', {
// ... existing image, ports, health check unchanged ...
environment: {
// ... existing environment variables preserved ...
PYTHONPATH: '/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python',
OTEL_SERVICE_NAME: '<service>',
},
});
mainContainer.addMountPoints({
sourceVolume: 'opentelemetry-auto-instrumentation-python',
containerPath: '/otel-auto-instrumentation-python',
readOnly: false,
});If the container already sets PYTHONPATH, keep its existing value in the middle so the instrumentation paths are prepended rather than overwriting it:
/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:<EXISTING_PYTHONPATH>:/otel-auto-instrumentation-pythonIf the container does not set PYTHONPATH today, use the two-segment form shown above. Check the existing container environment before writing this value — silently dropping an existing PYTHONPATH will break the application’s imports.
Set OTEL_SERVICE_NAME from the existing ECS service or container name. Optionally add OTEL_RESOURCE_ATTRIBUTES for attributes such as deployment.environment=production.
mainContainer.addContainerDependencies({
container: initContainer,
condition: ecs.ContainerDependencyCondition.SUCCESS,
});Without this, the application can start before the SDK finishes copying and PYTHONPATH will point at directories that do not exist yet.
If the task definition is managed as JSON (or through Terraform’s container_definitions), the same four pieces are:
Terraform note: in aws_ecs_task_definition, container_definitions is jsonencode of only
the containers array — volumes is a sibling HCL block, not a key inside that JSON. Pasting the
whole object below into jsonencode(...) silently drops the volume, after which every mount path is
missing and the app starts uninstrumented:
resource "aws_ecs_task_definition" "app" {
# ...
volume { name = "opentelemetry-auto-instrumentation-<lang>" }
container_definitions = jsonencode([ /* the containers array only */ ])
}{
"volumes": [{ "name": "opentelemetry-auto-instrumentation-python" }],
"containerDefinitions": [
{
"name": "init",
"image": "public.ecr.aws/aws-observability/adot-autoinstrumentation-python:v0.19.0",
"essential": false,
"command": ["cp", "-a", "/autoinstrumentation/.", "/otel-auto-instrumentation-python"],
"mountPoints": [
{ "sourceVolume": "opentelemetry-auto-instrumentation-python", "containerPath": "/otel-auto-instrumentation-python", "readOnly": false }
]
},
{
"name": "<service>-container",
"environment": [
{ "name": "PYTHONPATH", "value": "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python" },
{ "name": "OTEL_SERVICE_NAME", "value": "<service>" }
],
"mountPoints": [
{ "sourceVolume": "opentelemetry-auto-instrumentation-python", "containerPath": "/otel-auto-instrumentation-python", "readOnly": false }
],
"dependsOn": [{ "containerName": "init", "condition": "SUCCESS" }]
}
]
}Gunicorn with its default sync workers and no --preload needs no special handling — instrumentation loads and produces both server spans and nested client spans normally.
The configurations that more often need attention are --preload and async worker classes (gevent, eventlet). If the workload uses one of those and no telemetry appears, the usual remedy is a Gunicorn post_fork hook that re-initializes the SDK — flag that to the user rather than changing the server configuration silently.
After the user deploys and tasks recycle:
# The init container should have exited 0
aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> \
--query 'tasks[0].containers[?name==`init`].[name,lastStatus,exitCode]'Then check the application container’s CloudWatch Logs. ADOT Python prints no line containing “opentelemetry” — the real startup signal is Configuration of aws_configurator not loaded, configurator already loaded. Until a receiver exists at the default OTLP endpoint, exporter connection errors are expected — that is the next step, not a failure of instrumentation.
Tell the user:
“I’ve wired ADOT Python auto-instrumentation into your ECS task definition.
Changes:
opentelemetry-auto-instrumentation-pythoninit container that copies the ADOT Python SDK into that volumePYTHONPATH (prepended to any existing value) and OTEL_SERVICE_NAME to the application container, plus the volume mount and a SUCCESS dependency on initNot changed: your application image, your application source, the task role’s IAM policies, and the service’s sidecars — no CloudWatch Agent or collector was added.
Next steps:
PYTHONPATH value preserves any path the container already relied on.localhost:4318). Two ways to fix that: deploy an OTel Collector alongside it and export to that (collector-ecs.md), or point OTEL_EXPORTER_OTLP_ENDPOINT at an OTLP endpoint you already have.Let me know if you’d like adjustments before you deploy.”