Ecs Troubleshooting Guide

• Verify Dependencies
• Exit Code Reference
• OOM Kills Deep Dive
• Task Placement Failures
• Health Check Debugging Checklist
• Image Pull Errors
• Private Subnet Networking
• ENI Trunking for EC2 awsvpc Density
• Security Considerations

Verify Dependencies

Operators MUST confirm the following before proceeding:

Exit Code Reference

Key Diagnostic Rules

• Exit code 137 means the container received SIGKILL. Check stoppedReason from describe-tasks first: if it contains “OutOfMemoryError”, investigate OOM — see OOM Kills Deep Dive. If the task was being stopped (deployment, scale-in) and stoppedReason does NOT mention OOM, the container likely did not handle SIGTERM within stopTimeout — add a SIGTERM handler and verify stopTimeout is sufficient.
• Exit code 143 is expected during normal operations (deployments, scale-in). It means the container handled SIGTERM gracefully. It is NOT an error.
• Exit code 1 requires application log analysis — check CloudWatch Logs for the container’s last output.

OOM Kills Deep Dive

Exit code 137 commonly indicates the container exceeded its memory limit and was killed by the kernel (OOM killer) or the Docker daemon. It can also occur when a container does not exit within stopTimeout after receiving SIGTERM.

Container Memory Hard Limit vs Task-Level Memory

On Fargate, the task-level memory is the overall ceiling. If a container definition sets a memory hard limit, Fargate enforces it — the container is killed if it exceeds that limit. If no per-container memory is set, a single container MAY consume all task memory, starving sidecars.

Diagnosing OOM Kills

Look for:

• stoppedReason containing “OutOfMemoryError” or “oom”
• Container reason containing “OutOfMemoryError: Container killed due to memory usage”
• exitCode: 137 on the affected container

JVM Fix: Use MaxRAMPercentage Instead of Fixed Xmx

• In containerized environments, -XX:MaxRAMPercentage is preferred over fixed -Xmx because the heap scales automatically when the container memory limit changes. Fixed -Xmx values also work but require manual adjustment and must account for non-heap memory.
• A starting value of 75.0 leaves ~25% for JVM non-heap memory (metaspace, thread stacks, direct buffers, GC overhead). Workloads with many threads or large direct buffers may need a lower percentage (e.g., 50–70%); simple applications may safely use 80% or higher.
• On Fargate (Platform 1.4+), HotSpot-based JVMs (OpenJDK, Corretto, Temurin) correctly detect the task memory limit via cgroup. OpenJ9 has a known bug where it may not detect the limit correctly (openj9#11998 (opens in a new tab)) — set container-level memory as a workaround if using OpenJ9.

Quick Memory Check

Task Placement Failures

When ECS cannot place a task, the service event log shows the reason. Common failures:

Diagnosing Placement Failures

Health Check Debugging Checklist

When tasks are being killed by ALB health checks, follow these steps in order:

Step 1: Verify the Health Check Endpoint Responds Locally

Confirm the application responds on the health check path and port. Use ECS Exec if available:

Step 2: Check healthCheckGracePeriod

If tasks are killed before the application finishes starting, healthCheckGracePeriod is too low or not set.

This value MUST be greater than the application startup time. Operators SHOULD set it to at least 60 seconds.

Step 3: Verify Target Group Health Check Settings

Check that:

• Health check path matches the application’s actual health endpoint.
• Health check port matches the container port (or is set to traffic-port).
• Healthy threshold, interval, and timeout are reasonable.

Step 4: Check Security Group Rules

The ALB security group MUST be allowed to reach the container port on the task security group.

Step 5: Check Container Logs for Startup Errors

Step 6: Verify the Container Is Listening on the Correct Interface

The application MUST listen on 0.0.0.0 (all interfaces), not 127.0.0.1 (localhost only). In awsvpc mode, the ALB health check comes from the ALB’s IP, not localhost.

Image Pull Errors

Diagnosing Image Pull Failures

Private Subnet Networking

When ECS tasks run in private subnets (no internet gateway route), the following VPC endpoints are required:

Required Endpoints (Minimum for ECS Fargate)

Additional Endpoints by Feature

Verifying Endpoint Connectivity

Operators MUST verify:

1. Endpoints are in available state.
2. Interface endpoints have security groups that allow inbound HTTPS (port 443) from the task security group.
3. Interface endpoints are associated with the same subnets as the ECS tasks.
4. The S3 gateway endpoint route table is associated with the task subnets.

ENI Trunking for EC2 awsvpc Density

By default, each ECS task using awsvpc network mode on EC2 consumes one ENI on the host instance. This limits the number of tasks per instance to the instance’s ENI limit minus one (reserved for the host).

ENI trunking allows multiple tasks to share a trunk ENI, significantly increasing task density.

Enabling ENI Trunking

Requirements

• Instance MUST be launched after the setting is enabled. Existing instances are NOT affected.
• Instance type MUST support ENI trunking (most c5, m5, r5 and newer generation types).
• The ECS agent on the instance MUST be version 1.28.1 or later, with ecs-init version 1.28.1-2 or later.

Verifying ENI Trunking

Task Density Comparison (Example: c5.large)

Exact limits vary by instance type — see Supported instance types for ENI trunking (opens in a new tab).

Operators SHOULD enable ENI trunking for any EC2 cluster using awsvpc network mode to avoid RESOURCE:ENI placement failures.

Security Considerations

• The troubleshooting commands in this guide require read-only permissions (ecs:Describe*, ecs:List*, logs:GetLogEvents, ec2:DescribeSecurityGroups, ec2:DescribeVpcEndpoints, elbv2:DescribeTargetHealth). Do not grant broader permissions for debugging.
• ECS Exec (execute-command) provides shell access to running containers. Restrict ssmmessages:* permissions to authorized operators only and audit usage via CloudTrail.
• VPC endpoint security groups MUST restrict inbound HTTPS (port 443) to the task security group — do not use 0.0.0.0/0.
• When reviewing container logs for errors, be aware that application logs may contain sensitive data. Use CloudWatch Logs encryption with a KMS key for log groups containing sensitive output.
• The 0.0.0.0 listen address in Health Check Step 6 refers to the container’s network interface binding, not a security group rule. In awsvpc mode, each task has its own ENI and the ALB health check arrives from the ALB’s IP, requiring the application to listen on all interfaces.