Tracing

• ADOT vs X-Ray SDK
• Trace structure
• Annotations vs metadata
• Sampling rules
• ADOT collector configuration
• Instrumentation patterns
• Migration constraints
• Common mistakes

ADOT vs X-Ray SDK

Migration path: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.

Trace structure

• Trace — collection of all segments from a single request, identified by trace ID
• Segment — JSON document with a 64 KB documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
• Subsegment — granular detail within a segment (downstream calls, custom code blocks)
• Inferred segment — generated by X-Ray from subsegments for uninstrumented downstream services

Trace ID format

Format: 1-{8 hex epoch}-{24 hex unique}. W3C trace IDs are supported (reformatted).

Retention

• Trace data: 30 days (not configurable)
• Service graph: 30 days

Annotations vs metadata

Rule of thumb: If you need to search for it → annotation. If you just need to store it → metadata.

WARNING: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.

Sampling rules

Default rule

• Reservoir: 1 request per second (shared across all instances)
• Rate: 5% of additional requests
• Conservative default to control costs

Rule evaluation

• Rules evaluated in ascending priority order (1–9999, lower = higher priority)
• Default rule priority = 10000 (always last)
• First matching rule wins

Rule parameters

Parent-based sampling (critical concept)

Sampling decision is made once by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.

Adaptive sampling (newer)

• SamplingRateBoost — auto-increases rate during anomalies
• MaxRate — ceiling for boosted rate
• CooldownWindowMinutes — prevents continuous boosts (recommended when SamplingRateBoost is configured)

ADOT collector configuration

Architecture

CloudWatch + X-Ray pipeline

The 0.0.0.0 receivers below listen on every interface with no TLS and no authentication. Bind them to 127.0.0.1 when the senders are on the same host; otherwise it is recommended to restrict the OTLP 4317 / 4318 ports via security groups or host firewall and to avoid co-locating untrusted workloads. This guide does not apply those controls, so assess and configure them for your environment.

EKS DaemonSet deployment

Cardinality prevention (three-layer defense)

1. OTel SDK level: Don’t emit high-cardinality attributes (ContainerID, CustomerID, RequestID)
2. ADOT Collector level: Use Filter Processor to drop metrics by name/attribute
3. Backend level: Use backend-specific dimension filtering (CloudWatch: dimension_rollup_option + metric_declarations; Prometheus: metric_relabel_configs)

Filter as early as possible in the pipeline to reduce cost and cardinality.

Instrumentation patterns

Lambda: enable active tracing (CDK)

API Gateway: enable tracing

Or via CLI: aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true

Trace-log correlation

Inject trace ID into application logs for cross-pillar correlation:

Migration constraints (X-Ray SDK → OTel)

Annotations require explicit opt-in

In OTel, all span attributes become X-Ray metadata by default. To make an attribute a searchable X-Ray annotation, add its key to the aws.xray.annotations list:

Without this, you lose all annotation-based filtering after migration.

Centralized sampling requires a proxy

The ADOT collector config must include the awsproxy extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):

SDK env vars: OTEL_TRACES_SAMPLER=xray and OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000

Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.

Mixed propagation during incremental migration

OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:

Without this, traces break at service boundaries between old and new instrumentation.

Port conflict: stop X-Ray daemon before starting ADOT

Both use port 2000. Running both simultaneously causes silent data loss.

Lambda ADOT layer adds cold start latency

ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don’t need OTel’s multi-backend capabilities, X-Ray SDK may still be preferable.

W3C trace ID version requirement

ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.

Common mistakes

1. Using X-Ray SDK for new projects — Maintenance mode. Use ADOT/OpenTelemetry.

2. Storing searchable data as metadata — Metadata is NOT indexed. Use annotations for data you need to filter by.

3. Exceeding 50 annotations per trace — Hard limit. Plan your annotation schema.

4. Not stripping X-Amzn-Trace-Id from untrusted requests — Users can inject trace IDs or sampling decisions.

5. Default sampling for all services — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.

6. StepFunctions tracing overrides Lambda — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda’s own config.

7. Cross-account tracing — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.