Subchapter 20.3
references/observability.mdMarkdown6 KBView on GitHub
Set up logging, tracing, and monitoring for your AgentCore agent.
AgentCore automatically enables:
These are on by default whether you’re running deployed (agentcore deploy + invoke) or locally (agentcore dev). The dev server auto-instruments your agent with the AWS OpenTelemetry distro the same way the deployed runtime does; opt out with agentcore dev --no-traces.
Two prerequisites for the local path to work end-to-end:
agentcore traces list and agentcore run eval --session-id return empty.After deploy, AgentCore Runtime also auto-instruments the container (the default CMD wraps the app with opentelemetry-instrument). You don’t need to configure OTEL in your code for either path — but you do need your agent code to be instrumented correctly.
Three things must be true for logs to appear:
Your Dockerfile CMD must use the OpenTelemetry wrapper:
CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]For CodeZip builds, this is handled automatically. For Container builds, you must add it.
Your runtime execution role needs:
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"xray:PutTraceSegments",
"xray:PutTelemetryRecords"
],
"Resource": "*"
}AgentCore captures structured logs via the Python logging module. print() statements go to stdout but are not captured by the OTEL pipeline.
import logging
logger = logging.getLogger(__name__)
# Good — captured by CloudWatch
logger.info("Processing request", extra={"session_id": session_id})
# Bad — not captured
print(f"Processing request {session_id}")Traces show the full execution path of one agent invocation — model calls, tool calls, and timing.
# List recent traces
agentcore traces list --runtime <AgentName> --since 1h --limit 10
# Get a specific trace
agentcore traces get <traceId> --runtime <AgentName>Trace delay: Traces appear ~10 seconds after invocation (previously 30–60s). Don’t panic if they’re not immediate, and don’t bake longer waits into scripts — older skills and docs that say “30–60 seconds” or “2–5 minutes” are stale.
Also verify Transaction Search is enabled in CloudWatch — this is a prerequisite for trace visibility in the console.
# Stream recent logs
agentcore logs --runtime <AgentName> --since 30m
# Filter by level
agentcore logs --runtime <AgentName> --level error --since 1h
# Search for specific text
agentcore logs --runtime <AgentName> --query "timeout" --since 2hFor production agents, set up a CloudWatch dashboard with:
These metrics are available in the AWS/BedrockAgentCore namespace after deploy.
If your agents are spread across accounts (typical setup: separate prod / staging / dev accounts), use CloudWatch cross-account observability to view metrics, traces, and logs from one central monitoring account.
The setup order matters — do it in this sequence or the console won’t show source-account data:
Order-of-operations trap: if you deploy agents in source accounts before linking, the telemetry still flows correctly — it just won’t be visible from the monitoring account until the link is active. You don’t need to redeploy, just wait a few minutes after linking.
Traces: cross-account trace viewing uses X-Ray’s existing cross-account sharing model. If the CloudWatch cross-account link is set up correctly for Logs and Metrics but traces don’t show, check X-Ray’s cross-account config separately.
IAM: no extra IAM on the agent execution roles for cross-account observability. The cross-account feature operates at the CloudWatch/X-Ray layer, not at the source of the telemetry.
See cross-account observability (opens in a new tab) for the current console flow and edge cases.
agents-debugagents-hardenreferences/evals.md