Troubleshooting

/ready hook caused build failure

The /ready and /validate hooks are asynchronous — return 503 until the application is ready/validated, and the platform will retry. If the application blocks the request instead of returning 503 promptly, a single hung call can consume the whole hook timeout window before the platform gets a chance to retry. If /ready never returns 200 (or the timeout elapses), the build fails. Look at the application’s CloudWatch logs (/aws/lambda-microvms/<image-name>) for stack traces from your hook server.

/validate hook caused build failure

The validation phase launches a test MicroVM from the snapshot and calls /validate. This is an application-level check — use it to verify your app behaves correctly after resume (e.g., that connections are re-established, state is valid).

Run failures (RunMicrovm)

Connect / auth failures

401/403 from the proxy

• Token expired (max 60 min). Mint a new one with create-microvm-auth-token.
• Token was issued to a different MicroVM ID.
• Token has allowedPorts and you’re requesting a port outside the allowed ranges.

502 from the proxy

A 502 can be returned during the first few seconds after the MicroVM is run while the snapshot is being restored.

• App not listening on the routed port (default 8080; use X-aws-proxy-port to redirect).
• TLS mismatch — the app is speaking TLS without ALPN advertising HTTP/1.1, or speaking plaintext on a port the proxy thinks is TLS.
• App crashed after run. Check CloudWatch logs.

429 from the proxy

Per-MicroVM or account-level rate limit. Back off and retry. Check the docs for the run-rate quota.

5xx while a run was in flight

If the proxy receives traffic before RunMicrovm finishes, it may 5xx. Retry connections with backoff rather than polling get-microvm state (which is eventually consistent), or rely on autoResumeEnabled semantics if you’re hitting a recently-suspended MicroVM.

Auto-resume not working

• Confirm idlePolicy.autoResumeEnabled: true was set at run time.
• Confirm the MicroVM’s state is SUSPENDED (not TERMINATED — auto-resume doesn’t revive terminated VMs).
• The proxy invokes resume then retries connecting a few times with a delay between attempts. If your /resume hook is slow or the app doesn’t respond in time, the proxy may return an error to the caller.

Hook timeouts at runtime

Hook server must bind to 0.0.0.0 on the configured port (commonly 9000). 127.0.0.1-only listeners are unreachable from Lambda’s hook caller.

Network connector troubleshooting

Snapshot / uniqueness symptoms

• All MicroVMs serve the same instance ID / UUID. → Generation happened at build time. Move to /run or first-request init. See snapshots-and-uniqueness.md.
• TLS handshakes intermittently fail across multiple VMs. → Predictable PRNG seeded once at build. Switch to a CSPRNG.
• “AWS SDK calls fail with credentials errors after a few hours.” → SDK client picked up creds at boot and didn’t refresh. Most modern SDKs refresh; verify version.

Image storage costs growing

Old image versions persist (and bill) even when unused.

Or mark old versions INACTIVE first if you want to keep them around for rollback. INACTIVE versions are still billed.

Where logs live

Avoid logging secrets, tokens, or PII to stdout/stderr — application output is forwarded to CloudWatch.

Custom CloudWatch metrics are not auto-emitted — publish from inside your application (or run the CloudWatch Agent inside the container). Set up CloudWatch Alarms on key metrics to catch operational issues early. Consider enabling CloudTrail data events for Lambda MicroVM API calls in production environments.

When all else fails

1. Run with the SHELL_INGRESS network connector attached, get a shell auth token, connect via the console “Connect” button.
2. The shell drops into the same container as the running app — same network namespace, same filesystem. You can inspect files, processes, and network state directly.