Setting the file. One moment.
Subchapter 63.7
references/troubleshooting.mdMarkdown7 KBView on GitHub
Expanded troubleshooting reference for issues beyond the SKILL.md summary. Grouped by failure class.
Scripts
Wa ReviewAccessDenied / UnauthorizedOperation. Caller lacks permissions for DocumentDB, DMS, CloudWatch, EC2, or Secrets Manager. Attach AmazonDocDBReadOnlyAccess + CloudWatchReadOnlyAccess for read-only flows. For migration add AmazonDMSVPCManagementRole and scoped write actions. Do NOT grant admin as a workaround.
ExpiredToken / RequestExpired. Refresh your credentials (e.g. aws sso login, or renew the credentials for your configured profile) and verify with aws sts get-caller-identity, then retry.
Connection refused / timeout on port 27017. DocumentDB is VPC-only (no public endpoint). Run aws ec2 describe-security-groups --group-ids <sg> — the DocumentDB SG needs inbound TCP 27017 from the client SG. Reference by SG id, not CIDR. From outside the VPC use CloudShell VPC environment, EC2 in the VPC, or SSH tunnel via bastion.
TLS handshake failed / certificate verify failed. Download the RDS global bundle: curl -s https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -o global-bundle.pem. Verify the tlsCAFile path matches where the cert actually is. When tunneling, pass --tlsAllowInvalidHostnames to mongosh because the hostname resolves to 127.0.0.1, not the cluster endpoint.
Server selection timed out after 30000ms. TLS cert at wrong path, or endpoint unreachable. Verify cert path and run nc -zv <endpoint> 27017 from the client network.
getaddrinfo failed. Wrong endpoint. Run aws docdb describe-db-clusters --db-cluster-identifier <name> --query 'DBClusters[*].[Endpoint,Port]' to get the correct endpoint.
“not master” / “not primary”. Missing replicaSet=rs0 in the connection string. DocumentDB always uses rs0.
Intermittent write errors under load. Missing retryWrites=false. DocumentDB does not support retryable writes — drivers default to true.
Java driver only connects to primary. Using applyToClusterSettings with a single host sets mode to SINGLE and breaks failover. Use applyConnectionString instead — see references/connection-drivers.md.
Connection storm / DatabaseConnections spike. Creating MongoClient per request skips connection pooling. Create the client once at module scope (Lambda: outside the handler) and reuse across requests.
Cursor timeout / CursorNotFound. Idle cursors close after 10 minutes. Process results faster, add early $match/$limit to reduce set size, or use noCursorTimeout sparingly. cursor.maxTimeMS resets on each getMore — DocumentDB differs from MongoDB here.
DMS task refuses to start — “Test connection should be successful”. DMS requires explicit aws dms test-connection calls that pass for both source and target endpoints before starting a task. Call test-connection for each endpoint, then poll describe-connections until both return successful:
aws dms describe-connections \
--filters Name=endpoint-arn,Values=<src-arn>,<tgt-arn> \
--region <region>DMS target endpoint socket timeout. Target endpoint MUST use --ssl-mode verify-full with --certificate-arn pointing to the RDS global bundle imported into DMS. Without TLS, DocumentDB rejects the connection.
Import the RDS bundle into DMS if missing:
curl -s https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -o /tmp/global-bundle.pem
aws dms import-certificate \
--certificate-identifier global-bundle \
--certificate-pem file:///tmp/global-bundle.pem \
--region <region>Also verify the DocumentDB SG allows inbound TCP 27017 from the DMS replication instance’s SG (self-referencing rule if in same SG, or cross-SG rule).
DMS CDC task fails — “No tables found at task initialization”. Source has no collections yet. Set FailOnNoTablesCaptured: false in the replication task settings.
“AllowMajorVersionUpgrade flag must be present”. --allow-major-version-upgrade is mandatory on every modify-db-cluster MVU command. Include it.
“Must explicitly specify a new DB cluster parameter group”. The cluster uses a custom parameter group, but you didn’t specify one for the target engine family. Create a new PG for the target (e.g. docdb5.0, docdb8.0) and pass --db-cluster-parameter-group-name.
MVU fails / rolls back. In-place MVU auto-rolls back on failure. Check cluster events for “Database cluster is in a state that cannot be upgraded.” Verify: no db.r4 instances (not supported on 4.0+), no pending OS maintenance, burstable-instance index counts within limits (t4g.medium: 3,000; t3.medium: 10,000). Contact AWS support before re-attempting.
Post-upgrade performance degradation (5.0->8.0). Index metadata refresh is running — wait for “Index metadata refresh process completed” event (up to 2 hours). Do NOT reboot or failover the writer during this window.
MVU CDC migrator connection error. Source URI must NOT include readPreference=secondaryPreferred — change streams only work on the primary.
Clone creation fails. Source cluster must be in “available” state. Check encryption settings are compatible. Ensure IAM permissions include rds:RestoreDBClusterToPointInTime.
CDC replication lag not decreasing. DMS: check CloudWatch logs and increase parallel threads. MVU tool: verify network connectivity and that change stream retention hasn’t expired (default 3 hours — raise to 24 hours with change_stream_log_retention_duration=86400).
High CPU with no obvious slow queries. Run db.adminCommand({currentOp: 1}) — look for index builds. Check CloudWatch OpcountersInsert + OpcountersDelete for TTL activity. If query volume: scale up instance or add read replicas.
GC pressure / AvailableMVCCIds dropping. Long-running ops blocking MVCC garbage collection. Kill them: db.adminCommand({killOp: 1, op: <opid>}). Recommend avoiding transactions longer than 1 minute.
explain() output differs from MongoDB. Field names and nesting are different. Read output manually; do not parse it programmatically assuming MongoDB format.
ReadTimeoutError, ThrottlingException, Rate exceeded. Retry once; if persistent, narrow scope (single cluster, shorter window, smaller batch).
DBClusterNotFoundFault. Verify region and cluster identifier spelling. For Global Clusters use describe-global-clusters. Empty DMS endpoint list — confirm resources exist in the region you queried.