Setting the file. One moment.
Chapter 52 · Managing Amazon Msk
Subchapter 52.7
references/streaming-tables-troubleshooting.mdMarkdown5 KBView on GitHub
| State | Meaning |
|---|---|
CREATING| Provisioning in progress |
ACTIVE | Delivering data |
UPDATING | Config change in progress (still delivering) |
DELETING | Being removed |
FAILED | Creation failed — check DescribeChannel + CW Logs |
Provisioning is in progress. Run aws kafka describe-channel to monitor. If it transitions to FAILED, check the failure detail. If it stays in CREATING well beyond expected time, contact Support.
Configuration errors (missing permissions, nonexistent destination, unresolvable schema) cause
FAILED, not prolongedCREATING.
| Cause | Fix |
|---|---|
Trust policy missing kafka.amazonaws.com | Fix trust policy principal and conditions |
| Destination bucket doesn’t exist | Create the S3 Table bucket in the same Region as the cluster. The Iceberg table is auto-created by Streaming Tables (enableTableCreation: true) — you must not pre-create the Iceberg table; pre-creating it will break Streaming Tables delivery. Only the bucket must exist. |
| Schema can’t be resolved from GSR | Verify: aws glue get-schema --schema-id SchemaArn=<ARN> |
| Missing IAM permissions | Compare role policy against reference in streaming-tables.md or data-delivery-for-general-purpose-s3.md |
| Wrong catalog ARN | Must be arn:aws:glue:REGION:ACCOUNT_ID:catalog/s3tablescatalog/BUCKET |
Recovery: Delete failed channel, fix config, recreate.
Prerequisite: Ensure CloudWatch Logs delivery is enabled for the Streaming Tables. Without it, diagnostic logs (e.g.,
AccessDeniederrors) are not available. Enable logging when creating or updating the channel.
BytesIn metric — if zero, no data being producedAccessDeniedCauses: Large table metadata/snapshots, low throughput, transient issues.
Fix:
dataFreshnessSecondsSchema evolution is NOT supported. A change makes records incompatible.
Fix: Revert schema OR delete channel and create new one.
Records don’t conform to schema. Common causes:
Fix: Inspect DLQ messages + CloudWatch Logs for error details. Fix producer or schema.
Warning: DLQ messages contain full record payloads which may include sensitive data (PII, financial data, etc.). Ensure the DLQ bucket has encryption enabled (SSE-KMS recommended), access is restricted to authorized personnel, and consider enabling S3 access logging for audit purposes.
| Error | Cause | Fix |
|---|---|---|
| Missing required field | Record lacks field in schema "required" | Add to producer or remove from "required" (needs new channel) |
| Type mismatch | Value doesn’t match schema type | Fix producer serialization |
| Schema resolution failure | Embedded schema ID doesn’t match GSR (JSON_SCHEMA_GSR) | Verify producer GSR config |
kafka.amazonaws.com (do not use any other service principal)Namespace: AWS/Kafka | Dimensions: ClusterName, ChannelName, Topic
DeliveryToIceberg.DataFreshness, BytesIn, BytesProcessed, BytesOut, TotalRowCount, SuccessfulRowCount, FailedRowCount, CommitSuccess, DLQDeliverySuccess
DeliveryToS3.DataFreshness, BytesIn, BytesProcessed, BytesOut, RecordCount, SuccessfulRecordCount, FailedRecordCount, DeliverySuccess, DLQDeliverySuccess
| Alarm | Condition | Action |
|---|---|---|
| High data freshness | DataFreshness > configured interval | Investigate throughput |
| Failed records | FailedRowCount/FailedRecordCount > 0 for 5 min | Check DLQ + schema compatibility |
| DLQ deliveries | DLQDeliverySuccess > 0 | Inspect DLQ messages |
Symptom: Streaming Tables tab not visible or CreateChannel errors.
Cause: Cluster uses Standard brokers, or is MSK Serverless.
Fix: Streaming Tables requires MSK Provisioned with Express brokers only.