Msk Replicator

AWS::MSK::Replicator‘s KafkaClusters list takes exactly two KafkaClusterProperty entries, each typed as either AmazonMskCluster or ApacheKafkaCluster. Choose the type based on whether the replicator can resolve the cluster natively, not just on whether it happens to be running Kafka:

Use AmazonMskCluster and reference the cluster by ARN whenever the replicator and that cluster share an account — this gets you IAM auth and skips the cluster-ID lookup entirely. Use ApacheKafkaCluster whenever the replicator cannot resolve the cluster’s ARN directly, which includes any cross-account Amazon MSK cluster. When the remote side is ApacheKafkaCluster because it’s a cross-account Amazon MSK cluster, apply the MSK-specific auth and secret rules below. When it’s genuinely third-party Kafka, configure auth to match however that broker is actually set up instead.

Set vpcConfig only on the AmazonMskCluster entry

Configure the replicator’s ENI placement (subnetIds, securityGroupIds) exclusively on the AmazonMskCluster entry’s vpcConfig. vpcConfig is not a supported field on ApacheKafkaCluster and the API rejects it:

Make sure the VPC behind the AmazonMskCluster entry already has a network path to the remote broker (VPC peering, PrivateLink, Transit Gateway, or Direct Connect) — the replicator uses existing routing and does not establish new connectivity for you.

Add a self-referencing ingress rule for the Kafka ports (9092/9094/9096/9098) on the security group used in vpcConfig, so the replicator’s own ENIs can reach the local cluster’s brokers over that same security group.

Use SASL/SCRAM mechanism SHA512 when the remote cluster is Amazon MSK

When the remote side is a cross-account Amazon MSK cluster referenced as ApacheKafkaCluster, set clientAuthentication.saslScram.mechanism to 'SHA512'. Amazon MSK only enables SCRAM-SHA-512 (opens in a new tab) for SASL/SCRAM, never SHA-256, even though AWS’s published CreateReplicator examples for self-managed sources show "mechanism": "SHA256" (that value applies to third-party brokers actually configured for SHA-256). Using SHA256 against an MSK broker fails with:

Look up the real Kafka cluster ID for apacheKafkaClusterId

Set apacheKafkaClusterId to the cluster’s actual Kafka cluster ID, retrieved directly from the broker:

Do not pass a human-readable name or label — the field accepts any string at creation time with no validation, but the replicator will fail once it attempts to connect, reporting ConnectivityError.SourceApacheClusterIdInvalid or TargetApacheClusterIdInvalid after sitting in CREATING for several minutes. Look up the cluster ID separately for every physical cluster you reference this way — a Standard and an Express cluster in the same account have distinct cluster IDs even if they serve the same logical workload.

Create a copy of the SASL/SCRAM secret in the replicator’s own account

Store the replicator’s clientAuthentication.saslScram.secretArn in a Secrets Manager secret that lives in the same account as the replicator, even when the remote cluster and its own copy of the same secret live in a different account (required for BatchAssociateScramSecret on that cluster). In cross-account setups this means maintaining two Secrets Manager secrets with the same username/password: one in the source cluster’s account for BatchAssociateScramSecret, and one in the replicator’s account purely for the replicator config. The API enforces this directly:

Verify SCRAM secret association explicitly — do not rely on API success alone

After calling BatchAssociateScramSecret (directly or via a custom resource), confirm the association succeeded by running:

BatchAssociateScramSecret can return HTTP 200 while every secret in the request fails to associate — the per-secret failure is nested in responseElements.unprocessedScramSecrets, visible only in CloudTrail, not surfaced as an API error. A common cause is a missing KMS grant:

MSK performs the KMS CreateGrant call as the IAM principal that invoked BatchAssociateScramSecret (e.g. a Lambda execution role behind a custom resource), not as the kafka.amazonaws.com service principal. Grant kms:CreateGrant and kms:DescribeKey on the secret’s KMS key to that calling identity, scoped to the key’s ARN, rather than to the MSK service principal on the key policy.

Set topicNameConfiguration explicitly — do not rely on the default

Set topicNameConfiguration: { type: 'IDENTICAL' } whenever you need replicated topics to keep the exact source topic name — for example, transparent active-active/active-passive failover without reconfiguring clients, or any scenario that depends on the replicator’s loop-prevention behavior (the __mskmr header check only matters when source and target topic names can collide). If you omit topicNameConfiguration, the replicator defaults to PREFIXED_WITH_SOURCE_CLUSTER_ALIAS, producing names like <source-cluster-alias>.<original-topic-name> — a different, valid choice when you don’t need name parity, but treat it as a deliberate choice rather than an assumed default.

For bidirectional replication using IDENTICAL naming:

• Create both directions’ replicators (forward and reverse) in the same AWS account.
• Reference each physical cluster with a single, consistent type (AmazonMskCluster or ApacheKafkaCluster) across both replicator definitions. Do not swap a cluster’s type between the two directions — this breaks cross-account alias resolution and causes records to be re-replicated instead of being recognized as already-replicated.

Size partition count before scaling the replicator

Partition count on the source and target clusters is the binding constraint on replication throughput, not replicator capacity. Replicator scaling only converts into throughput when there are enough partitions to spread load across, so check partition count first whenever throughput sits below target or ReplicationLatency climbs. AWS’s own remedy for rising ReplicationLatency is to check whether the clusters have enough partitions (opens in a new tab).

Scaling cannot rescue an under-partitioned topic: 2 GB/s spread over 5 topic-partitions is unreachable at any replicator size, while the same 2 GB/s over ~100 partitions is achievable. Throughput also has to be balanced across those partitions — one hot partition caps the topic regardless of how many partitions exist.

Published sizing table

The recommended minimum partition counts (opens in a new tab):

Every row is exactly 0.3 MB/s per partition — the conservative cross-Region figure. Use the table as published for cross-Region replication. For same-Region it over-provisions by roughly 10-30x, so treat it as a safe upper bound rather than a requirement and size from the per-partition ranges below.

Per-partition throughput ranges

Capacity-modelling estimates only. These move with record size, batching, and cluster and replicator configuration, and are not service commitments — do not quote them to customers as guaranteed numbers.

Aggregate throughput in the multiple-GB/s range is reachable same-Region when load spreads evenly over a sufficient partition count, subject to the per-replicator quota below. If customer is unable to reach desired throughput, recommend opening an AWS Support Case to investigate.

Quotas that cap a single replicator

The 1 GB/s ingress cap lands exactly where the published sizing table tops out, so anything past ~1 GB/s has to shard across multiple replicators — bounded in turn by 15 per account. The 750-topic cap is the other reason to shard, and it selects in sorted order, so topics past the limit are skipped silently rather than reported as an error. Track TopicCount to confirm the replicator picked up every topic you expected.

Provision source and target clusters for replication traffic on top of existing production load — the replicator consumes from the source (egress) and produces to the target (ingress). AWS recommends identical capacity on both sides.

Confirm throughput from metrics, not configuration

Replicator metrics live in the AWS/Kafka namespace under the ReplicatorName dimension, in the target cluster’s Region, and are available per-topic via ReplicatorName, Topic:

Under IDENTICAL topic naming, ReplicatorBytesInPerSec exceeds ReplicatorThroughput by the volume of loop-filtered records, and you are billed on the larger number — bidirectional IDENTICAL setups pay data processing and transfer charges on records that get filtered rather than written. Under PREFIXED_WITH_SOURCE_CLUSTER_ALIAS nothing is filtered and the two metrics match, which makes an unexpected gap between them a useful signal that loop filtering is active.

A topic whose ReplicationLatency falls to 0 while the source topic’s MessagesInPerSec stays non-zero indicates a topic-level authorization failure, not an idle topic — AuthError does not cover topic-level auth errors. Check the replicator’s service execution role against that topic.

Grant kafka-cluster:DeleteTopic alongside standard produce/consume permissions

Include kafka-cluster:DeleteTopic in any IAM policy that will need to remove topics, including the replicator’s own internal bookkeeping topics during cleanup. A standard produce/consume/create-topic policy set (Connect, DescribeTopic, CreateTopic, AlterTopic, WriteData, ReadData) does not include it by default, and its absence only surfaces as TopicAuthorizationException at delete time.

See the IAM Replicator permissions reference (opens in a new tab) for the full set of kafka:* control-plane permissions the replicator’s own service execution role needs beyond AWSMSKReplicatorExecutionRole.

Clean up internal bookkeeping topics explicitly

Enumerate and delete the replicator’s internal bookkeeping topics as an explicit step in any teardown — deleting the replicator does not remove them. Each replication direction creates topics on both clusters resembling:

See manage-topics.md for topic deletion.

Enable log delivery at creation time

Configure logDelivery.replicatorLogDelivery.cloudWatchLogs when you create the replicator, not afterward. Without it, describe-replicator‘s terse StateInfo field is the only diagnostic signal available if something goes wrong. Adding log delivery to an existing replicator triggers a resource replacement (a new replicator, restarting replication from the configured starting position), so enable it up front instead.

Poll both StateInfo and CloudWatch Logs while a replicator is CREATING

Check describe-replicator‘s StateInfo field and the configured CloudWatch log group together, since either one may surface the actionable error first depending on the failure mode, and CREATING can take anywhere from a few minutes to 30+ minutes:

• An invalid apacheKafkaClusterId produces no CloudWatch log output at all until the replicator gives up — StateInfo is the only signal.
• An auth mechanism mismatch or bad credentials produces continuous CloudWatch log noise throughout CREATING.

Verify loop-prevention with an end-to-end produce/consume check, not resource state alone

Confirm replication is working correctly — and not looping — with a produce-then-double-read check, repeated per cluster type if you have separate Standard/Express replicator pairs:

1. Produce one record on the source topic.
2. Consume from earliest offset on the target topic — expect exactly 1 record.
3. Re-consume from earliest offset on the source topic — expect exactly 1 record (confirms the reverse replicator did not bounce the record back).

ReplicatorState: RUNNING and a correct list-scram-secrets result are necessary checks but do not by themselves confirm the absence of a duplicate-write loop.

Example: same-account MSK-to-MSK (CDK)

Example: cross-account MSK-to-MSK (remote referenced as self-managed)

References

• MSK Replicator overview (opens in a new tab)
• CreateReplicator API examples for self-managed Kafka clusters (opens in a new tab)
• Topic naming for MSK Replicator (opens in a new tab)
• Active-active replication (opens in a new tab)
• Active-passive cluster setup (opens in a new tab)
• IAM permissions reference for Replicator (opens in a new tab)
• SASL/SCRAM password limitations (opens in a new tab)
• Sizing and partitions best practices (opens in a new tab)
• Replicator metrics reference (opens in a new tab)
• Monitor an MSK Replicator (opens in a new tab)
• MSK Replicator quotas (opens in a new tab)