Setting the skill. One moment.
Analytics Skills
Skill 71 of 156
Operates Amazon MSK Provisioned clusters (Standard and Express brokers).
4 minutes · 975 words · 6 sections
Install
npx skills add aws/agent-toolkit-for-aws --skill managing-amazon-msknpx skills add aws/agent-toolkit-for-aws/plugin marketplace add aws/agent-toolkit-for-awsThe first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.
Domain expertise for operating Amazon MSK Provisioned clusters with Standard and Express broker types. Covers performance troubleshooting, consumer lag diagnosis, storage management, cluster sizing, client configuration, and CloudWatch monitoring.
Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.
Standard brokers use customer-managed EBS volumes for storage. You choose instance types (kafka.m5/m7g families), provision EBS, and manage storage scaling.
Express brokers use instance types prefixed with express.m7g and are the default recommendation for almost all MSK workloads — they typically cost less, not just less effort. Up to 3x ingress per broker (MSK Express broker types (opens in a new tab)) means fewer brokers for the same load, and storage is billed per GB-hour on data actually retained rather than provisioned up front on EBS that cannot shrink. They also scale 20x faster, rebalance partitions 180x faster (Intelligent Rebalancing (opens in a new tab)), recover 90% quicker (MSK Express broker types (opens in a new tab)), and have no maintenance windows. Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned throughput for Express clusters. Express brokers enforce fixed replication factor of 3 and min.insync.replicas=2. See size-and-choose-cluster.md (opens in a new tab) for the full Standard vs Express decision framework.
Determine the broker type first: aws kafka describe-cluster-v2 --cluster-arn <arn>. Check Provisioned.BrokerNodeGroupInfo.InstanceType — if it starts with express., it is an Express cluster.
| Customer Intent | Reference |
|---|---|
| High CPU, high latency, slow cluster, traffic shaping | troubleshoot-performance.md (opens in a new tab) |
| Consumer lag increasing, rebalance storms, stuck consumer groups | troubleshoot-consumer-lag.md (opens in a new tab) |
| Disk filling up, retention planning, tiered storage | manage-storage.md (opens in a new tab) |
| Choosing Standard vs Express, sizing a cluster, partition limits, broker count, monthly cost | size-and-choose-cluster.md (opens in a new tab) |
| Producer/consumer configuration, IAM/SCRAM/TLS auth | configure-clients.md (opens in a new tab) |
Creating/applying MSK configurations (server.properties); custom domain names on brokers via custom.advertised.listeners (advertised listeners, static/custom bootstrap endpoint) — validation rules, apply/rollback, scaling; migrating from the dynamic per-broker kafka-configs.sh override to the static property | configure-cluster.md (opens in a new tab) |
| Client-side connectivity for a custom domain: NLB + ACM certificate + Route 53 fronting, TLS handshake/termination, mTLS through an NLB, cross-zone load balancing | configure-clients.md (opens in a new tab) (Custom Domain Name Connectivity section) |
Create, update, delete, list, or describe Kafka topics; when to use the native topic APIs / CfnTopic vs. a Kafka admin client | manage-topics.md (opens in a new tab) |
| MSK Replicator setup, migration, DR, active-active/active-passive, cross-account replication, self-managed-to-MSK or MSK-to-MSK, loop-prevention/topic-naming behavior | msk-replicator.md (opens in a new tab) |
Replication throughput per partition, how many partitions a replicator needs, why replicator scaling isn’t increasing throughput, replicator quotas (topics, ingress, record size), ReplicationLatency/MessageLag/ReplicatorThroughput | msk-replicator.md (opens in a new tab) (Size partition count section) |
| Setting up monitoring, dashboards, alarms | monitor-and-alarm.md (opens in a new tab) |
| Full CloudWatch metric list (Standard or Express) | Prefer monitor-and-alarm.md (opens in a new tab) for strategic recommendations and how to interpret metrics, only search documentation if you need to understand a metric not included in this reference file (MSK Standard CloudWatch Metrics (opens in a new tab), MSK Express CloudWatch Metrics (opens in a new tab)) for full list |
| Rolling restart impact, patching, maintenance resilience | maintenance-operations.md (opens in a new tab) |
| Deliver streaming data to Apache Iceberg tables on S3 Tables with low cost in a fully managed service (Streaming Tables) — setup, IAM, schema, create/update/delete/list/describe channels | streaming-tables.md (opens in a new tab) |
| Deliver topic data to S3 bucket as JSON/ByteArray/String objects with low cost in a fully managed service (Data Delivery for General Purpose S3 buckets) — setup, IAM, output key templates, create/update/delete/list/describe channels | data-delivery-for-general-purpose-s3.md (opens in a new tab) |
| Build a lakehouse / data lake from Kafka; make streaming data queryable in Athena | streaming-tables.md (opens in a new tab) |
| Alternative to Kafka Connect S3 Sink or Amazon Data Firehose for MSK; zero-ops streaming delivery to S3 | data-delivery-for-general-purpose-s3.md (opens in a new tab) |
| Streaming Tables / Data Delivery CloudWatch metrics and alarms, DLQ errors, failed deliveries, channel state transitions, freshness lag | streaming-tables-troubleshooting.md (opens in a new tab) |
| “Can I use Streaming Tables / Data Delivery on MSK Serverless / Standard brokers?” — eligibility routing | streaming-tables.md (opens in a new tab) (answer is always: Express brokers only, use Firehose, Flink, or Kafka Connect for Standard and Serqverless - Firehose integration for Amazon MSK (opens in a new tab)) |
| What are the current supported Kafka versions for MSK? | Supported Apache Kafka versions (opens in a new tab) |
| Does MSK support KRaft clusters, and how do I migrate between ZooKeeper and KRaft mode clusters? | zk-to-kraft-migration.md (opens in a new tab) |
| What are the current quotas for MSK Express (ingress, egress, partitions, broker count, etc.)? | MSK Express Quotas (opens in a new tab) |
| What are the current quotas for MSK Standard (partitions, broker count, etc.)? | MSK Standard Quotas (opens in a new tab), and MSK Standard best practices (opens in a new tab) for partition count limits |
| What broker-level configuration changes can I make on MSK Express or Standard brokers? | MSK Configuration (opens in a new tab) |
scripts/msk_sizing.py — MUST be run for any sizing question (broker count, instance choice, cost). Prices from the live AWS Pricing API when AWS credentials with pricing:GetProducts are available, and falls back to an embedded us-east-1 snapshot when they are not, so it runs with or without credentials. See size-and-choose-cluster.md (opens in a new tab) for the required workflow and script reference.This skill can be loaded two ways, and they resolve the skill’s own bundled files — the references/ documents and the scripts/ files
from different places. Determine how the skill was loaded before you read a reference or run a script:
retrieve_skill tool call. The skill is not
installed on the local filesystem; its reference files and scripts do not
exist on disk. You MUST fetch each reference or script through the same
retrieve_skill tool by passing the file parameter (for example,
file="references/configure-clients.md" or file="scripts/msk_sizing.py"),
and run a script from the content that tool returns. Do NOT file_read these
paths from the local or working directory, and do NOT search the filesystem
for them — they are not there, and any local file that happens to match the
name is unrelated to this skill..claude/skills/managing-amazon-msk/, ~/.claude/skills/managing-amazon-msk/, or
.kiro/skills/managing-amazon-msk/). Read references and run scripts from the
local skill directory using the relative paths shown throughout this
documentation.This distinction applies only to the skill’s own packaged files. Every artifact
created during a session or supplied by users are read from and written to
the user’s working directory regardless of how the skill was loaded. Never
fetch or write customer data through retrieve_skill.
Create/apply Amazon MSK configurations and set custom domain names — creating an Amazon MSK configuration (server.properties with the fileb:// real-newline requirement), applying it with update-cluster-configuration, and setting broker custom domain names via custom.advertised.listeners: see configure-cluster.md (opens in a new tab). For the NLB/certificate/DNS connectivity that fronts a custom domain, see configure-clients.md (opens in a new tab).
Operates Amazon MSK Provisioned clusters (Standard and Express brokers). Required for ANY MSK Provisioned task — training data conflates Standard and Express. Covers performance, consumer lag, storage, traffic shaping; sizing Standard vs Express; Kafka client tuning; CloudWatch alarms; cluster configs; maintenance, patching, rolling restarts; topic management; MSK Replicator setup for DR and active-active/active-passive replication (same-/cross-account), including throughput and partition sizing; Streaming Tables for S3 Tables and Data Delivery for General Purpose S3 — setup, IAM, monitoring. Prefer Flink for Kafka Iceberg sink questions. Migrations: use migrate-to-msk. Triggers: MSK Provisioned, Kafka, `kafka.*`/`express.*` types, AWS/Kafka namespace, cross-cluster replication, Kafka to Iceberg on S3 Tables, Kafka to S3, lakehouse, data lake from Kafka, Kafka Connect S3 Sink or Firehose alternative. DO NOT USE for MSK Connect — search docs instead. Serverless: only S3 Tables/data delivery eligibility.
The verbatim description from this skill’s front matter — the string an agent matches on to decide whether to load it.
skills/specialized-skills/analytics-skills/managing-amazon-msk/SKILL.mdmain, last pushed 7 November 2026.SKILL.md, not by matching a directory convention. 22 distinct layouts observed: plugins/aws-agents-for-devsecops/skills/*/SKILL.md, plugins/aws-agents/skills/*/SKILL.md, plugins/aws-core/skills/*/SKILL.md, plugins/aws-startup-advisor/skills/*/SKILL.md, skills/core-skills/*/SKILL.md, skills/specialized-skills/analytics-skills/*/SKILL.md, skills/specialized-skills/aws-marketplace-skills/*/SKILL.md, skills/specialized-skills/customer-communications-skills/*/SKILL.md, skills/specialized-skills/database-skills/*/SKILL.md, skills/specialized-skills/ec2-skills/*/SKILL.md, skills/specialized-skills/end-user-computing-skills/*/SKILL.md, skills/specialized-skills/media-services-skills/*/SKILL.md, skills/specialized-skills/migration-and-modernization-skills/*/SKILL.md, skills/specialized-skills/networking-and-content-delivery-skills/*/SKILL.md, skills/specialized-skills/operations-skills/*/SKILL.md, skills/specialized-skills/quantum-computing-skills/*/SKILL.md, skills/specialized-skills/resilience-skills/*/SKILL.md, skills/specialized-skills/security-and-identity-skills/*/SKILL.md, skills/specialized-skills/serverless-skills/*/SKILL.md, skills/specialized-skills/storage-skills/*/SKILL.md, skills/specialized-skills/system-table-skills/*/SKILL.md, skills/specialized-skills/web-and-mobile-development/*/SKILL.md.h1 and no skipped levels:.claude-plugin/marketplace.json by Amazon Web Services, declaring 5 plugins. It is read for editorial metadata only — never as the skill index, which is always the repository tree./aws/agent-toolkit-for-aws.md, and each skill at its own .md URL.15 files · 229 KB
Everything this skill ships beside its prose. All of it is set here, as subchapters of skill 71.
Documentation the agent loads on demand, rather than up front.
Executable code the skill can run.