Setting the file. One moment.
references/foundation-operations.mdMarkdown12 KBView on GitHub
Cross-cutting operational knowledge: MSF service quotas, ENI capacity planning for VPC apps, MSF vs EMR Flink decision, and source/sink selection. Use when planning capacity, choosing the right streaming service, or designing a new pipeline.
| Item | Value | Common wrong values |
|---|---|---|
| AWS CLI service / SDK client | kinesisanalyticsv2 | ❌ flink, msf, kinesisanalytics (v1, deprecated SQL apps only) |
| Service Quotas service-code | kinesisanalytics (no v2) | ❌ kinesisanalyticsv2, flink, msf |
| IAM action prefix | kinesisanalytics: (no v2) | ❌ kinesisanalyticsv2: |
| CloudWatch namespace | AWS/KinesisAnalytics | ❌ AWS/Flink, AWS/ManagedFlink |
| Trust policy principal | kinesisanalytics.amazonaws.com | ❌ kinesisanalyticsv2.amazonaws.com |
The CLI/SDK is the only identifier that uses the v2 suffix. Service Quotas, IAM actions, the CloudWatch namespace, and the trust principal all use the legacy kinesisanalytics name. Treating the v2 form as the “default” and applying it everywhere is the single most common source of permission failures, empty metric results, missing service-quota lookups, and trust policy errors.
The MSF execution role is assumed by the MSF service to access your data plane resources on behalf of the application. It is not used by the application code itself, and it does not call MSF’s own control plane. The principle of least privilege follows from that:
| Permission | Required in execution role? | Why |
|---|---|---|
Source/sink data plane (e.g. kinesis:GetRecords, s3:PutObject, kafka:DescribeCluster) | Yes — scoped to specific stream/bucket/cluster ARNs | The service uses this role to read sources and write sinks |
logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogStreams on the configured log group | Yes | The service writes application logs to the configured CloudWatch Logs group |
EC2 ENI permissions (ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces, ec2:DeleteNetworkInterface, ec2:CreateNetworkInterfacePermission, ec2:DescribeVpcs, ec2:DescribeSubnets, ec2:DescribeSecurityGroups) | Yes — only for VPC-enabled apps | The service creates ENIs on your behalf when VPC is configured |
kinesisanalytics:* actions | No | These are MSF control-plane actions consumed by humans/CI calling the MSF API, not by the service when it runs your application |
cloudwatch:PutMetricData | No | MSF publishes the standard metrics (cpuUtilization, downtime, numRecordsIn*, etc.) to the AWS/KinesisAnalytics namespace from the service plane, not via the execution role. Adding it does no harm if scoped to the namespace, but it’s noise — leave it out for a clean least-privilege role. The exception: if your application code explicitly calls CloudWatchAsyncClient.putMetricData() to emit custom application metrics, then you do need to grant it (still scope to the custom namespace) |
secretsmanager:GetSecretValue | Only if you use Secrets Manager for connector credentials | Scope to the specific secret ARN |
kms:Decrypt | Only when reading from a KMS-encrypted source/sink or KMS-encrypted secret | Scope to the specific key ARN |
When generating an execution role, default to omitting kinesisanalytics:* and cloudwatch:PutMetricData unless the user explicitly says they emit custom metrics. The CloudWatch metrics you see in the AWS console come from the service, not from this role.
| Quota | Default | Adjustable |
|---|---|---|
| Applications per region | 50 | Yes |
| KPUs per application | 64 (configurable to 250) | Yes |
| Application snapshots per application | 1000 | Yes |
| Parallelism per application | 256 | Yes |
Check current values:
aws service-quotas list-aws-default-service-quotas \
--service-code kinesisanalytics --region "$REGION"The Service Quotas service-code is kinesisanalytics (legacy name), not kinesisanalyticsv2. The KPUs-per-application quota code is L-3A88E041 (Apache Flink Kinesis Processing Units (KPUs)).
Request increases via request-service-quota-increase with the quota code from the list output. Increases for KPU and snapshot quotas typically approve within a business day.
MSF creates one ENI per allocated KPU, in each subnet, for VPC-enabled applications. Subnet sizing and the regional ENI quota both matter.
ENIs_per_subnet = KPUs
required_subnet_IPs = ENIs_per_subnet + 20% headroom for scaling and rolling restartsExample: 16 KPU app 16 ENIs per subnet. Allow 20 available IPs per subnet to account for 20% buffer. A /28 subnet (11 usable IPs) is too small; use /27 (27 usable) or larger.
Regional ENI quota (vpc service code, Network interfaces per region) defaults to 5,000. Each VPC-enabled MSF KPU consumes one. Multiple large MSF apps in the same region can pressure this quota — check before deploying. Other services in the VPC (Lambda, ECS, RDS) also consume ENIs from the same quota.
Subnets should span at least 2 AZs for fault tolerance.
| Source/Sink | VPC Required |
|---|---|
| Kinesis Data Streams | No (public endpoint) |
| Amazon S3 | No (public endpoint, S3 gateway endpoint optional) |
| DynamoDB | No (public endpoint) |
| Firehose | No |
| Amazon MSK (Kafka) | Yes |
| RDS / Aurora | Yes |
| ElastiCache | Yes |
| OpenSearch in VPC | Yes |
| Self-hosted Kafka | Yes if private |
| Public Kafka / external API | No, but needs NAT gateway in VPC subnets |
VPC apps without a NAT gateway lose access to public AWS endpoints (CloudWatch, S3 if not using gateway endpoint, Kinesis). Symptoms: silent failure of metric publishing, S3 checkpoint failures, deserialization errors trying to call schema registry.
| Factor | MSF | EMR Flink |
|---|---|---|
| Operations | Fully managed, no clusters | Self-managed EC2/EKS clusters |
| Scaling | KPU autoscaling (CPU-only) | Manual cluster scaling |
| Billing | Per KPU-hour | Per EC2-hour (+ EMR surcharge) |
| Flink version | AWS-managed (1.15, 1.18, 1.19, 1.20, 2.2) | Any Flink version, including custom builds |
| Custom connectors | Limited to bundled JARs / fat-JAR upload | Full Flink ecosystem |
| Job isolation | One job per application | Multiple jobs per cluster |
| Startup time | 1–3 min | 5–15 min cluster boot |
| Max parallelism | 256 (quota-adjustable) | Unlimited (cluster size) |
| State backend | Managed RocksDB | Self-managed RocksDB / heap |
Choose MSF when: zero infrastructure management, single-job-per-app is acceptable, parallelism fits within KPU limits, fast iteration matters.
Choose EMR Flink when: custom Flink connectors not in MSF, multiple Flink jobs sharing infrastructure, specific Flink version or patch control, parallelism exceeds 256, fine-grained CPU/memory ratios needed.
Cost crossover: EMR is typically cheaper at large scale (10+ KPU equivalent, 24/7) due to EC2 commitment savings, but the operational overhead (cluster patching, scaling, monitoring) typically erases the savings unless EMR expertise already exists.
| Source | Best For | Throughput | VPC | Ordering | Retention |
|---|---|---|---|---|---|
| Kinesis Data Streams (KDS) | AWS-native ingestion, < 1 GB/s, Lambda integration | Per-shard (1 MB/s in, 2 MB/s out) | No | Per-shard | 1–365 days |
| Amazon MSK | Kafka ecosystem, complex routing, > 1 GB/s | Per-broker (hundreds MB/s) | Yes | Per-partition | Unlimited (storage-based) |
| Amazon S3 | Batch-to-stream replay, reprocessing | Bulk file scan | No | File ordering | Indefinite |
KDS pitfalls: shard count drives parallelism; shard count is hard to change post-creation. Cost scales linearly with shard count, not throughput.
MSK pitfalls: broker provisioning takes hours; cross-AZ replication doubles network cost; SASL/IAM auth requires careful security group setup.
S3 source pitfalls: no event-time ordering across files unless designed in; file enumeration is the bottleneck for large prefix counts.
For source-side EFO and Kinesis polling tradeoffs, see kinesis-efo-guide.md. For MSK setup, see the iac-and-deployment.md deployment patterns.
| Sink | Best For | VPC | Ordering Preserved |
|---|---|---|---|
| S3 (Parquet/ORC) | Data lake, batch analytics, Athena | No | File-level only |
| Iceberg | Transactional data lake, schema evolution, time travel | No (catalog-dependent) | Yes (commit-order) |
| Kinesis Data Streams | Real-time downstream consumers | No | Per-shard |
| MSK | Kafka ecosystem | Yes | Per-partition |
| OpenSearch | Search, log analytics, dashboards | Yes (VPC mode) or No (public) | No (eventually consistent) |
| DynamoDB | Low-latency key-value lookups | No | No (last-write-wins per key) |
| RDS / Aurora | Relational writes, joins on results | Yes | Yes (transaction order) |
| Firehose | Managed delivery to S3/Redshift/OpenSearch with batching | No | Within batch |
S3 small files anti-pattern: Streaming writers commit on every checkpoint, creating one file per checkpoint per partition. With 60s checkpoints and 8 partitions, a job writes 11,520 files/day. Use Iceberg with compaction, or batch via Firehose, or increase checkpoint interval. See iceberg-tuning-and-operations.md.
RDS/Aurora sink anti-pattern: Per-record JDBC writes are 10–100× slower than batched writes. Use the JDBC sink with batching enabled. RDS connection pool limits (typically 5,000) cap effective parallelism.
| Pattern | Use Case | VPC |
|---|---|---|
| KDS → MSF → S3/Iceberg | Data lake ingestion | No |
| MSK → MSF → MSK | Stream-to-stream enrichment | Yes |
| KDS → MSF → DynamoDB | Real-time aggregation serving | No |
| KDS → MSF → OpenSearch | Real-time search index | Optional |
| CDC source → MSF → Iceberg | Database replication to lake | Yes (DB side) |
| MSK → MSF → S3 + KDS | Fan-out to lake and downstream consumers | Yes (MSK side) |
For private connections (MSK, RDS, OpenSearch in VPC), MSF KPUs must be in the same VPC or peered VPC as the source/sink. Cross-AZ data transfer between MSF and MSK is always billed at $0.01/GB each direction — at 100 MB/s sustained that is $26K/year. Place MSF subnets in the same AZs as MSK brokers and use rack-aware producers/consumers where supported.
| Mistake | Impact |
|---|---|
Using kinesisanalytics for the CLI / SDK control plane | Returns v1 SQL apps only, not Flink — use kinesisanalyticsv2 for the CLI/SDK |
kinesisanalyticsv2 for the Service Quotas service-code | NoSuchResourceException — Service Quotas uses the legacy kinesisanalytics code |
kinesisanalyticsv2: IAM actions | All API calls denied — IAM uses kinesisanalytics: prefix |
AWS/Flink namespace in CloudWatch | Empty metric results |
kinesisanalyticsv2.amazonaws.com trust principal | MSF cannot AssumeRole — trust principal is kinesisanalytics.amazonaws.com |
| Hardcoding region lists for cross-region discovery | Disabled regions cause AccessDeniedException — use describe-regions |
| Sizing subnets without ENI headroom | Application stuck STARTING; ENIs cannot be created |
| Choosing EMR for a single-job pipeline | Operational overhead exceeds infrastructure savings |
| Mismatched MSF and source AZs | Persistent cross-AZ data transfer cost |