Event Sources

Contents

• SQS event source mapping
• DynamoDB Streams triggers
• Event filtering
• Partial batch failure reporting
• Error handling and concurrency

SQS event source mapping

Lambda long-polls SQS and invokes your function synchronously with a batch.

MaximumConcurrency and Provisioned Mode are mutually exclusive on the same ESM.

Lambda invokes when any of: batching window expires, batch size reached, or payload hits 6 MB.

Scaling

Standard queues: concurrency starts at 5 concurrent invocations and scales up by ~300 per minute to a default maximum of 1,250 — so a sudden burst (e.g. a backlog of 10,000 messages) is not absorbed immediately; it takes several minutes to ramp. For high scale-out or high-throughput workloads, use Provisioned Mode, which starts higher, ramps significantly faster, and reaches a much larger ceiling. The 1,250 default is also bounded by account concurrency; confirm current account limits with aws service-quotas get-service-quota --service-code lambda.

FIFO queues: concurrency capped by the lower of (number of message group IDs, MaximumConcurrency); order preserved per group ID.

CDK: fn.addEventSource(new SqsEventSource(queue, { batchSize, maxBatchingWindow, reportBatchItemFailures: true, maxConcurrency })). Harden the queue: new sqs.Queue(this, 'MyQueue', { encryption: sqs.QueueEncryption.SQS_MANAGED, enforceSSL: true, ... }) (set the same on the DLQ).

DynamoDB Streams triggers

Lambda polls stream shards 4×/second; invokes synchronously, in-order per partition key.

Key behaviors

• Max 2 Lambda readers per shard (single-region tables). Global tables: limit to 1.
• TRIM_HORIZON recommended — LATEST may miss events during ESM creation.
• 100 shards × ParallelizationFactor 10 = up to 1,000 concurrent invocations (order maintained at partition-key level).
• Stream retention is 24 hours — a poison record can block a shard for that entire window if retries aren’t bounded. Set MaximumRetryAttempts / MaximumRecordAgeInSeconds / BisectBatchOnFunctionError.

Encrypt the source table at rest — CDK new dynamodb.Table(this, 'MyTable', { encryption: dynamodb.TableEncryption.AWS_MANAGED, ... }), or a customer managed KMS key when compliance requires key control.

SNS filter policies (not Lambda FilterCriteria) are SNS-managed; set FilterPolicyScope to MessageAttributes (default) or MessageBody.

Event filtering

FilterCriteria applies to ESMs only (not SNS/push triggers).

• Up to 5 filters per ESM (request increase to 10). Multiple filters are OR‘d; fields within one filter are AND‘d.
• DynamoDB numeric filtering is unsupported — numbers are stored as strings in the DynamoDB stream JSON. Use {"S": [...]}/{"N": ["123"]} string matches, not {"numeric": [...]}.
• Format mismatch drops the record: if the incoming body is plain string but the filter is JSON (or vice versa), Lambda drops the message.

Operators: ["value"], {"equals-ignore-case"}, [null], [""], {"anything-but"}, {"numeric": [">", 10, "<=", 20]}, {"exists": true}, {"prefix"}, {"suffix"}, "$or": [...].

Partial batch failure reporting

Set FunctionResponseTypes: [ReportBatchItemFailures] and return the failed identifiers.

• SQS: return failed messageId values in batchItemFailures.
• Streams (DynamoDB/Kinesis): return the failed SequenceNumber; Lambda checkpoints at the lowest returned sequence number and retries everything from that point.
• FIFO: stop after the first failure; return that message plus all unprocessed ones (preserves ordering).

Response edge cases that cause a complete batch retry: itemIdentifier empty/null, a bad key name, or any unhandled exception. An empty/null batchItemFailures = complete success.

Streams interaction: an unhandled exception triggers BisectBatchOnFunctionError (no response returned, so ReportBatchItemFailures has no effect); a success with batchItemFailures checkpoints at the lowest failed sequence number.

The Powertools Batch Processor (process_partial_response) handles all of this — prefer it over hand-rolled loops. See assets/powertools-handler.py.

Error handling and concurrency

Concurrency formulas

Idempotency

All event sources deliver at-least-once — duplicates happen. Make handlers idempotent (Powertools Idempotency utility, keyed e.g. on SQS messageId). See production.md.