Scaling And Concurrency

Concurrent Execution Environments

Each execution environment handles multiple concurrent requests.

Lambda Web automatically load balances and scales an endpoint’s environments by managing a dynamic per-environment concurrency limit and target utilization. For example, Lambda Web may not route new requests to an environment with high sustained CPU utilization; Lambda Web will instead scale out new environments to handle the load.

serviceConfig.maxConcurrencyPerEnvironment defines the upper bound of the internally managed dynamic concurrency limit. It is configured on the revision, defaults to 64, and can be set between 1–128. Most users don’t need to change this field and can rely on Lambda Web’s dynamic concurrency limit to balance environment utilization and performance.

See Multi-Concurrency Considerations in architecture-patterns.md (opens in a new tab).

New Endpoints Start With a Low Concurrency Limit

The dynamic limit starts conservatively on a new endpoint, including a new endpoint added to an existing function, and rises as Lambda learns the function’s resource usage. It is kept when the endpoint scales to zero, so later traffic starts from the learned level. A new revision whose performance differs a lot from the previous one also goes through a short learning period.

While the limit is low, each environment takes only a few concurrent requests, so a burst needs more environments than it will later. If scalingConfig.maxEnvironments is set low, that burst hits the cap and gets 429 with lambda-web-error-type: Platform.EndpointMaxEnvThrottle until the limit rises. Left unset, the endpoint scales out instead and the burst is served.

Do not set maxEnvironments unless the user asks for a cap. When they do, tell them a low cap on a new endpoint can throttle bursts and load tests for the first few minutes, and that results from a first load test are not representative of steady state; repeat the test and use the later runs.

Endpoint Scaling Controls

There are two controls to limit an endpoint’s scale within a region. Both are endpoint properties and can be updated without creating a new revision:

scalingConfig.maxEnvironments constrains the number of environments an endpoint can scale to. Optional, default is null.

• If a request arrives when all maxEnvironments environments are fully utilized, the request will be immediately throttled rather than queued (429, Platform.EndpointMaxEnvThrottle). On a new endpoint this happens at lower load than later, see New Endpoints Start With a Low Concurrency Limit.
• During revision updates, internal fleet cycling, and internally managed operational events, Lambda Web may temporarily exceed the maxEnvironments as Lambda Web generally prioritizes availability over strictly honoring the maxEnvironments.
• A large maxEnvironments value does not guarantee the ability to scale to that many environments. An endpoint’s environment scaling is always limited by the regional vCPU account quota. See Service Limits in SKILL.md (opens in a new tab).
• If maxEnvironments is configured, an endpoint’s total concurrency limit can be thought of as maxEnvironments × dynamic concurrency limit.

throttleConfig.rateLimit constrains the endpoint’s request arrival rate to a value lower than account quotas. Optional, default is null.

• An endpoint’s request rate is always limited by the two related account quotas: Per-endpoint rate limit and account total rate limit. See Service Limits in SKILL.md (opens in a new tab).
• A rateLimit of 0 blocks all traffic.

Reference Table