Performance Model Schema

This document specifies the JSON shape of benchmark_config.json, what each knob does, and how the defaults relate to measurement quality.

Minimal working example

Mode presets

mode (optional, default "standard") picks a preset for the timing/seed knobs. Explicit values in the config always win; the preset only fills fields the user did not set.

Unknown mode values are rejected at script startup (so a typo like "quik" can’t silently fall back to defaults).

Fields

Required

Resource naming

Tagging

Cold-start discipline

These three knobs control how bias from fresh-table capacity, client warmup, and partition priming is excluded from steady-state numbers. Tightening them trades measurement quality for total runtime. See SKILL.md ## Live validation — “Cold-start discipline” for the rationale.

Measurement window

Safety

Read-pattern sampling

How the knobs interact with measurement quality

• Bias removal is cumulative. The settle → seed → warmup sequence each removes a different class of bias. Skipping any one of them does not sum to “just a bit noisier” — the result is that the corresponding class of bias silently contaminates the steady-state numbers. Cold-start discipline is strongest when all three run.
• Sample size matters for percentiles. p99 over 60 samples is not a p99. If clamp(peak_rps × scale_factor, min, max) × duration_seconds < 100, the report flags that pattern as “percentiles are not statistically meaningful” and widens to p50/p95 only. Raise min_rps_per_pattern or widen duration_seconds to address.
• Extrapolation assumes linearity. Extrapolated monthly cost = mean observed CU × declared peak_rps × seconds-per-month × unit price. This is valid for per-op unit cost and GSI amplification; it is not valid for throttle behavior (throttles scale nonlinearly with partition-key skew — Mechanics #3). Patterns that hit max_rps_per_pattern and had throttles are flagged so the user does not read the extrapolation as a capacity-sustain claim. "representative" mode deliberately surfaces this nonlinearity: it drives hot-key skew to make throttles appear, and the report quarantines those throttle/latency numbers in a “Load-risk signals” section with an explicit “do not extrapolate” caption. The cost figure stays linear and valid because it uses declared peak and scale-invariant per-op CU — never the elevated driven RPS. Vector bytes extrapolate on the same basis and are similarly scale-invariant per call: metered search bytes measured exactly linear in TopK within a configuration, and index population is not a linear driver between 10 and 200 vectors in the searched partition because approximate-nearest-neighbour search prunes. One caveat on that last point: an empty 1024-dimension index measured roughly twice a populated one, so a benchmark run against a barely-seeded vector index can report higher per-search bytes than production will — raise seed_items_per_table before treating a vector search figure as representative.
• Shared-account throttle sensitivity. If the sandbox account has other workloads running, raise abort_on_throttle_rate only if you’ve verified there’s no external contention. The abort guard exists so a bad run stops instead of silently producing corrupted numbers.

Environment preconditions (not config knobs, but scripts will refuse without them)

• boto3 >= 1.34 installed. deploy_model.py and benchmark_model.py refuse with pip install boto3>=1.34 on import failure.
• Vector designs additionally need botocore/boto3 >= 1.43.64 (AWS CLI v2 >= 2.36.16). Below that floor there is no search_vectors operation and no VectorIndexes parameter at all — the failure looks like the feature does not exist rather than like a stale SDK. deploy_model.py checks the floor and refuses before creating anything. The benchmark Lambda gets a vetted SDK vendored into its bundle from the local install (only for designs that declare a vector index, so every other design keeps its small package), because the Lambda runtime’s built-in boto3 is older; the driver also fails loudly if search_vectors is missing rather than letting the pattern report zero consumed capacity and read as free.
• Valid AWS credentials resolvable via aws_profile. Expired SSO / missing token produces a specific remediation message (aws sso login --profile <p>) before any resource creation.
• Account identity does not contain prod markers. Even with --yes-deploy, deploy_model.py aborts if sts get-caller-identity returns an ARN or alias containing prod, production, prd, or live.
• dynamodb_data_model.json has non-empty access_patterns and every pattern has peak_rps > 0. Unknown RPS is a design gap (Mechanics #2), not a benchmark input.

Tuning guide