Alarms

• Alarm types
• Missing data treatment
• Evaluation mechanics
• Composite alarms
• Anomaly detection
• Recommended defaults
• Common mistakes
• CDK patterns

Alarm types

Metric Alarm

Watches a single metric or metric math expression.

• States: OK, ALARM, INSUFFICIENT_DATA
• Actions: SNS, EC2 (stop/terminate/reboot/recover), Auto Scaling, Lambda, SSM OpsItems, SSM Incident Manager, CloudWatch Investigations
• M-of-N evaluation: DatapointsToAlarm (M) out of EvaluationPeriods (N)
• Rate limit: PutMetricAlarm = 3 TPS (adjustable)

put-metric-alarm is an UPSERT, keyed on --alarm-name. Reusing the name of an existing alarm silently overwrites its whole configuration — threshold, evaluation periods, and its actions, including the SNS topics it notifies. There is no error and no previous version to restore, so an alarm someone depends on can stop paging with nothing to indicate it changed. Before writing to a name you did not just create, check whether it exists (describe-alarms --alarm-names <name>); if it does, show the customer the diff and confirm rather than overwriting on inference.

Composite Alarm

Combines states of other alarms with Boolean logic.

• Rule operators: AND, OR, NOT, AT_LEAST(M, STATE, (alarms...))
• AT_LEAST supports percentages: AT_LEAST(50%, ALARM, (a1, a2, a3))
• Actions: SNS, Lambda, SSM — cannot perform EC2 or Auto Scaling actions
• Limits: max 100 underlying alarms per composite, 150 composites per underlying, 500 rule elements
• Composite and all underlying alarms must be in the same account and Region
• Action suppression: ActionsSuppressor alarm can suppress composite alarm actions during known events (deployments, maintenance)

PromQL Alarm (OpenTelemetry metrics)

Monitors OTel metrics using PromQL instant queries with duration-based pending/recovery periods. Use for metrics sent via OTLP (150 labels, 30-day retention).

Missing data treatment

Four options — the most misunderstood CloudWatch feature.

Note: The CloudWatch console defaults DynamoDB alarms to ignore instead of the usual missing. The API stores whatever you specify.

Premature alarm transitions

With treatMissingData=missing, the pattern M, M, B, M, M can trigger ALARM even with only 1 breaching datapoint. CloudWatch goes to ALARM when the oldest available breaching datapoint is at least as old as datapointsToAlarm and all more recent points are breaching or missing.

Fix: For non-sparse metrics, explicitly set notBreaching or breaching — don’t rely on the default.

Evaluation mechanics

Three core settings

1. Period — seconds per data point aggregation (valid: 10, 20, 30, or any multiple of 60)
2. Evaluation Periods (N) — number of most recent periods to evaluate
3. Datapoints to Alarm (M) — how many of N must breach

Evaluation frequency

• Period ≥ 1 min → evaluated every minute
• Period = 10s/20s/30s → evaluated every 10 seconds
• If EvaluationPeriods × Period > 1 day → evaluated once per hour

Evaluation Range

CloudWatch fetches more data points than the configured Evaluation Periods — the actual lookback window is wider than expected.

Example: Alarm with 1-day period, 1 evaluation period, treatMissingData=breaching:

• You expect it to fire after 1 day of no data
• CloudWatch actually looks back ~3 days before firing
• Dead man switch alarms fire later than expected due to hourly evaluation

Evaluation period quotas

• Period ≥ 1 hour → max evaluation window: 7 days
• Period < 1 hour → max evaluation window: 1 day

Composite alarms

When to use

• Reduce alert fatigue: only page when BOTH high CPU AND high error rate
• Service-level health: aggregate per-resource alarms into one service alarm
• Suppress during deployments: use ActionsSuppressor to mute during known events

Rule expression syntax

Limitations

• Cannot perform EC2 actions (stop, terminate, reboot, recover)
• Cannot perform Auto Scaling actions
• Composite and all underlying alarms must be in the same account and Region (underlying alarms must be same account + Region; monitoring accounts via OAM can watch source account metrics)
• Cross-account observability monitoring account CAN watch source account alarms

Anomaly detection

• Uses ANOMALY_DETECTION_BAND function as threshold
• Band width = anomaly detection threshold value (configurable; higher value = thicker band of expected values)
• Trains on up to 2 weeks of metric data (works with less, accuracy improves over time)
• Cost: Higher than a regular alarm — see CloudWatch pricing (opens in a new tab) for current anomaly detection alarm rates
• Rate limit: 1,000 ANOMALY_DETECTION_BAND usages in GetMetricData per second
• Use when: baselines are unknown, workloads are seasonal/variable

Recommended defaults

WARNING: Never use Average for duration/latency alarms. Average hides tail latency — use p99 or p90. A function averaging 100ms but with p99 at 5s has a serious problem that Average won’t catch.

Common mistakes

1. M=N=1 with 1-minute periods — Too sensitive. The most recent datapoint may not have full information. Use “1 out of 2” or “1 out of 3” minimum.

2. Relying on default missing treatment — Explicitly configure for your metric type. Error metrics should use notBreaching. Health checks should use breaching.

3. Not understanding Evaluation Range — Alarms look back further than configured. Dead man switches with multi-day periods are evaluated once per hour, causing significant delay.

4. Metric math alarms for EC2 actions — Alarms based on metric math expressions cannot perform EC2 actions (stop, terminate, reboot, recover). Use a simple metric alarm instead.

5. High-resolution alarms without need — 10-second evaluation costs more. Each metric in a math expression is billed separately.

6. Using Average statistic for duration/latency alarms — Average hides tail latency. A function averaging 100ms with p99 at 5s has a serious problem Average won’t catch. Always use p99 or p90 via --extended-statistic p99.

7. Ignoring DynamoDB’s default override — DynamoDB alarms default to ignore for missing data, not the global missing.

8. Alarms on INSUFFICIENT_DATA state — Alarms invoke actions only on state changes, except Auto Scaling actions which continue invoking while in the new state.

CDK patterns

Error rate alarm (production pattern)

Note: Alarm on error rate (percentage via math expression), not raw error count. Raw counts trigger on a single error even during 10,000 successful invocations.

For CLI:

For CDK:

Duration/latency alarm (use p99, never Average)

For CLI:

Composite alarm

Anomaly detection alarm (CloudFormation)

Source

• references/cloudwatch/alarms.md(opens in a new tab)
• Raw file(opens in a new tab)
• aws/agent-toolkit-for-aws(opens in a new tab)