Observability

Pre-PMF (Seed / <$1M ARR)

• CloudWatch only. No Datadog, no New Relic, no Grafana Cloud. Third-party observability tools cost $15-50/host/month and grow linearly. At this stage, CloudWatch + Embedded Metric Format is sufficient and covered by AWS credits.
• 3 metrics, 3 alarms, 1 dashboard. Error rate, p99 latency, and request count. That’s it. You don’t need 47 dashboards for 1 service.
• Log retention: 7 days dev, 30 days prod. The default “never expire” will silently accumulate $50-200/month in log storage within 6 months.
• Skip X-Ray until you have 3+ services. Tracing a monolith tells you nothing useful. Tracing across service boundaries is where it helps.

Post-PMF / Growth ($1M-$10M ARR)

• Add X-Ray or OpenTelemetry when you have 3+ services and debugging cross-service issues takes >1 hour.
• Consider Datadog/New Relic ONLY if CloudWatch Logs Insights becomes too slow for your team’s debugging workflow. The convenience costs 5-10x.
• Increase log retention to 90 days prod when you have compliance requirements.
• Add anomaly detection on request count and latency (needs 2 weeks of baseline data).

Scale ($10M+ ARR)

• OpenTelemetry for vendor-neutral instrumentation.
• Centralized observability platform decision (CloudWatch vs third-party based on team size and budget).
• This is when the Datadog bill becomes justified ($50K+/year but saves engineering time at 20+ engineers).

Cost Traps

Counterintuitive Advice

• Fewer alarms = faster response. 3 well-chosen alarms that always require action beat 30 alarms that are “probably fine.” Every alarm should have a runbook. If you can’t write the runbook, delete the alarm.
• Alarm on symptoms, not causes. “Error rate >1%” tells you something is broken. “CPU >80%” tells you… maybe nothing is wrong. Users feel symptoms, not causes.
• CloudWatch Logs Insights is underrated. Startups reach for Elasticsearch/Datadog because they assume CloudWatch is limited. Logs Insights with structured JSON logging handles 95% of debugging queries for teams under 20 engineers.
• Embedded Metric Format is the single most important observability feature for Lambda-based startups. It turns log lines into metrics at zero additional cost. You get custom metrics for free.
• Don’t build a “golden dashboard” until you’ve been in production for 3 months. You’ll build the wrong dashboard on day 1 because you don’t know what fails yet. Start with the 3 basics and add panels as you learn from incidents.

Minimum Viable Observability

Seed Stage Checklist

• Not done: Structured JSON logging (enables Logs Insights queries)
• Not done: Log retention set on all log groups (7d dev, 30d prod)
• Not done: 3 alarms: error rate, p99 latency, 5xx count (SNS → email/Slack)
• Not done: 1 dashboard: request count, error rate, latency p99
• Not done: Lambda: use Embedded Metric Format for custom metrics (free)
• Not done: CloudTrail enabled (default trail — free for management events)

Post-PMF Additions

• Not done: X-Ray sampling on API Gateway + Lambda (1/sec + 5%)
• Not done: Anomaly detection on request count (catches traffic drops = outage)
• Not done: DLQ monitoring alarms (ApproximateNumberOfMessagesVisible > 0)
• Not done: Composite alarm for service health (error rate AND latency breach)
• Not done: Log retention 90 days in prod

When to Graduate

Credits Consideration

CloudWatch costs (logs, metrics, dashboards, X-Ray) are covered by AWS Activate credits. While you have credits:

• Don’t optimize log retention aggressively — keep 90 days everywhere for debugging convenience
• Use X-Ray at higher sampling rates to build intuition about your system
• Create dashboards freely for learning

Set a calendar reminder 6 months before credits expire to implement cost-optimized retention policies.