Observability

Why OpenSearch for observability

• Apache 2.0 license — no per-host or per-GB ingestion tax (unlike Splunk/Datadog).
• OpenTelemetry-native end-to-end. Logs/traces/metrics in one engine.
• PPL (Piped Processing Language) for logs/traces; PromQL for metrics.
• Trace Analytics built-in: service map, latency views, RED metrics computed from traces.
• Alerting plugin with native SNS/Lambda/Slack destinations.
• Cost predictability: cluster cost only; no surprise per-GB ingestion bill.

Observability features are exposed in OpenSearch UI (the newer dashboards experience), not the older OpenSearch Dashboards.

ISM lifecycle (the standard pattern)

Key thresholds

• UltraWarm cost-effective at ≥ ~2.5 TiB hot data
• UltraWarm storage: $0.024/GiB-month
• Cold storage: $0.022/GiB-month, no compute attached
• Per-node shard cap (current values): see sizing.md §Topology defaults.

Sample ISM policy (hot → warm → cold → delete)

ISM gotchas

• ISM jobs run every 5–8 minutes (or 30–48 min on pre-1.3 clusters)
• AWS-specific operations: warm_migration, cold_migration, cold_delete (idempotent — operations continue past timeout)
• open and close ops require ES/OS 7.4+; snapshot op requires 7.7+
• AWS-managed ISM cluster settings are restricted: only plugins.index_state_management.enabled, .history.enabled, and .rollover_alias are user-tunable
• Cold storage is NOT directly queryable — must thaw to UltraWarm before query (minutes-to-hours)
• ISM templates with ism_template.index_patterns apply on index creation; existing indexes need explicit _opendistro/_ism/add/<index> call

Index naming for time-series

ISM rollover is preferred — it manages the date math for you. Configure with min_size: 30gb (search) or min_size: 50gb (logs) and min_index_age: 1d.

Trace Analytics

OpenSearch has built-in Trace Analytics:

• Service map: visualize service-to-service dependencies, latencies, error rates
• RED metrics (Rate, Errors, Duration) per service, computed from traces
• Indexes follow otel-v1-apm-span-* and otel-v1-apm-service-map-*
• Ingest via OpenSearch Ingestion with the OTel processor, or directly via OTel Collector with the OpenSearch exporter

OTel pipeline (OSI)

Alerting

Native Alerting plugin:

• Per-monitor schedule: 1 minute minimum (cron or interval)
• Trigger types: query-based (search hits exceed threshold), aggregation, anomaly detector signal
• Destinations: SNS, Slack, Chime, custom webhook, Microsoft Teams, email
• Notification channels centralize destinations (configure once, reuse across monitors)

Sample monitor

PPL (Piped Processing Language)

PPL is the SQL/Splunk-style query language for logs. Pipe-separated commands.

Examples

PPL operators: where, stats, fields, eval, dedup, sort, head, tail, parse, rename, top.

Replacing Splunk

Migration scoping is anchored on detector / dashboard / pipeline count + complexity classification, not on calendar duration. Wall-clock depends on team size, parallelism, and reuse pace — pacing is the customer’s call, not the skill’s.

The streams that decompose any Splunk replatform:

• Discovery — inventory every SPL query, dashboard, alert, scheduled search, and custom app. The output is a count by category and a first-pass classification (see below). This is mandatory step 1 — without it the rest is a guess.
• Data pipeline migration — forwarders (UF / HF) → OpenSearch Ingestion or Fluent Bit / OTel.
• Query and dashboard rewrite — SPL → PPL or DSL. Classify each detector / saved-search:
  • PPL-translatable (search → stats / where / sort / dedup / fields) — typically the majority. Mechanical mapping; pattern reuse dominates after the first ~10.
  • DSL hand-translation required — correlation searches, multi-search joins, transactions, lookups against external KV stores, complex eventstats — these don’t have a clean PPL form and need rewriting against the Query DSL or restructured against _msearch / aggregations.
• Alert / detector rewrite — onto the Alerting plugin (monitors + triggers + destinations) and Anomaly Detection plugin where applicable; Security Analytics for security-domain detectors.
• Parallel-run validation — both stacks live, side-by-side, until detector parity is confirmed.

When responding to a Splunk replatform prompt: NAME the concrete detector / dashboard / pipeline counts the customer gave you and break them down by classification (PPL-translatable vs DSL hand-port; trivial vs complex; correlation searches as their own bucket). Surface the parallelism lever — “can be compressed by splitting across N engineers” — without declaring a wall-clock. Do NOT produce week / month / sprint estimates for coding effort: a dedicated team will deliver much faster than a generic estimate suggests, and the customer’s own staffing decides the calendar.

Replacing Datadog

Honest assessment:

• Datadog APM is more polished than OpenSearch Trace Analytics. If APM is your main use case, the gap is real.
• For pure logs + metrics + alerting, OpenSearch is competitive at a fraction of Datadog’s cost.
• Scope the rewrite by detector / dashboard / pipeline counts and complexity classification (PPL-translatable vs DSL hand-port), and run the parallel-run validation stream until parity is confirmed. Do not declare a calendar estimate — the universal no-timeline rule applies; pacing is the customer’s call.

Cost optimization at scale

The Kaltura case study

Kaltura achieved 60% cost reduction vs prior observability setup by moving to Amazon OpenSearch Service with aggressive ISM tiering. Key levers:

1. OR1 instances for ingest tier (logs are write-heavy; OR1 is ~40% cheaper for write workloads)
2. Aggressive ISM to UltraWarm at day 7 (or even day 3 for less-queried indexes)
3. Cold storage for compliance retention (logs > 90 days where queries are rare)
4. Single-AZ for non-prod observability — saves replica cost
5. Index-per-time-bucket with ISM rollover to keep shard counts predictable

Instance family selection for log workloads

For log-analytics workloads, default to OR1 (write-heavy log profile) with UltraWarm tiering for >7-day retention. Full instance family list: sizing.md §Instance family selection. Source of truth: supported-instance-types.html (opens in a new tab).

OR1 trade-offs (observability-specific):

• Replica simplicity: replica=1 is enough (S3 provides durability)
• LOSES on cache-miss aggregations and k-NN graphs (RAM-bound)
• Migration to OR1 is irreversible

Refresh interval tuning

For logs, set refresh_interval: 30s or 60s to reduce CPU overhead from frequent segment refreshes. Default 1s is search-app-tuned.

Bulk size for ingest

3–5 MiB per bulk request for general ingest; 10 MiB for OR1.

Replicas during ingest

Set number_of_replicas: 0 during initial bulk load; raise to target after. Halves storage and indexing cost during reindex.

Translog tuning

index.translog.durability:

• request (default): fsync per request — durable, slower ingest
• async: fsync every sync_interval (default 5s) — bigger throughput, seconds-of-data risk on crash

For non-critical observability indexes, async typically gives 2–5× ingest throughput improvement.

Force-merge after rollover

Once an index is rolled over (read-only), force-merge to 1 segment per shard:

Reduces segment count → improves search performance and reduces JVM overhead.

Watermarks for observability clusters

Defaults (also valid for OpenSearch):

• low watermark: 85% — no new shards allocated to this node
• high watermark: 90% — cluster actively relocates shards off this node
• flood_stage: 95% — applies index.blocks.read_only_allow_delete=true on every index

This is THE most common “cluster went read-only at 3am” cause. Set up alerting on FreeStorageSpace < 25 GB or storage usage > 80%.

Logstash with OpenSearch

Important license gotcha: the default Logstash distro has a license check that rejects OpenSearch. Two workarounds:

1. Use the OSS distro of Logstash (Apache 2.0)
2. Use the logstash-output-opensearch plugin

Or skip Logstash entirely and use OpenSearch Ingestion (managed Data Prepper) or Fluent Bit.

Anomaly Detection plugin

Built-in Anomaly Detection plugin runs Random Cut Forest models on time-series streams. Common observability uses:

• Detect anomalies in error rate, request rate, or latency per service
• Drive Alerting monitors based on anomaly score
• Train on 8+ days of historical data; updates incrementally

Common observability gotchas

1. CloudWatch Logs subscription can pipe directly to OSI — handy bridge from CloudWatch to OpenSearch.
2. Slow logs to CloudWatch are billable — turn them on selectively, not on all indexes.
3. AOS automated snapshots are kept 14 days — don’t rely on them as backup. Manual snapshots bill against your S3 bucket.
4. Cross-AZ data transfer within the cluster is free; transfer between your VPC and AOS endpoint is billed normally.
5. Master node sizing: master nodes scale with cluster size. OS 2.17+: 8 GiB master = up to 30 nodes/15K shards; 32 GiB = 120 nodes/60K shards.
6. Dashboards multi-tenancy is enabled by FGAC — supports private and shared tenants.