Log Analytics Guide

When to use this capability

SKILL.md routes here when the user is doing log analytics or observability on AOS / AOSS. Concrete triggers:

• Phrases: “PPL query”, “OpenSearch Dashboards”, “ingest logs”, “anomaly detection”, “alerting rule”, “Splunk replatform”, “Datadog alternative”, “OSI pipeline”, “log search”, “SIEM”
• Tasks: query logs, set up OSI ingestion, configure ISM tiering / UltraWarm, build dashboards, define alerts, replatform a Splunk/Datadog/ELK stack

All log-analytics files (capability index)

Cross-cutting refs you may also load: observability.md (ISM / UltraWarm / Cold tiering details), security.md, personas.md.

Cross-capability handoff

• For provisioning the OSI pipeline infra (CloudFormation, IAM, source connectors): see provisioning-reference.md.
• For migrating an existing Splunk / Datadog / ELK stack: see assessment-workflow.md (use the Splunk-replatform shape).
• For trace data on the same domain: see trace-analytics-trace-queries.md.

Overview

This guide instructs you on how to perform log analytics against an existing OpenSearch domain or collection. The approach is discovery-first: understand what indices exist, learn the schema, sample the data, then build queries. Do not assume any particular index pattern or field names — discover them.

Data Plane Access with awscurl

Use awscurl for SigV4-authenticated HTTP requests to AOS/AOSS endpoints.

Setup

Environment Variables

Base Commands

AOS (managed domains):

AOSS (serverless collections):

Use --service es for AOS domains, --service aoss for AOSS collections.

Security Considerations

• Verify domain/collection encryption at rest is enabled before ingesting sensitive data
• Use fine-grained access control (FGAC) to restrict index and field-level access
• Do not ingest PII or credentials without field-level encryption or masking
• Apply data retention policies via ISM to comply with regulatory requirements
• Enable CloudTrail logging to audit control plane API calls, and enable OpenSearch audit logs to track data plane operations (queries, indexing) for compliance

Connecting to the Domain/Collection

Determine the domain or collection type and endpoint using the AWS CLI (or call_aws if the AWS MCP server is available):

• If the user names a domain: aws opensearch describe-domain --domain-name <name> → extract Endpoint and ARN (region from ARN). With AWS MCP: call_aws opensearch describe-domain.
• If the user names a collection: aws opensearchserverless batch-get-collection --names <name> → extract collectionEndpoint. With AWS MCP: call_aws opensearchserverless batch-get-collection.
• If unclear: list with aws opensearch list-domain-names or aws opensearchserverless list-collections

This is important because the connection method, authentication, and available features differ between AOS domains and AOSS collections.

Phase 1 — Discover Available Indices

AOSS Note: OpenSearch Serverless does not support _cat APIs. Use --service aoss instead of --service es for all AOSS requests. For index discovery on AOSS, use PPL: source = * | stats count() by index.

Before writing any query, find out what log indices exist on the domain or collection.

List All Indices

Look for indices that suggest logs: names containing log, logs, events, audit, access, syslog, otel, cwl (CloudWatch Logs), or date-based patterns like logs-2024.01.15.

List Index Patterns with Aliases

Check Data Streams

After discovering indices, ask the user which index or index pattern they want to analyze if it’s not obvious. If there are multiple log indices, ask about the relationship between them (e.g., are they daily rollover indices for the same data? different applications? different log levels?).

Phase 2 — Understand the Schema

Once you know the target index pattern, inspect its mapping to learn the available fields.

Get Index Mapping

Via PPL:

Use a concrete index name (e.g., logs-2024.01.15) for describe, not a wildcard pattern.

Identify Key Fields

From the mapping, identify:

1. Timestamp field — usually @timestamp, timestamp, time, or event.created
2. Log level field — level, log.level, severity, severityText, loglevel
3. Message field — message, msg, body, log, event.original
4. Service/source field — service, service.name, host.name, source, kubernetes.pod.name, resource.attributes.service.name
5. Error fields — error.message, error.stack_trace, exception.type
6. Correlation fields — traceId, trace_id, spanId, request_id, correlation_id

If the mapping is large or unclear, ask the user: “I see fields like X, Y, Z — which field contains the log message? Which one is the log level?”

Sample Documents

Always look at a few real documents to understand the actual data shape — mappings alone can be misleading (e.g., dynamic fields, nested objects, multi-value fields):

Via PPL:

Review the sample documents to confirm:

• Which fields are actually populated (vs defined but empty)
• The format of timestamps, log levels, and messages
• Whether the message field is structured JSON or free-text
• Whether there are nested objects that need backtick-quoting in PPL

Phase 3 — Ask Clarifying Questions (If Needed)

If the schema is not self-explanatory, ask the user:

• “What does this index contain? Application logs, access logs, audit logs?”
• “I see multiple log indices (X, Y, Z) — are these from different services or different time periods?”
• “The message field appears to contain JSON — should I parse specific fields from it?”
• “I see a trace_id field — do you want to correlate logs with traces?”
• “What time range are you interested in?”

Do not skip this step if the data is ambiguous. Getting the schema right upfront saves failed queries later.

Phase 4 — Perform Analytics

With the schema understood, build PPL queries using the actual field names discovered above. All examples below use placeholder field names — substitute with the real ones.

Running PPL Queries

For AOSS, use --service aoss instead of --service es.

Log Volume Over Time

Error Count by Service

Error Rate Trend

Recent Errors

Full-Text Search

Top Error Messages

Rare Error Messages

Log Pattern Discovery

Automatically cluster similar log messages:

Error Breakdown by Level and Service

Time-Filtered Queries

Unique Services/Hosts

Latency from Structured Logs

If logs contain a duration/latency field:

Extract Fields from Unstructured Messages

If the message field contains unstructured text, use grok or parse to extract fields:

Caveat: grok processes all matching rows in memory. Add | head N before grok on large indices to avoid resource errors.

Phase 5 — Advanced Analysis

Cross-Index Correlation

If logs span multiple indices (e.g., application logs + access logs), correlate using shared fields like request_id, trace_id, or timestamp proximity:

Step 1 — Find an event of interest in one index:

Step 2 — Look up the same correlation ID in the other index:

Anomaly Detection

Use PPL’s built-in anomaly detection on numeric fields (e.g., log volume, error count):

The ad command auto-detects input fields from the pipeline. It works best on time-series data with regular intervals.

Query DSL for Complex Aggregations

For queries that PPL doesn’t support well (nested aggregations, scripted fields), fall back to Query DSL:

Common Log Schemas Reference

When you encounter these common schemas, use the field mappings below:

Elastic Common Schema (ECS)

Timestamp: @timestamp, Level: log.level, Message: message, Service: service.name, Host: host.name, Error: error.message

OTel Logs (logs-otel-v1-*)

Timestamp: @timestamp, Level: severityText, Message: body, Service: `resource.attributes.service.name` (backtick-quoted), Trace: traceId, Span: spanId

Simple JSON Logs

Timestamp: timestamp or @timestamp, Level: level, Message: message or msg, Service: service, Host: host

Syslog

Timestamp: @timestamp, Level: severity, Message: message, Host: host, Program: program, Facility: facility

Apache/Nginx Access Logs

Client: clientip, Request: request, Status: response, Bytes: bytes, Method: verb, Agent: agent

Key PPL Tips for Log Analytics

• Always backtick-quote dotted field names: `log.level`, `host.name`
• Use head N before memory-intensive commands (grok, streamstats, eventstats)
• Use span(<timestamp>, <interval>) for time bucketing — common intervals: 5m, 15m, 1h, 1d
• Use match() for full-text search, like for wildcard patterns, match_phrase() for exact phrases
• Use patterns for automatic log message clustering
• Use dedup to find unique error messages: dedup <MESSAGE_FIELD> | fields <MESSAGE_FIELD>

Index Management with awscurl

Create Log Index with Mappings

Bulk Index Log Documents

Create ISM Policy for Log Rotation

Create Data Stream for Time-Series Logs