> **service-mesh-observability** — chapter 38 of 180 in [wshobson/agents](https://skillsdocs.com/wshobson/agents).
>
> Book (all chapters, one file): https://skillsdocs.com/wshobson/agents.md
> Machine manifest: https://skillsdocs.com/wshobson/agents/.well-known/agent-skills/index.json
> Install the book: `npx skills add wshobson/agents`
> Upstream: https://github.com/wshobson/agents/blob/main/plugins/cloud-infrastructure/skills/service-mesh-observability/SKILL.md @ `main`
> Raw bytes, no header: https://raw.githubusercontent.com/wshobson/agents/main/plugins/cloud-infrastructure/skills/service-mesh-observability/SKILL.md
> Base for relative paths: https://raw.githubusercontent.com/wshobson/agents/main/plugins/cloud-infrastructure/skills/service-mesh-observability/
> Licence: MIT — https://spdx.org/licenses/MIT.html
>
> Bundled files (1), referenced from this skill's directory:
>   - `references/details.md` — https://raw.githubusercontent.com/wshobson/agents/main/plugins/cloud-infrastructure/skills/service-mesh-observability/references/details.md
>
> Content © its authors, served unmodified. Takedown: https://github.com/kyleledbetter/skillsdocs/issues/new?labels=takedown&title=Takedown+request

<!-- Verbatim upstream SKILL.md follows, YAML frontmatter included. -->

---
name: service-mesh-observability
description: Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SLOs for service communication.
---

# Service Mesh Observability

Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.

## When to Use This Skill

- Setting up distributed tracing across services
- Implementing service mesh metrics and dashboards
- Debugging latency and error issues
- Defining SLOs for service communication
- Visualizing service dependencies
- Troubleshooting mesh connectivity

## Core Concepts

### 1. Three Pillars of Observability

```
┌─────────────────────────────────────────────────────┐
│                  Observability                       │
├─────────────────┬─────────────────┬─────────────────┤
│     Metrics     │     Traces      │      Logs       │
│                 │                 │                 │
│ • Request rate  │ • Span context  │ • Access logs   │
│ • Error rate    │ • Latency       │ • Error details │
│ • Latency P50   │ • Dependencies  │ • Debug info    │
│ • Saturation    │ • Bottlenecks   │ • Audit trail   │
└─────────────────┴─────────────────┴─────────────────┘
```

### 2. Golden Signals for Mesh

| Signal         | Description               | Alert Threshold   |
| -------------- | ------------------------- | ----------------- |
| **Latency**    | Request duration P50, P99 | P99 > 500ms       |
| **Traffic**    | Requests per second       | Anomaly detection |
| **Errors**     | 5xx error rate            | > 1%              |
| **Saturation** | Resource utilization      | > 80%             |

## Templates and detailed worked examples

Full template library and detailed worked examples live in `references/details.md`. Read that file when you need the concrete templates.

## Best Practices

### Do's

- **Sample appropriately** - 100% in dev, 1-10% in prod
- **Use trace context** - Propagate headers consistently
- **Set up alerts** - For golden signals
- **Correlate metrics/traces** - Use exemplars
- **Retain strategically** - Hot/cold storage tiers

### Don'ts

- **Don't over-sample** - Storage costs add up
- **Don't ignore cardinality** - Limit label values
- **Don't skip dashboards** - Visualize dependencies
- **Don't forget costs** - Monitor observability costs