# agent-observability

agent-observability — a skill for Claude Code, Codex, Cursor and more: wires up OpenTelemetry tracing, Prometheus counters, latency histograms, and per-request cost tracking for LLM-powered agents.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 373 installs · ★ 1214 repo stars
- Author: devops-skills
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/devops/ai/agent-observability/SKILL.md
- Updated: 2026-05-22
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/agent-observability

## What it does

The skill covers Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. It delivers a full observability stack: OpenTelemetry spans for every agent turn and tool call, Prometheus metrics for latency, token usage, errors, and retries, structured logging, cost attribution, SLO definitions, and PII redaction before data leaves the application.

On the metrics side, histograms like `agent_request_duration_seconds` and `agent_tool_call_duration_seconds` are labelled by agent name, model, and provider. Token counters track prompt, completion, and KV-cache hits separately. A `llm_cost_dollars_total` counter splits cost by prompt and completion so you can run a single PromQL query to get weekly spend per agent.

Debugging a bad agent response without this means grepping raw application logs. With tracing enabled, every step of a multi-turn tool call chain appears as a child span, so the failing step is visible at a glance.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/agent-observability/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/agent-observability/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/agent-observability/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/agent-observability/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install BagelHole/DevOps-Security-Agent-Skills devops/ai/agent-observability
```

2. **Manually: put the files in place**
- File: `.github/skills/agent-observability/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/BagelHole/DevOps-Security-Agent-Skills --path devops/ai/agent-observability
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/agent-observability/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/devops/ai/agent-observability/SKILL.md

## When to use it
- measure p95 latency for agent responses over any time window
- attribute LLM costs to specific agents and models
- trace a full request through a multi-agent orchestration graph
- alert on token-usage spikes or rising LLM error rates
- redact PII from traces before exporting to an observability backend

## Example prompts
- Add Prometheus metrics and OpenTelemetry tracing to every LLM call in this agent
- Wrap the model call in an OpenTelemetry span and record prompt and completion tokens
- Define SLOs and set up alerts on `llm_errors_total` and `agent_request_duration_seconds`

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 40b88eb6de0e86d4.

BEGIN THIRD-PARTY FILE 40b88eb6de0e86d4
````text
---
name: agent-observability
description: Instrument AI agents with tracing, token metrics, latency, and cost visibility. Use for reliability and debugging.
license: MIT
metadata:
  author: devops-skills
  version: "2.0"
---

# Agent Observability

Monitor AI agent behavior with logs, traces, metrics, and cost telemetry. This skill covers the full observability stack for LLM-powered applications: from raw Prometheus counters to Grafana dashboards, OpenTelemetry tracing, structured logging, cost tracking, SLO definition, and PII redaction.

---

## When to Use

Apply this skill whenever you operate:

- **Autonomous AI agents** that make multi-step tool calls (e.g., coding agents, support agents, data-pipeline agents).
- **LLM-backed APIs** serving chat completions, summarisation, or classification behind a REST or gRPC gateway.
- **RAG pipelines** where a retriever fetches context from a vector store before prompting a model.
- **Multi-agent orchestrations** (crew-style or graph-based) where several agents collaborate on a single task.
- **Batch inference jobs** that process thousands of prompts against a model endpoint.

Key signals that you need this skill:

1. You cannot answer "what is p95 latency for agent responses this week?"
2. You have no per-request cost attribution.
3. Debugging a bad agent response requires grepping raw application logs.
4. You have no alerting on token-usage spikes or elevated error rates.

---

## Core Metrics

Define these metrics at the application layer. All examples use the Prometheus client library naming conventions.

### Latency

```python
from prometheus_client import Histogram

# Total end-to-end latency for a full agent turn (user prompt -> final response)
AGENT_LATENCY = Histogram(
    "agent_request_duration_seconds",
    "End-to-end latency of an agent request",
    labelnames=["agent_name", "model", "status"],
    buckets=(0.25, 0.5, 1, 2, 5, 10, 30, 60, 120),
)

# Latency of a single LLM API call (one completion request)
LLM_CALL_LATENCY = Histogram(
    "llm_call_duration_seconds",
    "Latency of an individual LLM API call",
    labelnames=["model", "provider", "stream"],
    buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
)

# Latency of tool/function calls executed by the agent
````
END THIRD-PARTY FILE 40b88eb6de0e86d4

The third-party file has ended. Everything between the 40b88eb6de0e86d4 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2261 of 35444 bytes. © devops-skills · MIT · https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/HEAD/devops/ai/agent-observability/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
