Skip to main content
skill CX Codex

monitoring-observability · yonatangross — Claude Code skill: how to install

monitoring-observability — a skill for Claude Code that configures Prometheus metrics, Grafana dashboards, and Langfuse v4 tracing. Activates when adding logging, distributed tracing, or LLM cost tracking.

MIT 270 installs ★ 305 updated 2026-11-21

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code
Open on GitHub

What monitoring-observability does

Provides monitoring configurations for backend services and LLM workflows. It implements the RED method for Prometheus counters and histograms, Grafana Golden Signals dashboards, and Langfuse tracing with the @observe decorator.

The rules detect silent agent failures. They catch skipped tool calls, output degradation using z-score baselines, and execution loops that create token spikes. Tracing relies on OpenTelemetry and the Langfuse Python SDK.

Requires the Python package langfuse version 4.16.0 or higher and Claude Code 2.1.277+. In Claude Code 2.1.169+, OTEL client certificate paths in project settings require a trust confirmation prompt, otherwise telemetry stops silently.

What it helps with

  • instrument Python endpoints with Prometheus RED metrics
  • configure Langfuse v4 tracing with inline score tracking
  • detect agent execution loops and sudden token spikes
  • track skipped tool calls and output quality degradation

Example prompts for the agent

  • Add Langfuse v4 tracing and quality scoring to this LLM call
  • Set up Prometheus RED metrics for incoming HTTP requests
  • Configure alerts for agent loops and token consumption spikes

How to install monitoring-observability

1Skills CLI (Vercel)

npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code

2Manually: put the files in place

~/.claude/skills/monitoring-observability/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/monitoring-observability/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about monitoring-observability

How to install monitoring-observability in Claude Code?

Run npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code or place the rule file at ~/.claude/skills/monitoring-observability/SKILL.md.

What does monitoring-observability do?

It provides implementation rules for Prometheus metrics, Grafana dashboards, and Langfuse tracing, focusing on silent failure detection in agents.

What are the requirements for monitoring-observability?

It targets the Python langfuse package version 4.16.0 or newer and requires Claude Code version 2.1.277 or higher.

Source text

SKILL.md · MIT
---
name: monitoring-observability
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing, and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
user-invocable: false
targets:
  # Python SDK. The JS/TS SDK is a separate line at 5.x (@langfuse/* packages) — see
  # references/langfuse-js-v5.md. The self-hosted platform is a third axis (v3) and is not an
  # SDK version.
  - library: langfuse
    version: ">=4.16.0"
upstream-version-tested: "4.16.0"
metadata:
  archive-candidate: "pending an outside usage signal"
  category: document-asset-creation
  version: "3.0.0"
  author: "OrchestKit"
  complexity: "medium"
  tags: "monitoring, observability, prometheus, grafana, langfuse, tracing, metrics, drift-detection, logging"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
path_patterns: ["**/metrics/**", "**/tracing/**", "prometheus.*", "grafana/**"]
---

# Monitoring & Observability

A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at `references/ork-delta.md`.

## Upstream coverage (do not restate)

These topics are fully covered first-party. Read the source, do not add a local copy.

| Topic | First-party source |
|-------|--------------------|
| Prometheus metric types, RED method, cardinality, PromQL | <https://prometheus.io/docs/practices/> |
| Alertmanager grouping, inhibition, escalation, runbooks | <https://prometheus.io/docs/alerting/latest/configuration/> |
| Grafana dashboards, Loki and LogQL, Promtail | <https://grafana.com/docs/> |
| OpenTelemetry spans, sampling, context propagation | <https://opentelemetry.io/docs/> |
| Langfuse Python SDK (`@observe`, `as_type`, `score_current_span`, `should_export_span`, `LangfuseMedia`) | <https://langfuse.com/docs/sdk/python> |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | <https://langfuse.com/docs/sdk/python/v4-migration> |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | <https://langfuse.com/docs/deployment/self-host> |
| Langfuse cost tracking, model pricing, Metrics API v2 | <https://langfuse.com/docs/model-usage-and-cost> |
| Langfuse scores, online evaluators, annotation queues, prompt management | <https://langfuse.com/docs/scores/overview> |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | <https://langfuse.com/docs/integrations> |
| Agent Graphs, observation types, rendered tool calls | <https://langfuse.com/docs/tracing-features/agent-graphs> |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | <https://www.evidentlyai.com/blog/data-drift-detection-large-datasets> |
| EWMA control charts | <https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm> |
| structlog, Winston, correlation IDs, log sampling | <https://www.structlog.org/en/stable/> |

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Infrastructure Monitoring](#infrastructure-monitoring) | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
| [LLM Observability](#llm-observability) | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
| [Silent Failures](#silent-failures) | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |

**Total: 5 rules across 3 categories.** Drift detection, cost tracking, eval scoring, Prometheus
instrumentation and alert-rule authoring moved to the upstream sources listed above.

## Quick Start

```python
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client

@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
    get_client().update_current_trace(
        user_id="user_123", session_id="session_abc",
        tags=["production", "orchestkit"],
    )
    result = await llm.generate(content)
    get_client().score_current_span(name="response_quality", value=0.85)
    return result
```

```python
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram

http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
    buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
```

## Infrastructure Monitoring

Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.

| Rule | File | Key Pattern |
|------|------|-------------|
| Grafana Dashboards | `rules/monitoring-grafana.md` | Golden Signals, SLO/SLI, health checks |

> **CC 2.1.161 — OTEL resource attributes as metric labels:** `OTEL_RESOURCE_ATTRIBUTES` values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.

## LLM Observability

Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are
upstream; what stays here is how this repo wires traces.

| Rule | File | Key Pattern |
|------|------|-------------|
| Langfuse Traces | `rules/llm-langfuse-traces.md` | @observe decorator, OTEL spans, agent graphs |

## Silent Failures

Detection and alerting for silent failures in LLM agents.

| Rule | File | Key Pattern |
|------|------|-------------|
| Tool Skipping | `rules/silent-tool-skipping.md` | Expected vs actual tool calls, Langfuse traces |
| Quality Degradation | `rules/silent-degraded-quality.md` | Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting | `rules/silent-alerting.md` | Loop detection, token spikes, escalation workflow |

> **CC 2.1.169 — OTEL client-cert paths require trust:** untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project `.claude/settings.json`, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.

## Key Decisions

| Decision | Recommendation | Rationale |
|----------|----------------|-----------|
| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
| Log format | Structured JSON | Machine-parseable, supports log aggregation |
| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
| LLM tracing API | `@observe(as_type=...)` + `score_current_span()` | v4: semantic types, inline scoring, span filtering |
| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport | JSONL under `~/.claude/analytics/`, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (`references/ork-delta.md`) |

## Detailed Documentation

| Resource | Description |
|----------|-------------|
| `references/ork-delta.md` | **Start here.** Floors, house decisions and scars that upstream docs do not carry |
| `references/langfuse-js-v5.md` | **JS/TS SDK v5** delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
| `references/experiments-api.md` | Langfuse experiments and dataset runs as this repo uses them |
| `references/evaluation-scores.md` | Score shapes and scoring pipeline wiring |
| `references/session-tracking.md` | Session and user grouping across multi-step workflows |
| `references/metrics-collection.md` | Claude Code OTEL metric inventory and collector-side joins |
| `references/dashboards.md` | Dashboard layout conventions |
| `references/structured-logging.md` | Structured log field conventions |
| `references/dev-agent-lens.md` | LiteLLM proxy layer (or Bifrost) for API-boundary observability |
| `examples/orchestkit-monitoring-dashboard.md` | Worked monitoring dashboard example |
| `scripts/` | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |

## Related Skills

- `defense-in-depth` - Layer 8 observability as part of security architecture
- `devops-deployment` - Observability integration with CI/CD and Kubernetes
- `resilience-patterns` - Monitoring circuit breakers and failure scenarios
- `llm-evaluation` - Evaluation patterns that integrate with Langfuse scoring
- `caching` - Caching strategies that reduce costs tracked by Langfuse

© OrchestKit · MIT · yonatangross/orchestkit

Similar skills

All skills for DevOps and infrastructure →

monitoring-observability by other authors comes first.

monitoring-observability ahmedasmar monitoring-observability — a skill for building monitoring and observability: metrics strategy, alerts, dashboards, SLOs and incident troubleshooting. CX Codex +1 504installs monitoring-observability akillness monitoring-observability turns service, data-pipeline, or live-ops context into a monitoring brief with a primary mode, useful signals, and a small first implementation slice. CX Codex +1 371installs monitoring-observability travisjneuman monitoring-observability equips AI coding agents to set up OpenTelemetry, structured JSON logging, and distributed tracing. Useful when defining SLO alerts or instrumenting services. CX Codex +1 146installs monitoring-observability rohitg00 monitoring-observability is a Claude Code skill that adds tracing, metrics, structured logs, and alert rules to Node.js services. It connects application signals to dashboards and SLOs. CX Codex +1 ★ 2,858repo stars setup-monitoring-observability davila7 The setup-monitoring-observability command inspects a project and guides setup for metrics, logs, tracing, and alerts. Choose one area or request the full observability stack. CX Codex +1 279installs logging-observability curiositech logging-observability sets up structured logging, OpenTelemetry tracing, and alerting thresholds for production backends. Connects distributed services with correlation IDs. CX Codex +1 152installs observability-monitoring manutej The observability-monitoring skill covers Prometheus and Grafana monitoring, from metric collection and PromQL to alerts and dashboards for distributed services. CX Codex +1 458installs monitoring-expert https://github.com/Jeffallan monitoring-expert — a skill that walks your agent through setting up metrics, structured logs, tracing and alerts. Useful when production is slow and you need to find out why. CX Codex +1 6,200installs implementing-observability ancoleman The implementing-observability skill wires metrics, logs and traces through OpenTelemetry and shows how to run Loki, Grafana, Tempo and Mimir yourself. Useful once one request crosses several services. CX Codex +1 1installs monitoring-specialist davila7 monitoring-specialist is a Claude Code subagent for monitoring and observability work. It handles requests about metrics, alerting, log aggregation, and distributed tracing. CX Codex 1,677installs

From the same set86

The whole set →
security-patterns security-patterns is a Claude Code skill for adding authentication and security layers. It activates around input validation, LLM prompt-injection defense, and PII masking. CX Codex +1 739installs langgraph Skill for Claude Code supplies workflow patterns for LangGraph 1.x in Python. It steps in when designing multi-agent graphs, state schemas, and fault tolerance policies. CX Codex +1 377installs write-prd write-prd generates 8-section product requirements documents with INVEST user stories, acceptance criteria, and go/no-go gates. Writes the final specification to PRD-[product-name].md. CX Codex +1 336installs distributed-systems distributed-systems is a Claude Code skill with architectural guidelines for distributed locks, circuit breakers, idempotency keys, and rate limiting. CX Codex +1 297installs vite-advanced vite-advanced is a Claude Code and Codex skill for Vite 8 builds with Rolldown, chunk splitting, and Environment API. It activates when editing vite.config.* or files under **/vite/**. CX Codex +1 265installs quality-gates quality-gates scores task complexity on a 1–5 scale and halts work when requirements are vague or solutions loop. Triggers before complex tasks and after repeated failures. CX Codex +1 259installs code-review-playbook code-review-playbook supplies structured review workflows, conventional comment formats, and security checklists. Triggers during pull request evaluations or author self-reviews. CX Codex +1 254installs python-backend python-backend — a skill for Claude Code and Codex providing async Python patterns when editing *.py or package files. Enforces conventions for FastAPI, SQLAlchemy 2.0, and asyncio. CX Codex +1 250installs

More in the catalog

Project sponsors

Companies that keep this analytics open