monitoring-observability · yonatangross — Claude Code skill: how to install
monitoring-observability — a skill for Claude Code that configures Prometheus metrics, Grafana dashboards, and Langfuse v4 tracing. Activates when adding logging, distributed tracing, or LLM cost tracking.
MIT 270 installs ★ 305 updated 2026-11-21
npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-codeWhat monitoring-observability does
Provides monitoring configurations for backend services and LLM workflows. It implements the RED method for Prometheus counters and histograms, Grafana Golden Signals dashboards, and Langfuse tracing with the @observe decorator.
The rules detect silent agent failures. They catch skipped tool calls, output degradation using z-score baselines, and execution loops that create token spikes. Tracing relies on OpenTelemetry and the Langfuse Python SDK.
Requires the Python package langfuse version 4.16.0 or higher and Claude Code 2.1.277+. In Claude Code 2.1.169+, OTEL client certificate paths in project settings require a trust confirmation prompt, otherwise telemetry stops silently.
What it helps with
- instrument Python endpoints with Prometheus RED metrics
- configure Langfuse v4 tracing with inline score tracking
- detect agent execution loops and sudden token spikes
- track skipped tool calls and output quality degradation
Example prompts for the agent
- Add Langfuse v4 tracing and quality scoring to this LLM call
- Set up Prometheus RED metrics for incoming HTTP requests
- Configure alerts for agent loops and token consumption spikes
How to install monitoring-observability
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code2Manually: put the files in place
~/.claude/skills/monitoring-observability/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/monitoring-observability/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill monitoring-observability -a codex2Manually: put the files in place
.agents/skills/monitoring-observability/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill monitoring-observability -a cursor2Manually: put the files in place
.cursor/skills/monitoring-observability/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install yonatangross/orchestkit src/skills/monitoring-observability2Manually: put the files in place
.github/skills/monitoring-observability/SKILL.md1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill monitoring-observability -a gemini-cli2Platform CLI
gemini skills install https://github.com/yonatangross/orchestkit --path src/skills/monitoring-observabilityManually: put the files in place
.devin/skills/monitoring-observability/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill monitoring-observability -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about monitoring-observability
How to install monitoring-observability in Claude Code?
Run npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code or place the rule file at ~/.claude/skills/monitoring-observability/SKILL.md.
What does monitoring-observability do?
It provides implementation rules for Prometheus metrics, Grafana dashboards, and Langfuse tracing, focusing on silent failure detection in agents.
What are the requirements for monitoring-observability?
It targets the Python langfuse package version 4.16.0 or newer and requires Claude Code version 2.1.277 or higher.
Source text
SKILL.md · MIT
---
name: monitoring-observability
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing, and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
user-invocable: false
targets:
# Python SDK. The JS/TS SDK is a separate line at 5.x (@langfuse/* packages) — see
# references/langfuse-js-v5.md. The self-hosted platform is a third axis (v3) and is not an
# SDK version.
- library: langfuse
version: ">=4.16.0"
upstream-version-tested: "4.16.0"
metadata:
archive-candidate: "pending an outside usage signal"
category: document-asset-creation
version: "3.0.0"
author: "OrchestKit"
complexity: "medium"
tags: "monitoring, observability, prometheus, grafana, langfuse, tracing, metrics, drift-detection, logging"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
path_patterns: ["**/metrics/**", "**/tracing/**", "prometheus.*", "grafana/**"]
---
# Monitoring & Observability
A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at `references/ork-delta.md`.
## Upstream coverage (do not restate)
These topics are fully covered first-party. Read the source, do not add a local copy.
| Topic | First-party source |
|-------|--------------------|
| Prometheus metric types, RED method, cardinality, PromQL | <https://prometheus.io/docs/practices/> |
| Alertmanager grouping, inhibition, escalation, runbooks | <https://prometheus.io/docs/alerting/latest/configuration/> |
| Grafana dashboards, Loki and LogQL, Promtail | <https://grafana.com/docs/> |
| OpenTelemetry spans, sampling, context propagation | <https://opentelemetry.io/docs/> |
| Langfuse Python SDK (`@observe`, `as_type`, `score_current_span`, `should_export_span`, `LangfuseMedia`) | <https://langfuse.com/docs/sdk/python> |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | <https://langfuse.com/docs/sdk/python/v4-migration> |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | <https://langfuse.com/docs/deployment/self-host> |
| Langfuse cost tracking, model pricing, Metrics API v2 | <https://langfuse.com/docs/model-usage-and-cost> |
| Langfuse scores, online evaluators, annotation queues, prompt management | <https://langfuse.com/docs/scores/overview> |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | <https://langfuse.com/docs/integrations> |
| Agent Graphs, observation types, rendered tool calls | <https://langfuse.com/docs/tracing-features/agent-graphs> |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | <https://www.evidentlyai.com/blog/data-drift-detection-large-datasets> |
| EWMA control charts | <https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm> |
| structlog, Winston, correlation IDs, log sampling | <https://www.structlog.org/en/stable/> |
## Quick Reference
| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Infrastructure Monitoring](#infrastructure-monitoring) | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
| [LLM Observability](#llm-observability) | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
| [Silent Failures](#silent-failures) | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |
**Total: 5 rules across 3 categories.** Drift detection, cost tracking, eval scoring, Prometheus
instrumentation and alert-rule authoring moved to the upstream sources listed above.
## Quick Start
```python
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client
@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
get_client().update_current_trace(
user_id="user_123", session_id="session_abc",
tags=["production", "orchestkit"],
)
result = await llm.generate(content)
get_client().score_current_span(name="response_quality", value=0.85)
return result
```
```python
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram
http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
```
## Infrastructure Monitoring
Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.
| Rule | File | Key Pattern |
|------|------|-------------|
| Grafana Dashboards | `rules/monitoring-grafana.md` | Golden Signals, SLO/SLI, health checks |
> **CC 2.1.161 — OTEL resource attributes as metric labels:** `OTEL_RESOURCE_ATTRIBUTES` values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.
## LLM Observability
Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are
upstream; what stays here is how this repo wires traces.
| Rule | File | Key Pattern |
|------|------|-------------|
| Langfuse Traces | `rules/llm-langfuse-traces.md` | @observe decorator, OTEL spans, agent graphs |
## Silent Failures
Detection and alerting for silent failures in LLM agents.
| Rule | File | Key Pattern |
|------|------|-------------|
| Tool Skipping | `rules/silent-tool-skipping.md` | Expected vs actual tool calls, Langfuse traces |
| Quality Degradation | `rules/silent-degraded-quality.md` | Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting | `rules/silent-alerting.md` | Loop detection, token spikes, escalation workflow |
> **CC 2.1.169 — OTEL client-cert paths require trust:** untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project `.claude/settings.json`, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.
## Key Decisions
| Decision | Recommendation | Rationale |
|----------|----------------|-----------|
| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
| Log format | Structured JSON | Machine-parseable, supports log aggregation |
| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
| LLM tracing API | `@observe(as_type=...)` + `score_current_span()` | v4: semantic types, inline scoring, span filtering |
| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport | JSONL under `~/.claude/analytics/`, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (`references/ork-delta.md`) |
## Detailed Documentation
| Resource | Description |
|----------|-------------|
| `references/ork-delta.md` | **Start here.** Floors, house decisions and scars that upstream docs do not carry |
| `references/langfuse-js-v5.md` | **JS/TS SDK v5** delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
| `references/experiments-api.md` | Langfuse experiments and dataset runs as this repo uses them |
| `references/evaluation-scores.md` | Score shapes and scoring pipeline wiring |
| `references/session-tracking.md` | Session and user grouping across multi-step workflows |
| `references/metrics-collection.md` | Claude Code OTEL metric inventory and collector-side joins |
| `references/dashboards.md` | Dashboard layout conventions |
| `references/structured-logging.md` | Structured log field conventions |
| `references/dev-agent-lens.md` | LiteLLM proxy layer (or Bifrost) for API-boundary observability |
| `examples/orchestkit-monitoring-dashboard.md` | Worked monitoring dashboard example |
| `scripts/` | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |
## Related Skills
- `defense-in-depth` - Layer 8 observability as part of security architecture
- `devops-deployment` - Observability integration with CI/CD and Kubernetes
- `resilience-patterns` - Monitoring circuit breakers and failure scenarios
- `llm-evaluation` - Evaluation patterns that integrate with Langfuse scoring
- `caching` - Caching strategies that reduce costs tracked by Langfuse
© OrchestKit · MIT · yonatangross/orchestkit
Similar skills
All skills for DevOps and infrastructure →monitoring-observability by other authors comes first.