Skip to main content
skill CX Codex

prometheus-expert — Claude Code skill: PromQL, alerts, K8s setup

prometheus-expert — a skill that gives your coding agent solid Prometheus knowledge: metric collection, PromQL, recording and alerting rules. Useful when exporters have to go up on Kubernetes or a query just will not return what you expect.

Apache-2.0 204 installs ★ 45 updated 2026-10-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add personamanagmentlayer/pcl --skill prometheus-expert -a claude-code
Open on GitHub

What prometheus-expert does

The skill hands the agent a structure for Prometheus work: collecting metrics, writing PromQL, recording rules, alerting, service discovery. Concrete material comes with it, including a node-exporter DaemonSet, Go client counters and histograms, and a list of anti-patterns such as querying a counter without rate. A rollout order is there too: golden signals first, then SLIs/SLOs, then running more than one Prometheus. Longer notes live in references/CORE_CONCEPTS.md and get read on demand.

Bash access is narrowed to kubectl and promtool, so the agent can lint a rule file or look at a live cluster. Versions matter: Prometheus 2.45 and Kubernetes 1.28 or newer. On older setups the ServiceMonitor examples and kubectl calls stop matching reality. Retention numbers in the skill (30d for production, 7d for development) are a starting frame; your thresholds are yours to pick.

What it helps with

  • Ship a node-exporter DaemonSet and pull host metrics
  • Write PromQL with rate and histogram_quantile correctly
  • Set an alert with for: 10m so it stops flapping
  • Strip high-cardinality labels out of recording rules
  • Pick retention and resource limits for Prometheus itself

Example prompts for the agent

  • Deploy the node-exporter DaemonSet into the monitoring namespace and list the metrics it exposes
  • Write an alert on p95 latency with histogram_quantile and add for: 10m
  • Find high-cardinality labels in our recording rules and suggest replacements

How to install prometheus-expert

1Skills CLI (Vercel)

npx skills add personamanagmentlayer/pcl --skill prometheus-expert -a claude-code

2Manually: put the files in place

~/.claude/skills/prometheus-expert/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/prometheus-expert/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about prometheus-expert

How do I install prometheus-expert in Claude Code?

Run npx skills add personamanagmentlayer/pcl --skill prometheus-expert -a claude-code. The skill file lands at ~/.claude/skills/prometheus-expert/SKILL.md.

What does prometheus-expert do?

It gives the agent working knowledge of Prometheus: metric collection, PromQL, recording and alerting rules, Kubernetes installation. Manifests and common mistakes are included as examples.

What are the requirements?

Prometheus 2.45 and Kubernetes 1.28 or newer. The agent may run kubectl and promtool and nothing else from the shell.

Source text

SKILL.md · Apache-2.0
---
name: prometheus-expert
version: 1.1.0
description: >-
  Expert-level Prometheus monitoring, metrics collection, PromQL queries, alerting, and
  production operations. Use when the user mentions monitoring, metrics, observability,
  alerting, or PromQL, or when the task involves Prometheus Architecture, Installation on
  Kubernetes, ServiceMonitor, or PromQL Queries.
category: devops
author: PCL Team
license: Apache-2.0
tags:
  - prometheus
  - monitoring
  - metrics
  - observability
  - alerting
  - promql
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash(kubectl:*, promtool:*)
  - Glob
  - Grep
requirements:
  prometheus: '>=2.45'
  kubernetes: '>=1.28'
---

# Prometheus Expert

You are an expert in Prometheus with deep knowledge of metrics collection, PromQL queries, recording rules, alerting rules, service discovery, and production operations. You design and manage comprehensive observability systems following monitoring best practices.

## Exporters

**Node Exporter (Infrastructure Metrics):**

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-exporter
          image: prom/node-exporter:latest
          ports:
            - containerPort: 9100
              name: metrics
          args:
            - --path.procfs=/host/proc
            - --path.sysfs=/host/sys
            - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: sys
              mountPath: /host/sys
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
```

**Custom Application Metrics (Go):**

```go
package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )

    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestsTotal)
    prometheus.MustRegister(httpRequestDuration)
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":9090", nil)
}
```

## Best Practices

### 1. Use Recording Rules for Complex Queries

```yaml
# Pre-compute expensive queries
- record: api:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (job)
```

### 2. Label Cardinality

```promql
# AVOID: High cardinality labels
http_requests_total{user_id="123"}  # BAD

# USE: Low cardinality labels
http_requests_total{endpoint="/api/users"}  # GOOD
```

### 3. Appropriate Retention

```yaml
# Balance storage vs history
retention: 30d  # Production
retention: 7d   # Development
```

### 4. Alert Fatigue Prevention

```yaml
# Use appropriate thresholds and durations
for: 10m # Avoid flapping
```

### 5. Use Histograms for Latency

```promql
# Better than average
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
```

## Anti-Patterns

**1. Missing Rate Function:**

```promql
# BAD: Raw counter
http_requests_total

# GOOD: Use rate
rate(http_requests_total[5m])
```

**2. Too Many Labels:**

```promql
# BAD: Unique labels per request
{request_id="abc123"}

# GOOD: Aggregate labels
{endpoint="/api/users"}
```

**3. No Resource Limits:**

```yaml
# GOOD: Set limits
resources:
  limits:
    memory: 4Gi
    cpu: 2
```

## Approach

When implementing Prometheus monitoring:

1. **Start with Golden Signals**: Latency, Traffic, Errors, Saturation
2. **Define SLIs/SLOs**: Service Level Indicators and Objectives
3. **Implement Recording Rules**: Pre-compute complex queries
4. **Set Up Alerting**: Alert on symptoms, not causes
5. **Monitor Prometheus**: Prometheus monitoring itself
6. **Retention Strategy**: Balance storage and history
7. **High Availability**: Run multiple Prometheus instances

Always design monitoring that is actionable, reliable, and maintainable.

## Reference Documentation

Detailed material lives alongside this skill and is read on demand:

- [Core Expertise](references/CORE_CONCEPTS.md) — Prometheus Architecture, Installation on Kubernetes, ServiceMonitor (Prometheus Operator), PromQL Queries, Recording Rules, Alerting Rules, Alertmanager Configuration

## Resources

- Prometheus Documentation: https://prometheus.io/docs/
- PromQL Guide: https://prometheus.io/docs/prometheus/latest/querying/basics/
- Prometheus Operator: https://prometheus-operator.dev/
- Best Practices: https://prometheus.io/docs/practices/

© PCL Team · Apache-2.0 · personamanagmentlayer/pcl

Similar skills

All skills for DevOps and infrastructure →

From the same set104

The whole set →
devops-expert devops-expert puts working CI/CD, infrastructure-as-code and release-strategy patterns in front of the agent whenever a task touches pipelines, automation or DevOps culture. CX Codex +1 210installs testing-expert testing-expert — a skill that gives the agent rules for writing tests: AAA structure, isolated cases, doubles and edge cases. TDD and BDD theory sits in a reference file the agent opens only when needed. CX Codex +1 206installs sre-expert sre-expert guides site reliability engineering practices, incident workflows, and error budget calculations. CX Codex +1 205installs elasticsearch-expert elasticsearch-expert is a Claude Code skill that helps design Elasticsearch mappings, build search queries and aggregations, and optimize ELK stacks. CX Codex +1 205installs kotlin-expert The kotlin-expert skill guides Kotlin work across Android, coroutines, JVM, and Kotlin Multiplatform. Use it when an agent needs help with Kotlin syntax, architecture, or asynchronous code. CX Codex +1 203installs react-expert react-expert configures coding agents for React 18+ and Next.js, enforcing clean TypeScript interfaces, proper hook usage, and tests with React Testing Library. CX Codex +1 200installs performance-expert performance-expert guides code profiling, database query tuning, and caching strategies to resolve latency and throughput issues. CX Codex +1 198installs javascript-expert javascript-expert — a skill that makes the assistant work as a seasoned JavaScript developer across ES2024+, Node.js and npm, so the code it writes stays in one style. CX Codex +1 197installs

Project sponsors

Companies that keep this analytics open