Skip to main content
skill CX Codex

performance-rca — Claude Code skill: profiling and setup

The performance-rca Claude Code skill investigates performance issues through Grafana Pyroscope profiles. It runs when CPU use spikes, memory grows, GC pressure rises, or performance regresses after a deployment.

Apache-2.0 464 installs ★ 316 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add grafana/skills --skill performance-rca -a claude-code
Open on GitHub

What performance-rca does

The skill compares profiles from affected and baseline workloads, identifies application stacks that changed, and tests possible causes against symptoms and deployment evidence. It covers CPU use, memory growth, garbage collection pressure, and lock contention.

Queries require gcx v1.3.0 or later and a context with access to a Grafana Pyroscope datasource. The investigation is read-only. go tool pprof is optional for profile comparisons. If a suitable baseline is missing, the skill can still attribute current costs but does not treat a regression as established.

It separates empty results from unsupported features and query errors. Missing measurements do not count as evidence that a service is healthy.

What it helps with

  • trace high CPU use to an application stack
  • compare profiles from before and after a deployment
  • investigate memory growth and GC pressure
  • test a lock-contention hypothesis against profile evidence

Example prompts for the agent

  • Investigate this service's CPU spike using its Grafana Pyroscope profiles.
  • Compare the profiles from before and after the deployment and find changed application stacks.
  • Check whether the memory growth points to GC pressure or a leak.

How to install performance-rca

1Skills CLI (Vercel)

npx skills add grafana/skills --skill performance-rca -a claude-code

2Manually: put the files in place

~/.claude/skills/performance-rca/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/performance-rca/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about performance-rca

How do I install performance-rca in Claude Code?

Run npx skills add grafana/skills --skill performance-rca -a claude-code. The skill is placed at ~/.claude/skills/performance-rca/SKILL.md.

What does performance-rca do?

It compares Grafana Pyroscope profiles for affected and baseline workloads, then traces changed application stacks. It tests root-cause hypotheses against symptoms and change evidence.

What does performance-rca need to run?

It requires gcx v1.3.0 or later and a context that can query a Grafana Pyroscope datasource. go tool pprof is optional and used for profile diffs.

Source text

SKILL.md · Apache-2.0
---
name: performance-rca
license: Apache-2.0
compatibility: >
  Requires gcx v1.3.0 or later (https://github.com/grafana/gcx) with a context
  that can query a Grafana Pyroscope datasource. Optional go tool pprof for
  profile diffs.
description: >
  Investigates CPU, memory, and contention problems with Grafana Pyroscope
  profiles via the gcx CLI. Compares affected and baseline workloads, locates
  changed application stacks, and tests root-cause hypotheses against symptom
  and change evidence. Use when a service shows high CPU, growing memory, GC
  pressure, lock contention, or hot functions, when code regressed after a
  deploy, or when given a suspect service, flame graph, trace, or profiling
  anomaly report. For profiling setup use pyroscope; for profilecli-based
  analysis use profilecli-insights.
---

# Performance RCA

Explain the observed performance problem: which workload changed, which code
path accounts for it, and what evidence supports the mechanism. A hotspot is a
location; an anomaly is a lead; neither alone proves a root cause. Stop at the
strongest supported conclusion, including an inconclusive result. Code changes
and proactive optimization are separate tasks.

## Investigation contract

1. Follow Sections 1–5 in order. Reuse verified evidence from an earlier step
   or referring investigation; do not repeat discovery merely to satisfy the
   sequence. Establish each step's prerequisites before interpreting its result.
2. Fix the context, datasource, UTC windows, selectors, and profile type before
   comparison. Discover missing identifiers; never invent them. Record any
   justified change to the scope or comparison.
3. Qualify the control in Section 2 before claiming a regression. Without one,
   continue only to current cost attribution and label the comparison missing.
4. Distinguish empty results, unsupported capabilities, and query failures.
   Apply Section 1's exits to the affected profiling path; use its documented
   fallbacks only when their prerequisites hold.
5. Select output formats explicitly: JSON for discovery/metrics, table to rank
   functions, DOT to trace callers and callees, and pprof for export. Retain
   diagnostics and limits.
6. Always produce Section 5's report, including after an exit. Fill unsupported
   fields with `unknown`, `not measured`, or `not applicable`, plus a reason.
   A missing measurement must never become evidence that the service is healthy.

## 1. Preserve the question and check usable profiles

Reuse the context, datasource, service, symptom, incident window, deployment
facts, and trace/profile links supplied by the user or a referring skill. Pin
absolute UTC incident and baseline times. Derive a missing incident interval
from timestamped symptom evidence, or ask for it before querying. Carry the same
`--context <context>` on remote commands (omitted below), and pass the
datasource UID explicitly. Keep the investigation read-only; do not switch
contexts or change collection.

```bash
gcx help-tree profiles -o text
```

The examples below need gcx v1.3.0 or later; check with `gcx --version`. If
gcx is missing or older, ask the user to install or upgrade it from
https://github.com/grafana/gcx before continuing. Do not guess command renames
or silently drop time filters. Commands, flags, and output formats can still
be unsupported by the backend; Section 4 and the advanced reference describe
those fallbacks.

```bash
# Discover only if the datasource UID is unknown; choose the target environment.
gcx datasources list -t grafana-pyroscope-datasource -o json
gcx profiles data-range -d <pyro-uid> -o json
gcx profiles list-profile-types -d <pyro-uid> \
  --from <incident-from> --to <incident-to> -o json
```

CLI support does not imply backend support. `data-range` returns tenant-wide
`dataIngested`, `oldestProfileTime`, and `newestProfileTime` (Unix
milliseconds; zero means unknown). These bounds do not prove that the selected
service or lens has continuous coverage. Do not assume a fixed retention
period.

Apply these outcomes before proceeding. Every exit stops the affected profile
queries and produces the Section 5 report; other usable signals may continue
through gcx's bundled `debug-with-grafana` skill
(`gcx agent skills get debug-with-grafana`) with the missing-evidence question
made explicit.

- **EXIT A — profiling unavailable:** no target datasource, missing access, or
  a required CLI capability is unavailable. Record the exact limitation; use
  `gcx login` and `gcx config check` for missing setup or access, or a
  compatible gcx build for a CLI mismatch.
- **EXIT B — no ingestion reported:** a successful stats response says
  `dataIngested: false`. Report the instrumentation/data gap,
  not service health.
- **EXIT C — incident not covered:** the incident has no overlap with known
  queryable bounds. Report both ranges; do not replace the incident with today.
  For partial overlap, analyze only that overlap, record the coverage gap, and
  limit conclusions to the observed interval.
- **EXIT D — no usable incident profiles:** after checking the selector, window,
  and lens against discovered facts, incident samples remain absent or zero. A
  missing baseline alone is handled in Section 2, not by this exit. Report the
  unresolved coverage/instrumentation gap, not proof of zero cost.

**Continue with scoped checks** when stats are absent, unsupported, or have
unknown bounds. Missing stats alone must not block usable type/profile queries.
These queries still need matching samples before they can support attribution.

For query errors, retain the operation and diagnostic. Correct a demonstrated
selector/flag mistake; use a documented fallback for unsupported capabilities.
Do not retry authentication failures or vary selectors blindly. A failed query
is not an empty successful result.

## 2. Choose the lens and a qualified comparison

Copy exact profile type IDs from discovery, including their units. Discover
labels rather than assuming Prometheus jobs, Tempo service names, Kubernetes
workloads, and Pyroscope `service_name` have identical values.

| Symptom | Available lens to look for | Interpretation limit |
| --- | --- | --- |
| CPU high or CPU-bound latency | CPU time | Does not measure all request latency, scheduling delay, or throttling |
| Allocation/GC pressure | Allocated bytes/objects | Churn is not retained heap or a memory leak |
| Growing heap/OOM | In-use bytes/objects, alongside allocations | Live-heap samples are snapshots; not process RSS, native memory, or retention ownership |
| Contention or blocked work | Mutex/block delay and counts | Runtime-specific sampling and attribution; not CPU time |
| Suspected goroutine/thread leak | Goroutine/thread counts and stacks | Growth alone is not proof of leaked work |

If the needed lens is absent, record the instrumentation gap; do not substitute
CPU to rule out waiting or allocations to prove a leak.

A baseline must represent expected behavior, not simply precede the incident.
Prefer equal-duration intervals with comparable request mix/rate, replica count,
collection coverage, runtime, and sampling settings. Diurnal workloads may need
a matching earlier time of day. Check profile types and data in the baseline
window too. Compare an unaffected peer when no historical baseline is suitable.
Without a qualified control, report current cost locations without claiming a
regression. A stable but inefficient hot path may still explain a new symptom
under increased load; an unchanged profile shape does not exonerate it.

For a deployment comparison, keep stable workload labels fixed, but select the
old version/pod for the baseline when the incident's version/pod did not exist.
Record both selectors and why they are comparable. Never compare an absent old
pod/version with a populated new one and call it an infinite regression.

An anomaly result can supply the service, lens, onset, affected labels, and a
candidate baseline. Preserve its link, interval, observed/expected values and
score if supplied; validate them with profiles and symptom evidence. Read
[advanced analysis](references/advanced-analysis.md#anomaly-led-entry) for the
anomaly entry path; it requires no anomaly detector to run this skill.

## 3. Localize the change: service → workload slice

Start with the known suspect, rather than a tenant-wide leaderboard. If the
service is unknown, discover `service_name` values with `profiles labels`, then
rank within the known environment; use a tenant-wide query only if necessary.

```bash
gcx profiles labels -d <pyro-uid> \
  --from <incident-from> --to <incident-to> -o json
gcx profiles labels -d <pyro-uid> --label service_name \
  --from <incident-from> --to <incident-to> -o json
# Establish onset and coverage; repeat for the baseline window.
gcx profiles metrics -d <pyro-uid> '{service_name="<suspect>"}' \
  --profile-type <lens> --from <incident-from> --to <incident-to> -o json
# Pick a discovered label that can distinguish affected/unaffected workloads.
gcx profiles metrics -d <pyro-uid> '{service_name="<suspect>"}' \
  --profile-type <lens> --from <incident-from> --to <incident-to> \
  --top --group-by <label> --limit 20 -o json
```

Use `profiles series` to discover actual label combinations for a service
without requiring a profile type. See
[scoped series discovery](references/advanced-analysis.md#scoped-series-discovery)
for label projection. Tenant-wide label inventories alone do not prove labels
coexist on the suspect.

Repeat the chosen grouping for the baseline. Leaderboards are capped candidates,
not a fleet census; missing groups may have fallen outside the limit. A single
group establishes no peer comparison. Similar totals do not establish uniform
per-instance behavior, and the largest group may simply serve more traffic or
contain more replicas. Use request counts or replica exposure from existing
metrics when needed; if unavailable, state the confounder. Do not use a fixed
ratio threshold to declare a slice anomalous.

Use time-series values to check persistence/onset; `--top` collapses the window.
Keep units and aggregation semantics explicit, especially for live-heap or
count snapshots. Do not sum snapshot values into "bytes allocated" or infer
request-normalized cost from raw profile totals.

## 4. Locate the changed application stack

Run both queries for the incident selector below and for the qualified baseline
selector/window. Omit the slice matcher when analyzing the whole service.

```bash
# Rank functions.
gcx profiles query -d <pyro-uid> \
  '{service_name="<suspect>",<label>="<value>"}' \
  --profile-type <lens> --from <incident-from> --to <incident-to> -o table
# Trace callers and callees of the ranked functions.
gcx profiles query -d <pyro-uid> \
  '{service_name="<suspect>",<label>="<value>"}' \
  --profile-type <lens> --from <incident-from> --to <incident-to> -o dot
```

Take function rankings and shares from the table. It aggregates each function
and lists the top 20 by SELF. Its PERCENTAGE column is SELF divided by the
profile total, not the function TOTAL share. The table omits everything past
the top 20, so to total the share of a set of functions, such as all GC
functions, sum SELF over the full profile from `-o json` or a pprof export.
Names longer than 60 characters are cut and end in `...`; take the full name
from the matching DOT node before reporting or interpreting it.

Use DOT for caller/callee context, source locations, and full function names.
Its nodes can be single source lines (`file.go:line`), so one function may
appear as several nodes. A label splits the function name across lines,
starting with the last element of its package path, then reads
`self (%) of cumulative (%)`, or a single value when they are equal. Do not
rank functions or read self shares from DOT nodes. gcx falls back
to table output with a warning when the backend explicitly rejects DOT as
unsupported or returns a flame graph instead. Other errors still fail the query.
Inspect the returned format and retain stderr. JSON encodes flame-graph wire
data, not the human function table; agent output may instead reference a spill
file that must be read.

DOT is pruned by default; dotted edges can skip intermediate frames. When
caller context is missing, repeat the same query with `-o dot --max-nodes 250`
in both windows. This raises the node limit; it does not guarantee an unpruned
graph. Neither format can rule out a missing path. For ambiguous attribution or
numerical differences, read
[pprof comparison](references/advanced-analysis.md#pprof-comparison) and export
both profiles. Do not assume an exported profile repairs collection gaps.

Interpret the evidence with these constraints:

- Compare absolute cost and share in each window, with units and a denominator.
  Equal shares can hide doubled cost; larger shares can reflect other work
  disappearing. Percentages are not request latency or proof of regression.
- Self/flat means work at that frame; total/cumulative includes descendants.
  Use caller/callee context to locate a meaningful application path. Ancestor
  totals overlap: do not add them or assign runtime work to a guessed caller.
- Runtime symbols suggest mechanisms to test. `mallocgc` suggests allocation
  work; GC workers may run separately from allocation sites; `nanotime` alone
  does not prove busy-waiting. CPU samples in `futex`/locks do not measure time
  asleep. Confirm waiting with a supported wait lens and other evidence.
- In Go, heap stacks identify allocation sites, not the object retaining
  references. Mutex profiles attribute contention to the holder/unlock path;
  block profiles to the waiter. Verify the actual runtime semantics before
  transferring this interpretation to another language or collector.

When a matching checkout/build is available, inspect the implicated function to
validate the mechanism. Read profile locations and verified deployment revision
metadata; do not assume a Build ID is a Git SHA or that `service_repository`
always exists. Report file:line only when symbolization and revision support it;
otherwise retain function-level attribution and say what is missing. Do not
edit source as part of this investigation.

## 5. Test the cause and report

Carry the suspect path back to the original symptom. Use the relevant evidence
through gcx's `debug-with-grafana` skill: request/CPU/heap trends, throttling,
queueing, dependency latency, logs, configuration or deployment changes. Reuse
existing findings; query only what can distinguish the leading explanation from
a material alternative. For a supplied request, see
[trace correlation](references/advanced-analysis.md#trace-correlation).

A deployment timestamp or detector score is correlation. Strengthen a causal
claim with matching onset, a specific mechanism, affected/unaffected controls,
and repeatable differences or an already-observed recovery. Do not perform a
rollback or load test merely to validate the hypothesis.

Stop when the user's question is answered or missing evidence prevents further
attribution. Return a root cause only when the mechanism and symptom connection
are supported; otherwise return a suspected cause, cost location, or
inconclusive result. Never turn "not observed in these profiles" into
"not in this service".

Use this report for every outcome. Keep the field names and order; add details
under Evidence only when needed to support the conclusion. A blocked or empty
investigation is `inconclusive`, with the exit and missing evidence recorded.
Without a qualified baseline, report measured cost locations without regression
claims. Unknown fields remain explicit; do not fill them with guessed values.

```text
Conclusion:     <established cause | suspected cause | cost location |
                 inconclusive> — <answer to the original question>
Exit:           <A | B | C | D | none> — <reason, if applicable>
Scope:          <context, datasource UID, service>
Lens:           <exact profile type and units>
Incident:       <UTC from/to; exact selector; observed coverage>
Baseline:       <UTC from/to; exact selector; or unavailable with reason>
Control:        <why comparable; workload/collection differences or gaps>
Slice:          <affected label/value; evidence of scope or unknown>
Location:       <application stack; file:line and revision only if verified>
Cost:           <incident vs baseline absolute values, units, aggregation>
Share:          <incident vs baseline; self/cumulative and denominator>
Normalization:  <method and measured denominator | not applied, with reason>
Mechanism:      <supported explanation or explicitly labeled hypothesis>
Evidence:       <profile artifacts/links and corroborating symptom evidence>
Counterevidence: <alternatives checked and results | not checked, with reason>
Confidence:     <high | medium | low> — <reason tied to available evidence>
Limitations:    <coverage, sampling, baseline, symbolization or access gaps>
Next action:    <smallest missing evidence or handoff question | none>
```

© grafana · Apache-2.0 · grafana/skills

Similar skills

All skills for DevOps and infrastructure →

From the same set53

The whole set →
audit-and-reduce-dependencies audit-and-reduce-dependencies shrinks JavaScript dependency footprint in pnpm projects and cleans up lockfiles while preserving workspace layouts. Best for trimming node_modules in Grafana plugins. CX Codex +1 3,200installs k6-trend-analysis k6-trend-analysis analyzes Grafana Cloud k6 test runs over time to spot metric drift before thresholds fail. Useful for checking headroom, catching regressions, and reviewing test health. CX Codex +1 3,100installs k6-test-maintenance k6-test-maintenance — a skill for maintaining existing k6 test scripts: tightens thresholds from trend data, migrates scripts to new k6 releases, and fixes tests after service changes. CX Codex +1 3,100installs k6-cloud-investigate-test k6-cloud-investigate-test walks the agent through a 9-step investigation of a Grafana Cloud k6 test or run: script, run history, pass/fail, metrics and logs. CX Codex +1 3,000installs synthetic-monitoring-checks Writes Grafana Cloud Synthetic Monitoring checks: k6 scripted and browser ones. Useful when you need to watch a login or checkout flow from several regions. CX Codex +1 2,700installs profilecli-insights The profilecli-insights skill pulls live profiles from a Pyroscope server with profilecli, reads them through pprof, and ties hot functions back to the code in your checkout. CX Codex +1 2,400installs frontend-session-rca The frontend-session-rca skill diagnoses a single real-user session from Grafana Frontend Observability: what happened, which errors fired, in what order, and what to do about them. CX Codex +1 1,500installs k6-browser-test This skill turns a plain-language user journey into a working k6/browser test. It explores the live site to find usable locators. CX Codex +1 678installs

Project sponsors

Companies that keep this analytics open