# exploring-mcp-tool-quality

exploring-mcp-tool-quality reads PostHog MCP analytics to rank tools by failure share, latency and reach, then drills into the ones that break.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 202 installs · ★ 101 repo stars
- Author: PostHog
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/exploring-mcp-tool-quality/SKILL.md
- Updated: 2026-10-24
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/exploring-mcp-tool-quality

## What it does

Each call made by an MCP server that runs PostHog's analytics SDK shows up as a `$mcp_tool_call` event in the shared `events` table. There is no separate table, so the `$mcp_*` fields live as properties on that event and every metric is an aggregation over it. The skill holds the query shapes for those metrics.

For one tool it uses the typed calls. `posthog:query-mcp-tool-stats` takes a `toolName` and `dateRange` and returns calls, errors, p50/p95, users and sessions. `posthog:query-mcp-tool-failures` buckets error messages by harness. Ranking tools side by side has no typed equivalent, so that part goes through HogQL in `posthog:execute-sql`. Before quoting any failure-rate headline the skill looks for the governed metric `mcp_tool_call_fail_pct` in `posthog:metric-list` and runs it with `posthog:data-catalog-metric-run` once it is approved.

Hand-written queries break on two details. Grouping by the raw tool name hides real tools under the single-exec wrapper, so use the effective name from `$mcp_exec_tool_call_name`. The properties are strings, which means `$mcp_is_error` needs `toBool`. Leave out the time range and the query scans all of `events`; leave out a volume floor and a tool with three calls at 100% errors climbs to the top.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/exploring-mcp-tool-quality/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/exploring-mcp-tool-quality/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/exploring-mcp-tool-quality/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/exploring-mcp-tool-quality/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install PostHog/ai-plugin skills/exploring-mcp-tool-quality
```

2. **Manually: put the files in place**
- File: `.github/skills/exploring-mcp-tool-quality/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/PostHog/ai-plugin --path skills/exploring-mcp-tool-quality
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/exploring-mcp-tool-quality/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/exploring-mcp-tool-quality/SKILL.md

## When to use it
- rank MCP tools by error share without small-sample noise
- pull p50 and p95 latency for a single tool
- group failures by error type to see what actually breaks
- weigh call volume against failure rate before blaming a tool
- get a day-by-day trend for one tool's errors

## Example prompts
- Which MCP tool has the worst error rate over the last 30 days?
- Break down failures for `<tool>`: error types and the messages behind them.
- Build the tool-quality matrix: error share, p50, p95 and reach.

## Similar
- [exploring-llm-clusters](https://affpapa.org/en/nevibecode/skills/exploring-llm-clusters.md) — skill · 436 installs
- [exploring-llm-costs](https://affpapa.org/en/nevibecode/skills/exploring-llm-costs.md) — skill · 416 installs
- [analyze-data-quality](https://affpapa.org/en/nevibecode/skills/analyze-data-quality.md) — skill · 9 installs
- [firetiger](https://affpapa.org/en/nevibecode/skills/firetiger.md) — skill · 827 installs
- [posthog-analytics](https://affpapa.org/en/nevibecode/skills/posthog-analytics.md) — skill · 827 installs
- [adding-analytics](https://affpapa.org/en/nevibecode/skills/adding-analytics.md) — skill · 410 installs

From the same set: [posthog/ai-plugin](https://affpapa.org/en/nevibecode/by/posthog/ai-plugin.md) — 82 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
