exploring-mcp-tool-quality — Claude Code skill: MCP tool error rates
exploring-mcp-tool-quality reads PostHog MCP analytics to rank tools by failure share, latency and reach, then drills into the ones that break.
no license 202 installs ★ 101 updated 2026-10-24
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-codeWhat exploring-mcp-tool-quality does
Each call made by an MCP server that runs PostHog's analytics SDK shows up as a $mcp_tool_call event in the shared events table. There is no separate table, so the $mcp_* fields live as properties on that event and every metric is an aggregation over it. The skill holds the query shapes for those metrics.
For one tool it uses the typed calls. posthog:query-mcp-tool-stats takes a toolName and dateRange and returns calls, errors, p50/p95, users and sessions. posthog:query-mcp-tool-failures buckets error messages by harness. Ranking tools side by side has no typed equivalent, so that part goes through HogQL in posthog:execute-sql. Before quoting any failure-rate headline the skill looks for the governed metric mcp_tool_call_fail_pct in posthog:metric-list and runs it with posthog:data-catalog-metric-run once it is approved.
Hand-written queries break on two details. Grouping by the raw tool name hides real tools under the single-exec wrapper, so use the effective name from $mcp_exec_tool_call_name. The properties are strings, which means $mcp_is_error needs toBool. Leave out the time range and the query scans all of events; leave out a volume floor and a tool with three calls at 100% errors climbs to the top.
What it helps with
- rank MCP tools by error share without small-sample noise
- pull p50 and p95 latency for a single tool
- group failures by error type to see what actually breaks
- weigh call volume against failure rate before blaming a tool
- get a day-by-day trend for one tool's errors
Example prompts for the agent
- Which MCP tool has the worst error rate over the last 30 days?
- Break down failures for
<tool>: error types and the messages behind them. - Build the tool-quality matrix: error share, p50, p95 and reach.
How to install exploring-mcp-tool-quality
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code2Manually: put the files in place
~/.claude/skills/exploring-mcp-tool-quality/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/exploring-mcp-tool-quality/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a codex2Manually: put the files in place
.agents/skills/exploring-mcp-tool-quality/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a cursor2Manually: put the files in place
.cursor/skills/exploring-mcp-tool-quality/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install PostHog/ai-plugin skills/exploring-mcp-tool-quality2Manually: put the files in place
.github/skills/exploring-mcp-tool-quality/SKILL.md1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a gemini-cli2Platform CLI
gemini skills install https://github.com/PostHog/ai-plugin --path skills/exploring-mcp-tool-qualityManually: put the files in place
.devin/skills/exploring-mcp-tool-quality/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about exploring-mcp-tool-quality
How do I install exploring-mcp-tool-quality in Claude Code?
Run npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code. The skill file lands at ~/.claude/skills/exploring-mcp-tool-quality/SKILL.md.
What does the skill do?
It turns $mcp_tool_call events into an error share, p50/p95 latency and reach per tool, then names the ones that fail or drag.
What has to be set up first?
A PostHog MCP server exposing posthog:execute-sql and posthog:query-mcp-tool-stats, plus $mcp_tool_call events from MCP servers that run PostHog's analytics SDK.