Skip to main content
skill CX Codex

exploring-mcp-tool-quality — Claude Code skill: MCP tool error rates

exploring-mcp-tool-quality reads PostHog MCP analytics to rank tools by failure share, latency and reach, then drills into the ones that break.

no license 202 installs ★ 101 updated 2026-10-24

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code
Open on GitHub

What exploring-mcp-tool-quality does

Each call made by an MCP server that runs PostHog's analytics SDK shows up as a $mcp_tool_call event in the shared events table. There is no separate table, so the $mcp_* fields live as properties on that event and every metric is an aggregation over it. The skill holds the query shapes for those metrics.

For one tool it uses the typed calls. posthog:query-mcp-tool-stats takes a toolName and dateRange and returns calls, errors, p50/p95, users and sessions. posthog:query-mcp-tool-failures buckets error messages by harness. Ranking tools side by side has no typed equivalent, so that part goes through HogQL in posthog:execute-sql. Before quoting any failure-rate headline the skill looks for the governed metric mcp_tool_call_fail_pct in posthog:metric-list and runs it with posthog:data-catalog-metric-run once it is approved.

Hand-written queries break on two details. Grouping by the raw tool name hides real tools under the single-exec wrapper, so use the effective name from $mcp_exec_tool_call_name. The properties are strings, which means $mcp_is_error needs toBool. Leave out the time range and the query scans all of events; leave out a volume floor and a tool with three calls at 100% errors climbs to the top.

What it helps with

  • rank MCP tools by error share without small-sample noise
  • pull p50 and p95 latency for a single tool
  • group failures by error type to see what actually breaks
  • weigh call volume against failure rate before blaming a tool
  • get a day-by-day trend for one tool's errors

Example prompts for the agent

  • Which MCP tool has the worst error rate over the last 30 days?
  • Break down failures for <tool>: error types and the messages behind them.
  • Build the tool-quality matrix: error share, p50, p95 and reach.

How to install exploring-mcp-tool-quality

1Skills CLI (Vercel)

npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code

2Manually: put the files in place

~/.claude/skills/exploring-mcp-tool-quality/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/exploring-mcp-tool-quality/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about exploring-mcp-tool-quality

How do I install exploring-mcp-tool-quality in Claude Code?

Run npx skills add PostHog/ai-plugin --skill exploring-mcp-tool-quality -a claude-code. The skill file lands at ~/.claude/skills/exploring-mcp-tool-quality/SKILL.md.

What does the skill do?

It turns $mcp_tool_call events into an error share, p50/p95 latency and reach per tool, then names the ones that fail or drag.

What has to be set up first?

A PostHog MCP server exposing posthog:execute-sql and posthog:query-mcp-tool-stats, plus $mcp_tool_call events from MCP servers that run PostHog's analytics SDK.

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set82

The whole set →
building-workflows building-workflows is a Claude Code skill for building, patching, and testing PostHog automation graphs over MCP before publishing them in the visual editor. CX Codex +1 204installs exploring-replay-vision-observations The exploring-replay-vision-observations skill guides an agent through Replay Vision scanner results, from recurring patterns to individual session recordings. Use it to review findings and decide what to investigate next. CX Codex +1 203installs signals-scout-customer-analytics The signals-scout-customer-analytics skill tracks account-level engagement in PostHog and spots churn-risk or expansion patterns. Use it for recurring customer-health checks. CX Codex +1 203installs exploring-mcp-sessions exploring-mcp-sessions investigates PostHog MCP sessions to inspect tool call sequences, agent intent, and run failures. CX Codex +1 202installs exploring-mcp-intent-clusters exploring-mcp-intent-clusters analyzes PostHog MCP agent intent clusters to reveal what agents attempt to do, tool failure patterns, and semantic routing competition. CX Codex +1 202installs managing-reminders managing-reminders is a Claude Code skill that creates and manages private PostHog reminders with one-time or repeating schedules. It can nudge you about a task or resource. CX Codex +1 202installs signals-scout-apm signals-scout-apm watches OpenTelemetry traces in PostHog for error-rate and latency shifts per service, new error signatures and traffic drops, and files a report only on a confirmed change. CX Codex +1 202installs exploring-mcp-tool-usage exploring-mcp-tool-usage — a routing skill for PostHog MCP analytics: it offers a menu of questions about tool calls, then hands you to the query that answers it. CX Codex +1 200installs

Project sponsors

Companies that keep this analytics open