Skip to main content
skill CX Codex

exploring-ai-failures — Claude Code skill: find AI failures in traces

exploring-ai-failures reads real production traces from PostHog and returns a ranked list of the ways your LLM feature breaks.

no license 198 installs ★ 101 updated 2026-11-10

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code
Open on GitHub

What exploring-ai-failures does

Most AI failures raise nothing. The call returns 200, the answer is wrong, off-topic, ignores the instruction, or picks the wrong tool. This skill hunts those quiet cases next to the loud ones and names what it finds.

It runs in three moves. Narrow to a single use case first, because a support chat, a summarizer and an agent each break in their own way. Then pick which traces to open: error flags, metric tails such as tokens, cost or latency, spikes in evals you already run, or a mixed sample when there is no signal at all. Finally read them, around 20–30 per use case, with query-llm-trace; it takes a traceId from the query-llm-traces-list output.

Every lookup goes through the posthog: tools such as execute-sql, llma-evaluation-list and generate-app-url, so the PostHog MCP server has to be connected first. Heavy fields like $ai_input and $ai_output_choices sit in ai_events, not events, and a query aimed at the wrong table comes back empty. The skill states one more thing outright: a grouped table of error messages only points at which traces to open. It is not the finding.

What it helps with

  • find silent failures that never log an error
  • group one feature's traces into named failure modes
  • choose traces to read by cost, length or latency outliers
  • decide which prompt fix to ship first
  • turn a frequent failure mode into an automated eval

Example prompts for the agent

  • Read the production traces for the support agent and tell me which failure modes show up most
  • Why do the answers drift off-topic? Open a batch of traces and code what went wrong
  • Rank the failure modes you found and say what to fix first

How to install exploring-ai-failures

1Skills CLI (Vercel)

npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code

2Manually: put the files in place

~/.claude/skills/exploring-ai-failures/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/exploring-ai-failures/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about exploring-ai-failures

How do I install exploring-ai-failures in Claude Code?

Run npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code; the skill lands at ~/.claude/skills/exploring-ai-failures/SKILL.md.

What does exploring-ai-failures do?

It reads production LLM traces in PostHog and turns them into a ranked taxonomy of failure modes. Quiet wrong answers count as much as thrown exceptions.

What has to be set up before it works?

The PostHog MCP server, since every step uses its tools: query-llm-traces-list, query-llm-trace, execute-sql. Heavy $ai_* content lives in ai_events rather than events, so queries have to target that table.

Similar skills

All skills for AI and ML →

From the same set82

The whole set →
signals-scout-apm signals-scout-apm watches OpenTelemetry traces in PostHog for error-rate and latency shifts per service, new error signatures and traffic drops, and files a report only on a confirmed change. CX Codex +1 202installs exploring-mcp-tool-usage exploring-mcp-tool-usage — a routing skill for PostHog MCP analytics: it offers a menu of questions about tool calls, then hands you to the query that answers it. CX Codex +1 200installs checking-deploy-timing checking-deploy-timing tells you when a commit or PR reached prod-us or prod-eu by reading PostHog's hidden deploy annotations and matching them against the merge commit on GitHub. CX Codex +1 200installs setting-up-a-custom-rest-source setting-up-a-custom-rest-source — a skill for Claude Code and Codex that links arbitrary REST APIs to the PostHog data warehouse using a JSON manifest. Activates when importing custom endpoints without native connectors. CX Codex +1 199installs resolving-ingestion-warnings resolving-ingestion-warnings walks through PostHog ingestion warnings: dropped events, rejected person merges, oversized payloads. Use it when event counts look low or identify calls misbehave. CX Codex +1 197installs exploring-scouts exploring-scouts reads the state of PostHog Signals scouts: which ones a project has, how recent runs went, and where their reports ended up. Read-only. CX Codex +1 197installs debugging-surveys debugging-surveys investigates why PostHog Surveys fail to appear or record complete answers. Useful for support cases involving targeting, SDK behavior, and stored responses. CX Codex +1 197installs authoring-scouts authoring-scouts is a Claude Code skill for creating, tuning, and testing PostHog Signals scouts that inspect project metrics on a schedule and submit findings. CX Codex +1 196installs

Project sponsors

Companies that keep this analytics open