exploring-ai-failures — Claude Code skill: find AI failures in traces
exploring-ai-failures reads real production traces from PostHog and returns a ranked list of the ways your LLM feature breaks.
no license 198 installs ★ 101 updated 2026-11-10
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-codeWhat exploring-ai-failures does
Most AI failures raise nothing. The call returns 200, the answer is wrong, off-topic, ignores the instruction, or picks the wrong tool. This skill hunts those quiet cases next to the loud ones and names what it finds.
It runs in three moves. Narrow to a single use case first, because a support chat, a summarizer and an agent each break in their own way. Then pick which traces to open: error flags, metric tails such as tokens, cost or latency, spikes in evals you already run, or a mixed sample when there is no signal at all. Finally read them, around 20–30 per use case, with query-llm-trace; it takes a traceId from the query-llm-traces-list output.
Every lookup goes through the posthog: tools such as execute-sql, llma-evaluation-list and generate-app-url, so the PostHog MCP server has to be connected first. Heavy fields like $ai_input and $ai_output_choices sit in ai_events, not events, and a query aimed at the wrong table comes back empty. The skill states one more thing outright: a grouped table of error messages only points at which traces to open. It is not the finding.
What it helps with
- find silent failures that never log an error
- group one feature's traces into named failure modes
- choose traces to read by cost, length or latency outliers
- decide which prompt fix to ship first
- turn a frequent failure mode into an automated eval
Example prompts for the agent
- Read the production traces for the support agent and tell me which failure modes show up most
- Why do the answers drift off-topic? Open a batch of traces and code what went wrong
- Rank the failure modes you found and say what to fix first
How to install exploring-ai-failures
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code2Manually: put the files in place
~/.claude/skills/exploring-ai-failures/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/exploring-ai-failures/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a codex2Manually: put the files in place
.agents/skills/exploring-ai-failures/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a cursor2Manually: put the files in place
.cursor/skills/exploring-ai-failures/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install PostHog/ai-plugin skills/exploring-ai-failures2Manually: put the files in place
.github/skills/exploring-ai-failures/SKILL.md1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a gemini-cli2Platform CLI
gemini skills install https://github.com/PostHog/ai-plugin --path skills/exploring-ai-failuresManually: put the files in place
.devin/skills/exploring-ai-failures/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about exploring-ai-failures
How do I install exploring-ai-failures in Claude Code?
Run npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code; the skill lands at ~/.claude/skills/exploring-ai-failures/SKILL.md.
What does exploring-ai-failures do?
It reads production LLM traces in PostHog and turns them into a ranked taxonomy of failure modes. Quiet wrong answers count as much as thrown exceptions.
What has to be set up before it works?
The PostHog MCP server, since every step uses its tools: query-llm-traces-list, query-llm-trace, execute-sql. Heavy $ai_* content lives in ai_events rather than events, so queries have to target that table.