# exploring-ai-failures

exploring-ai-failures reads real production traces from PostHog and returns a ranked list of the ways your LLM feature breaks.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 198 installs · ★ 101 repo stars
- Author: PostHog
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/exploring-ai-failures/SKILL.md
- Updated: 2026-11-10
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/exploring-ai-failures

## What it does

Most AI failures raise nothing. The call returns 200, the answer is wrong, off-topic, ignores the instruction, or picks the wrong tool. This skill hunts those quiet cases next to the loud ones and names what it finds.

It runs in three moves. Narrow to a single use case first, because a support chat, a summarizer and an agent each break in their own way. Then pick which traces to open: error flags, metric tails such as tokens, cost or latency, spikes in evals you already run, or a mixed sample when there is no signal at all. Finally read them, around 20–30 per use case, with `query-llm-trace`; it takes a `traceId` from the `query-llm-traces-list` output.

Every lookup goes through the `posthog:` tools such as `execute-sql`, `llma-evaluation-list` and `generate-app-url`, so the PostHog MCP server has to be connected first. Heavy fields like `$ai_input` and `$ai_output_choices` sit in `ai_events`, not `events`, and a query aimed at the wrong table comes back empty. The skill states one more thing outright: a grouped table of error messages only points at which traces to open. It is not the finding.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/exploring-ai-failures/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/exploring-ai-failures/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/exploring-ai-failures/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/exploring-ai-failures/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install PostHog/ai-plugin skills/exploring-ai-failures
```

2. **Manually: put the files in place**
- File: `.github/skills/exploring-ai-failures/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/PostHog/ai-plugin --path skills/exploring-ai-failures
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/exploring-ai-failures/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill exploring-ai-failures -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/exploring-ai-failures/SKILL.md

## When to use it
- find silent failures that never log an error
- group one feature's traces into named failure modes
- choose traces to read by cost, length or latency outliers
- decide which prompt fix to ship first
- turn a frequent failure mode into an automated eval

## Example prompts
- Read the production traces for the support agent and tell me which failure modes show up most
- Why do the answers drift off-topic? Open a batch of traces and code what went wrong
- Rank the failure modes you found and say what to fix first

## Similar
- [exploring-llm-traces](https://affpapa.org/en/nevibecode/skills/exploring-llm-traces.md) — skill · 440 installs
- [exploring-llm-evaluations](https://affpapa.org/en/nevibecode/skills/exploring-llm-evaluations.md) — skill · 440 installs
- [langsmith-experiment-diagnosis](https://affpapa.org/en/nevibecode/skills/langsmith-experiment-diagnosis.md) — skill · 184 installs
- [agent-observability-session-classify](https://affpapa.org/en/nevibecode/skills/agent-observability-session-classify.md) — skill · 665 installs
- [agent-observability-eval-pipeline](https://affpapa.org/en/nevibecode/skills/agent-observability-eval-pipeline.md) — skill · 674 installs
- [langfuse](https://affpapa.org/en/nevibecode/plugins/aviv-sinai-langfuse.md) — plugin · ★ 1596 repo stars

From the same set: [posthog/ai-plugin](https://affpapa.org/en/nevibecode/by/posthog/ai-plugin.md) — 82 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
