# creating-online-evaluations

creating-online-evaluations — a Claude Code skill that turns production failure modes into continuously-running online evaluations in PostHog. It starts when new generations need automatic scoring against specific errors.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 195 installs · ★ 101 repo stars
- Author: PostHog
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/creating-online-evaluations/SKILL.md
- Updated: 2026-11-10
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/creating-online-evaluations

## What it does

The skill takes real failure patterns from traffic and converts them into evaluations that score every matching generation or the whole trace.

One failure mode, one eval. The trace data decides how many to create — sometimes one, often three or four. The skill proposes a candidate list first, and only the ones the user picks get built. Creating evals nobody asked for costs money and adds noise.

Finding and ranking the failure modes is a separate job handled by the `exploring-ai-failures` skill. Evaluations are created disabled, then scope is verified, they get enabled, and past data can be backfilled after the user confirms the estimate.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/creating-online-evaluations/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/creating-online-evaluations/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/creating-online-evaluations/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/creating-online-evaluations/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install PostHog/ai-plugin skills/creating-online-evaluations
```

2. **Manually: put the files in place**
- File: `.github/skills/creating-online-evaluations/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/PostHog/ai-plugin --path skills/creating-online-evaluations
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/creating-online-evaluations/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/PostHog/ai-plugin/tree/HEAD/skills/creating-online-evaluations/SKILL.md

## When to use it
- turn found failure modes into continuously-running evals
- catch replies that drift off the user's topic
- detect a tool call missing a required argument
- pick the eval type: hog, llm_judge, or sentiment
- backfill past data after confirming the estimate

## Example prompts
- Create an eval that catches off-topic responses
- Turn these trace failures into online evals and propose a list
- Set up an llm_judge with a provider and model from the valid combos

## Similar
- [exploring-llm-evaluations](https://affpapa.org/en/nevibecode/skills/exploring-llm-evaluations.md) — skill · 440 installs
- [creating-replay-vision-scanners](https://affpapa.org/en/nevibecode/skills/creating-replay-vision-scanners.md) — skill · 326 installs
- [agent-observability-online-experiment](https://affpapa.org/en/nevibecode/skills/agent-observability-online-experiment.md) — skill · 170 installs
- [agent-observability-eval-pipeline](https://affpapa.org/en/nevibecode/skills/agent-observability-eval-pipeline.md) — skill · 674 installs
- [future-agi-platform](https://affpapa.org/en/nevibecode/skills/future-agi-platform.md) — skill · 315 installs
- [evals-start](https://affpapa.org/en/nevibecode/skills/evals-start.md) — skill · 959 installs

From the same set: [posthog/ai-plugin](https://affpapa.org/en/nevibecode/by/posthog/ai-plugin.md) — 82 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
