creating-online-evaluations — Claude Code skill
creating-online-evaluations — a Claude Code skill that turns production failure modes into continuously-running online evaluations in PostHog. It starts when new generations need automatic scoring against specific errors.
no license 195 installs ★ 101 updated 2026-11-10
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-codeWhat creating-online-evaluations does
The skill takes real failure patterns from traffic and converts them into evaluations that score every matching generation or the whole trace.
One failure mode, one eval. The trace data decides how many to create — sometimes one, often three or four. The skill proposes a candidate list first, and only the ones the user picks get built. Creating evals nobody asked for costs money and adds noise.
Finding and ranking the failure modes is a separate job handled by the exploring-ai-failures skill. Evaluations are created disabled, then scope is verified, they get enabled, and past data can be backfilled after the user confirms the estimate.
What it helps with
- turn found failure modes into continuously-running evals
- catch replies that drift off the user's topic
- detect a tool call missing a required argument
- pick the eval type: hog, llm_judge, or sentiment
- backfill past data after confirming the estimate
Example prompts for the agent
- Create an eval that catches off-topic responses
- Turn these trace failures into online evals and propose a list
- Set up an llm_judge with a provider and model from the valid combos
How to install creating-online-evaluations
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code2Manually: put the files in place
~/.claude/skills/creating-online-evaluations/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/creating-online-evaluations/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a codex2Manually: put the files in place
.agents/skills/creating-online-evaluations/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a cursor2Manually: put the files in place
.cursor/skills/creating-online-evaluations/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install PostHog/ai-plugin skills/creating-online-evaluations2Manually: put the files in place
.github/skills/creating-online-evaluations/SKILL.md1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a gemini-cli2Platform CLI
gemini skills install https://github.com/PostHog/ai-plugin --path skills/creating-online-evaluationsManually: put the files in place
.devin/skills/creating-online-evaluations/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about creating-online-evaluations
How do I install creating-online-evaluations in Claude Code?
Run npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code, then open ~/.claude/skills/creating-online-evaluations/SKILL.md.
What does creating-online-evaluations do?
It converts production failure modes into PostHog online evaluations that score new generations or whole traces automatically. The skill proposes a candidate set first, then builds only the ones you pick.
How is it different from exploring-ai-failures?
exploring-ai-failures finds and ranks failure modes in traces, while creating-online-evaluations turns those modes into evals. If the modes aren't identified yet, run the former first.