Skip to main content
skill CX Codex

creating-online-evaluations — Claude Code skill

creating-online-evaluations — a Claude Code skill that turns production failure modes into continuously-running online evaluations in PostHog. It starts when new generations need automatic scoring against specific errors.

no license 195 installs ★ 101 updated 2026-11-10

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code
Open on GitHub

What creating-online-evaluations does

The skill takes real failure patterns from traffic and converts them into evaluations that score every matching generation or the whole trace.

One failure mode, one eval. The trace data decides how many to create — sometimes one, often three or four. The skill proposes a candidate list first, and only the ones the user picks get built. Creating evals nobody asked for costs money and adds noise.

Finding and ranking the failure modes is a separate job handled by the exploring-ai-failures skill. Evaluations are created disabled, then scope is verified, they get enabled, and past data can be backfilled after the user confirms the estimate.

What it helps with

  • turn found failure modes into continuously-running evals
  • catch replies that drift off the user's topic
  • detect a tool call missing a required argument
  • pick the eval type: hog, llm_judge, or sentiment
  • backfill past data after confirming the estimate

Example prompts for the agent

  • Create an eval that catches off-topic responses
  • Turn these trace failures into online evals and propose a list
  • Set up an llm_judge with a provider and model from the valid combos

How to install creating-online-evaluations

1Skills CLI (Vercel)

npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code

2Manually: put the files in place

~/.claude/skills/creating-online-evaluations/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/creating-online-evaluations/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about creating-online-evaluations

How do I install creating-online-evaluations in Claude Code?

Run npx skills add PostHog/ai-plugin --skill creating-online-evaluations -a claude-code, then open ~/.claude/skills/creating-online-evaluations/SKILL.md.

What does creating-online-evaluations do?

It converts production failure modes into PostHog online evaluations that score new generations or whole traces automatically. The skill proposes a candidate set first, then builds only the ones you pick.

How is it different from exploring-ai-failures?

exploring-ai-failures finds and ranks failure modes in traces, while creating-online-evaluations turns those modes into evals. If the modes aren't identified yet, run the former first.

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set82

The whole set →
resolving-ingestion-warnings resolving-ingestion-warnings walks through PostHog ingestion warnings: dropped events, rejected person merges, oversized payloads. Use it when event counts look low or identify calls misbehave. CX Codex +1 197installs exploring-scouts exploring-scouts reads the state of PostHog Signals scouts: which ones a project has, how recent runs went, and where their reports ended up. Read-only. CX Codex +1 197installs debugging-surveys debugging-surveys investigates why PostHog Surveys fail to appear or record complete answers. Useful for support cases involving targeting, SDK behavior, and stored responses. CX Codex +1 197installs authoring-scouts authoring-scouts is a Claude Code skill for creating, tuning, and testing PostHog Signals scouts that inspect project metrics on a schedule and submit findings. CX Codex +1 196installs signals-scout-insight-alerts signals-scout-insight-alerts — a Claude Code skill for reviewing insight alert history in PostHog. It scans recent alert triggers and flags material incidents that missed notifying the team. CX Codex +1 195installs signals-scout-data-warehouse signals-scout-data-warehouse monitors warehouse imports for failures, stale data, and sharp row-count drops, then flags recurring query patterns that may merit materialized views. CX Codex +1 195installs signals-scout-mcp-tool-calls signals-scout-mcp-tool-calls — a PostHog Signals skill that sifts $mcp_tool_call telemetry for MCP tools agents struggle with, then files one report per product category. CX Codex +1 194installs signals-scout-web-vitals signals-scout-web-vitals tracks page p75 LCP, INP, CLS, and FCP from PostHog events, spots performance degradation against Google thresholds, and links regressions to deploys. CX Codex +1 194installs

Project sponsors

Companies that keep this analytics open