Skip to main content
skill CX Codex

scanning-experiments-with-replay-vision — Claude Code skill

The scanning-experiments-with-replay-vision skill for Claude Code sets up a PostHog Replay Vision scanner targeted at an experiment. It watches exposed user recordings and analyzes differences across test variants with an LLM.

no license 148 installs ★ 101 updated 2026-10-24

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-code
Open on GitHub

What scanning-experiments-with-replay-vision does

This skill provisions an experiment type Replay Vision scanner tied to a specific PostHog A/B test. It fetches the underlying feature flag configuration via experiment-get, sets up a unified prompt, and balances session sampling across variants using balance_variants.

The LLM receives the visual change description but never gets informed which variant the recorded visitor actually saw. This prevents model bias and exposes broken feature flag rollouts directly from user actions. Cohort filtering runs entirely server-side; defining custom exposure filters inside query triggers an API error.

Draft experiments lacking a launch date cannot be provisioned, and ended runs require a backfill. The scanner is initially created disabled so the analysis prompt can be tested against real sessions before credits are spent.

What it helps with

  • scan session recordings for a specific PostHog experiment
  • balance session sampling evenly between test and control cohorts
  • draft unbiased LLM prompts that hide variant assignment from the model
  • estimate Replay Vision credit usage before turning the scanner on

Example prompts for the agent

  • Create a Replay Vision scanner for experiment 123 to review checkout drop-offs
  • Scan session recordings from our onboarding experiment using vision AI
  • Set up an experiment scanner to inspect user interactions across all test variants

How to install scanning-experiments-with-replay-vision

1Skills CLI (Vercel)

npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-code

2Manually: put the files in place

~/.claude/skills/scanning-experiments-with-replay-vision/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/scanning-experiments-with-replay-vision/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about scanning-experiments-with-replay-vision

How to install scanning-experiments-with-replay-vision in Claude Code?

Run npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-code in your terminal.

How does the experiment scanner avoid evaluation bias?

The LLM receives context about what changed on screen, but PostHog conceals the visitor's assigned variant so the summary reflects actual screen activity rather than expected outcomes.

When will the scanner refuse to provision?

It fails on draft experiments without a set start date, because the PostHog API requires an active or completed rollout with existing exposure data.

Similar skills

All skills for Testing and QA →

From the same set82

The whole set →
working-with-task-comments The working-with-task-comments Claude Code skill retrieves comments on PostHog tasks, their artifacts, and canvases. It activates when an agent needs to inspect feedback or follow a discussion thread. CX Codex +1 149installs signals-scout-customer-analytics-billing-and-usage signals-scout-customer-analytics-billing-and-usage — a skill for Claude Code that tracks product usage and forecast MRR changes within individual accounts. It spots product-level shifts hidden by a steady account total. CX Codex +1 149installs building-canvases The building-canvases Claude Code skill creates and edits standalone PostHog canvas apps. It activates for requests to build, update, or fix one of these apps. CX Codex +1 149installs adding-warehouse-person-properties The adding-warehouse-person-properties Claude Code skill maps columns from a synced warehouse table to PostHog person or group properties. It activates for requests to add warehouse data to profiles or manage an existing mapping. CX Codex +1 148installs creating-box-plot-insights The creating-box-plot-insights Claude Code skill builds box plot insights in PostHog from product analytics or SQL. It runs on requests to chart a numeric distribution, compare medians, or save an insight. CX Codex +1 148installs exploring-mcp-tool-original-user-motive The exploring-mcp-tool-original-user-motive Claude Code skill builds a taxonomy of the goals users had before reaching an MCP tool. It groups session openings and publishes the categories in a PostHog notebook. CX Codex +1 148installs context-layer-consolidation The context-layer-consolidation Claude Code skill keeps a context wiki coherent by working through a deterministic lint report queue. It handles the queue before dream publisher creates its branch and commit. CX Codex +1 148installs building-react-quill-canvases building-react-quill-canvases — a Claude Code skill for building PostHog canvases with React and Quill. It activates when building-canvases routes a dashboard, form, or tool request to a React implementation. CX Codex +1 148installs

Project sponsors

Companies that keep this analytics open