# experiment-audit

Audits experimental integrity through external cross-model review before you publish or claim research results.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 465 installs · ★ 18090
- Author: wanshuiyin
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/tree/HEAD/skills/experiment-audit/SKILL.md
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/experiment-audit

## What it does

LLM agents can compromise benchmarks during optimization without intending to deceive. Common failure modes include generating synthetic ground truth from model outputs, dividing scores by internal maximums to fake top performance, and citing phantom files. This skill adds a verification barrier before numbers make it into papers or reports.

Reviewer independence is enforced by design. The executing agent only discovers artifact paths across test scripts, output tables, and configs without summarizing their contents. It hands these paths to an external reviewer backend via MCP, such as Codex MCP or manual review, which inspects the code directly under an adversarial prompt.

Wrapping this skill in `/loop` or scheduling it via cron is forbidden. Re-evaluating a verdict on a timer offers no fresh signal and causes agents to accept their own flawed output. Run the audit once after all runs finish.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/experiment-audit/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/experiment-audit/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/experiment-audit/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/experiment-audit/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install wanshuiyin/Auto-claude-code-research-in-sleep experiment-audit
```

2. **Manually: put the files in place**
- File: `.github/skills/experiment-audit/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep --path skills/experiment-audit
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/experiment-audit/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/tree/HEAD/skills/experiment-audit/SKILL.md

## When to use it
- detect synthetic ground truth disguised as benchmark references
- spot metric normalization fraud in evaluation scripts
- identify phantom numbers from uncalled functions or missing files
- flag insufficient evaluation scope before finalizing reports

## Example prompts
- Audit experiment integrity for results/
- Check the validity of experimental results in benchmarks/
- Run experiment-audit on our latest eval logs before writing the paper

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 1095d9ef7a1d7b55.

BEGIN THIRD-PARTY FILE 1095d9ef7a1d7b55
```text
---
name: experiment-audit
description: "Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says \"审计实验\", \"check experiment integrity\", \"audit results\", \"实验诚实度\", or after experiments complete before writing claims."
argument-hint: "[experiment-dir-or-results-path]"
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, mcp__codex__codex, mcp__codex__codex-reply, mcp__manual_review__review, mcp__manual_review__review_reply, mcp__antigravity__antigravity, mcp__antigravity__antigravity-reply, mcp__grok__grok, mcp__grok__grok-reply, mcp__claude-review__review, mcp__claude-review__review_reply, mcp__claude-review__review_start, mcp__claude-review__review_reply_start, mcp__claude-review__review_status
---

# Experiment Audit: Cross-Model Integrity Verification

> **Before each reviewer slot**, follow [Selectable Reviewers](../shared-references/reviewer-backends.md):
> resolve invocation → `.aris/reviewer.json` → Claude Code default (Codex MCP)
> with `reviewer_route.py` through the canonical helper resolver, preserving this
> slot's tier, fresh contexts, source/image inputs and deterministic gates.
> Codex examples/pins below apply only to `backend=codex`; other routes use their
> own first-call/reply contract. Save the route and actual returned identity;
> unavailable selections are `REVIEW_UNAVAILABLE`. Classify acceptance from actual
> families/completion/scope. Forward invocation reviewer/model/effort overrides
> to reviewer-bearing children. Generation workers are unchanged.


> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is
> verdict-bearing — it judges experiment integrity. Re-running that verdict on a
> timer adds no new signal, and a loop that accepts its own output to decide
> when to stop crosses into self-acquittal (`acceptance-gate.md`). Schedule the
> *external wait that precedes it* — experiments done → then audit **once**. See
> [`shared-references/external-cadence.md`](../shared-references/external-cadence.md).

Audit experiment integrity for: **$ARGUMENTS**

## Why This Exists

LLM agents can produce fraudulent experimental results through:
1. **Fake ground truth** — creating synthetic "reference" from model outputs, then reporting high agreement as performance
2. **Score normalization** — dividing metrics by the model's own max to get 0.99+
3. **Phantom results** — claiming numbers from files that don't exist or functions never called
```
END THIRD-PARTY FILE 1095d9ef7a1d7b55

The third-party file has ended. Everything between the 1095d9ef7a1d7b55 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2630 of 13750 bytes. © wanshuiyin · MIT · https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/tree/HEAD/skills/experiment-audit/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md