Skip to main content
skill CX Codex

audit-analysis — Claude Code skill: what it does, how to install

audit-analysis launches an isolated subagent that adversarially checks changed empirical analysis code against a base branch, catching errors before a referee does.

MIT ★ 579 updated 2026-10-28

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add claesbackman/AI-research-feedback --skill audit-analysis -a claude-code
Open on GitHub

What audit-analysis does

The skill diffs the working tree against a base ref (main by default) and hands the changed files to an auditor running in a separate context. It checks sample construction (observation counts before and after each filter and merge), merge logic (keys, uniqueness, what happens to unmatched rows), variable construction (units, logs versus levels, lag alignment), silent failures (missing values coerced to zero, fillna(0), destring ... force), and estimation choices such as clustering level and what the fixed effects absorb.

Running the audit in a fresh subagent is deliberate: whoever wrote the code, including the current session, must not be able to steer the outcome. That is why the skill forbids reading the changed files or forming an opinion before launching the auditor. Every finding must point to a file, a line, and a quoted excerpt; anything that doesn't is dropped, and the count of dropped findings is reported back. Diffs above roughly 1,500 changed lines get split across two parallel auditors covering different categories instead of one.

What it helps with

  • catch errors in empirical code before submitting results
  • flag undocumented observation drops across filters and merges
  • surface silent failures like fillna(0) or destring force
  • verify merge keys and uniqueness assumptions
  • check clustering and fixed effects against the stated specification

Example prompts for the agent

  • Run /audit-analysis before I send these results to my advisor
  • Audit the changes against the develop branch
  • Check this analysis code for silent failures and merge issues before submission

How to install audit-analysis

1Skills CLI (Vercel)

npx skills add claesbackman/AI-research-feedback --skill audit-analysis -a claude-code

2Manually: put the files in place

~/.claude/skills/audit-analysis/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/audit-analysis/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about audit-analysis

How do I install audit-analysis in Claude Code?

Run npx skills add claesbackman/AI-research-feedback --skill audit-analysis -a claude-code; the skill lands at ~/.claude/skills/audit-analysis/SKILL.md.

What does audit-analysis do?

It spins up an isolated subagent that adversarially reviews changed analysis code against a base ref, looking for errors in sample construction, merges, variable construction, silent failures, and estimation, and reports each finding with a file, line, and quote.

How is audit-analysis different from review-paper-code?

audit-analysis does not check reproducibility or whether the code matches a paper; that's handled by the separate review-paper-code skill. This one targets correctness bugs in the changed code itself.

Source text

SKILL.md · MIT
---
name: audit-analysis
description: Adversarially audit changed analysis code against a base ref, hunting for correctness errors in sample construction, merges, variable construction, silent failures, and clustering or fixed effects. Runs in an isolated subagent. Use before circulating results or submitting. This is not a reproducibility or paper-to-code review — use review-paper-code for that.
argument-hint: "[optional: base ref, default main]"
allowed-tools: Bash, Read, Grep, Glob, Agent
disable-model-invocation: true
---

# Audit Analysis Code

Find errors in changed empirical code before a referee does.

The audit runs in a subagent with a clean context. That isolation is the point: whoever wrote the code — including this session, if it helped — must not be able to steer the findings. Do not read the changed files yourself before launching, do not form a view, and do not answer the auditor's questions mid-run.

## Phase 1: Establish scope

Set `BASE` from `$ARGUMENTS` if given, otherwise `main`.

Run, and stop with a short explanation if any of the first three fail:

- `git rev-parse --git-dir` — must be a repository
- `git rev-parse --verify BASE` — the base ref must exist
- `git diff --stat BASE` — if empty, there is nothing to audit
- `git log BASE..HEAD --oneline` — may legitimately be empty when the work is uncommitted, or when HEAD is BASE and only the working tree has changed. Note it and drop the commit-message check from the audit.

Report to the user in two or three lines: base ref, number of changed files, number of changed lines, and whether commit messages are available. Then launch immediately.

## Phase 2: Launch the auditor

One `Agent` call, `subagent_type: "general-purpose"`. Substitute `BASE` and pass this verbatim:

> Review empirical research code adversarially. The author wants it broken now
> rather than by a referee. Read `git log BASE..HEAD` and `git diff BASE`, then
> the changed files in full. Follow variables built outside the diff.
>
> Check, and report on each of:
> - Claims vs. code: do comments and commit messages match what runs? Quote
>   both sides of any disagreement.
> - Sample: N before and after every filter, merge, and collapse. Take N from
>   logs; write "N unverified" where there is no log. Flag undocumented drops.
> - Merges: key, uniqueness on the side that needs it, fate of unmatched
>   observations, whether `_merge` is inspected, duplicate id-period pairs after.
> - Variables: trace every regressor and outcome. Units, logs vs. levels,
>   deflation, lag alignment. Does construction match the name?
> - Silent failures: missings coerced to zero, `if x > 0` true on missing,
>   `destring ... force`, `replace` that changes nothing, loops that skip.
>   In Python, `fillna(0)`, silent dtype coercion, chained assignment.
> - Estimation: clustering level and cluster count, what the fixed effects
>   absorb, weights, whether estimation N matches the sample traced above.
>
> Each finding: file, line, quoted excerpt, what is wrong, consequence for the
> results. Tag CONFIRMED (visible in the code) or SUSPECTED (needs the data).
> Style and naming are not findings. Order by consequence, worst first, ten max.
> Then one line per category: what you found, or that you found nothing. Close
> with the one thing you could not check without the data. Change nothing.

If the diff exceeds roughly 1,500 changed lines, run two auditors in parallel instead — one taking claims, sample, and merges, the other taking variables, silent failures, and estimation — and concatenate their findings. Do not split a smaller diff; the categories inform each other.

## Phase 3: Relay without softening

Pass the findings through in the order returned, worst first. Do not reclassify a SUSPECTED finding as fine, do not add reassurance, and do not open with what the code gets right. The user asked for errors.

Drop any finding that lacks a file, a line, and a quoted excerpt, and tell the user how many you dropped. Unanchored findings are the failure mode this design exists to catch — an auditor told to find errors will manufacture them if nothing forces it to point at code.

Reproduce the per-category coverage lines verbatim, including the categories that came back clean, and the closing line about what could not be checked without the data. A clean category is a claim the auditor is on the record for.

Fix nothing. If the user wants repairs, that is a separate request.

© claesbackman · MIT · claesbackman/AI-research-feedback

Similar skills

All skills for Science and research →

From the same set13

The whole set →
pdf-to-markdown pdf-to-markdown splits PDF documents into page blocks and converts them to Markdown, automatically trimming bibliographies. CX Codex +1 ★ 579repo stars explain-diff The explain-diff skill turns a code-change review into a standalone HTML page with a quiz. Use it to explain a diff and examine its effects on empirical results. CX Codex +1 ★ 579repo stars review-paper-light review-paper-light is a Claude Code skill that checks economics papers before submission, assessing contribution, identification, and causal claims in about a minute. CX Codex +1 ★ 579repo stars review-paper-code review-paper-code reviews research papers in LaTeX alongside Stata, R, or Python code, verifying reproducibility and checking empirical claims against scripts. CX Codex +1 ★ 579repo stars explorable-deck The explorable-deck skill turns a paper, notes, or a codebase into a Quarto reveal.js talk with a clear narrative and live SVG stages. Use it for lectures, conference talks, or presenting research. CX Codex +1 ★ 579repo stars explorable-beamer explorable-beamer builds LaTeX Beamer decks in Nicky Case's explorable-explanation style: one idea per frame, assertion titles, a running example, TikZ stages the presenter walks through, and an appendix of side rooms. CX Codex +1 ★ 579repo stars explorable-pptx explorable-pptx generates explorable-style PowerPoint slide decks with per-frame speaker notes and automated screenshot validation. CX Codex +1 ★ 579repo stars paper-version paper-version turns a LaTeX research paper into a public-facing brief or summary and produces an HTML page. Use it to explain a study without assuming readers know economics. CX Codex +1 ★ 579repo stars

Project sponsors

Companies that keep this analytics open