Skip to main content
skill CX Codex

claim-verify — Claude Code skill: what it does, how to install

claim-verify is a Claude Code skill that checks data-backed claims in draft reports against raw datasets. It recalculates metrics and stress-tests fragile conclusions.

MIT ★ 185 updated 2026-06-30

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills
Open on GitHub

What claim-verify does

The skill runs an adversarial verification loop across a completed write-up. It reads the draft, extracts testable numerical assertions, and groups them by claim type. Each item gets a dedicated script inside ./sandbox to recompute the statistic from the source dataset. If an asserted number fails to replicate, it is marked as refuted immediately.

Numbers that reproduce undergo targeted stress testing. The assistant checks for outlier dependence, confounders, Simpson's reversals, and tiny subgroup sizes. When an apparent effect flips under stratification or hinges on five rows, the skill tags it as fragile. The resulting report keeps verified facts while qualifying, scoping, or cutting shaky interpretations.

Checks execute in your local shell using <analysis_cmd>, which defaults to python3. Python 3.9 or higher is required. Code relies primarily on standard library modules such as csv and statistics; missing packages like pandas trigger fallback logic or an explicit install request.

What it helps with

  • audit draft findings against underlying datasets before release
  • catch false causal claims caused by confounders or small samples
  • rephrase fragile statistical statements into qualified conclusions

Example prompts for the agent

  • Audit every factual claim in draft.md using the data in experiment_log.csv
  • Run claim-verify to stress-test the findings in summary.txt against raw_data.csv
  • Check numerical statements in results.md and rewrite fragile conclusions

How to install claim-verify

1Plugin marketplace

/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills

This installs the whole agent-loops plugin with all its skills

2Skills CLI (Vercel)

npx skills add gaasher/Agent-Loop-Skills --skill claim-verify -a claude-code

3Manually: put the files in place

~/.claude/skills/claim-verify/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/claim-verify/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about claim-verify

How to install claim-verify in Claude Code?

Add the marketplace source with /plugin marketplace add gaasher/Agent-Loop-Skills, then run /plugin install agent-loops@agent-loop-skills.

What is the primary role of claim-verify?

It acts as a pre-publication quality gate for existing drafts. It stress-tests stated metrics rather than conducting exploratory discovery from scratch.

What happens if third-party analytical packages are missing?

Verification scripts prioritize Python standard library tools. If dependencies like numpy or pandas are absent, the skill prompts before installing them via uv pip.

Source text

SKILL.md · MIT
---
name: claim-verify
description: >
  Use when the user has a results draft or a set of data-backed claims and wants each one
  adversarially verified against the underlying dataset before publishing — a pre-publication
  red-team of the findings. Extracts the discrete checkable claims from the draft, reproduces each
  claim's number against the data, stress-tests it against the threats most likely to kill it
  (outliers, confounds, Simpson's reversals, tiny subgroups, alternative specifications), and marks
  it verified, fragile, or refuted; fragile and refuted claims are revised — hedged, scoped, or
  retracted — until every claim is verified or appropriately qualified. The result is a draft where
  every surviving claim has been reproduced and survived a stress test. Not for open-ended discovery
  of new findings over a dataset (that is a data-analysis task), and not for diagnosing a single known
  anomaly or pipeline failure — this is a gate over an existing draft.
compatibility: Requires Python 3.9+
metadata:
  version: "0.1.0"
---

# Claim Verify Loop

A **claim-by-claim adversarial verification** loop over a results draft. The artifact is the draft;
the feedback signal is the count of **unverified claims** — claims not yet checked, or checked but not
yet survived a stress test. You drive it to zero: each claim ends **verified** (reproduces and
survives the obvious threats) or **appropriately qualified** (hedged, scoped, or retracted with the
reason).

The discipline: a number that merely reproduces is not trustworthy — most wrong findings reproduce
fine. A claim is verified only when it also **survives the threat most likely to kill it**: an
outlier, a confound, a subgroup too small to mean anything, a sign that flips under stratification.
This loop is a *gate on an existing draft*, not a generator of new findings.

## When to use

Use this when you have a draft (or a list of claims) drawn from a dataset and want each claim
red-teamed before it goes out. Default to verifying every discrete claim in the draft; if the user
flags a few high-stakes claims, prioritize those but still sweep the rest. Not for open-ended
discovery of new findings (that is the `data-analysis` loop) and not for diagnosing one known anomaly.

## Setup

Resolve bindings interactively. If `loop.run.yaml` exists in the working dir, load it, confirm the
values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is
available) infer a likely value for each binding and present it as the recommended option; on other
hosts ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format:
`examples/run.example.yaml`) and confirm the values before creating any other files.

| binding | meaning | default | how to infer |
|---|---|---|---|
| `<draft>` | results/claims document to verify (markdown/text) | — | scan the working dir for a results/report file |
| `<dataset>` | data the claims were drawn from; read-only ground truth | — | scan the working dir for the data file |
| `<analysis_cmd>` | interpreter that runs check snippets in the user's env | `python3` | `pyproject.toml`/`.venv`/`uv` in the working dir |
| `<report>` | the verified/revised draft this loop produces | `<sandbox_root>/verified.md` | — |
| `<sandbox_root>` | where check snippets + ledger live | `./sandbox` | — |
| `<budget>` | max iterations | 10 | — |

Check snippets run in the **user's environment** via `<analysis_cmd>`, so they may use whatever the
user has installed. Keep helper code **stdlib-first** (`csv`, `statistics`): if a snippet needs
`pandas`/`numpy`, probe with `try/except ImportError` and degrade to a stdlib path, or offer a
consented `uv pip install "pandas==<ver>"` — never assume the package is installed.

## The loop

Copy this checklist and tick items off:
- [ ] Iteration 0 — extract the discrete checkable claims from `<draft>`; record nothing as verified.
- [ ] Pick one unverified claim.
- [ ] Reproduce its exact number from `<dataset>`; if it does not reproduce → `refuted`.
- [ ] Stress-test against the threat(s) most likely to kill it; classify `verified` / `fragile`.
- [ ] Revise `<report>`: keep verified, hedge/scope/retract fragile, correct refuted.
- [ ] Append a ledger row; the claim leaves the unverified set. Stop when none remain or at `<budget>`.

**Iteration 0 — extract claims.** Read `<draft>` and list its discrete, checkable claims, each with
the number/effect it asserts and its **claim type** (a group difference, a correlation, a
causal/policy claim, a subgroup result, a rate). These are the live unverified set. If the draft is
prose, splitting it into discrete claims is the first job.

**Then, until stop (all claims resolved, or budget):**

1. **Pick one unverified claim.**
2. **Reproduce — the first gate.** Write `<sandbox_root>/iter<N>/check.py` to recompute the exact
   statistic the claim states from `<dataset>`. Run it with `<analysis_cmd>`, redirecting output to
   `<sandbox_root>/iter<N>/out.txt` (never flood your context). If the number does not reproduce →
   **refuted** (the number is wrong); skip to step 4.
3. **Stress-test — the second gate.** Hit the claim with the one or two threats most likely to kill it
   for its claim type:
   - **Outlier sensitivity** — recompute dropping extreme points / using a robust statistic. Does the
     effect survive, or was it driven by a handful of rows?
   - **Confound & Simpson's reversal** — stratify by the obvious confounder; does the effect hold
     within strata, or flip? A causal/policy claim that reverses within subgroups is **not** supported.
   - **Subgroup size & multiplicity** — how large is the subgroup? Is the result one of many
     comparisons? A striking rate on n=5 is noise.
   - **Alternative specification** — a defensible different cut (different bins, controlling for a
     covariate). Does the sign/size stay?

   Classify: **verified** (reproduces and survives) or **fragile** (reproduces but collapses or flips
   under a reasonable stress). A claim whose **number reproduces but whose implied interpretation is
   not supported** — a descriptive gap dressed up as causal ("treatment works"), a tiny-n rate sold as
   "superior", a one-point correlation called an "early-warning signal" — is **fragile**, not
   verified: the statistic is fine, the conclusion drawn from it is not.
4. **Revise the draft.** Update `<report>`:
   - **verified** → keep the claim, noting the robustness check it passed.
   - **fragile** → **hedge, scope, or down-weight** it to what the data supports (e.g. "descriptively
     higher, but the within-stratum comparison reverses — not evidence the treatment causes
     recovery"), or retract it. Never leave a fragile claim standing as first written.
   - **refuted** → correct it with the right number, or remove it.
   Record the verdict and the evidence.
5. **Log** one ledger row and continue; the claim leaves the unverified set.

## Ledger

`<sandbox_root>/ledger.tsv`, tab-separated, never commas in the text. Header:
```
iter	claim	verdict	threat	resolution
```
`verdict` ∈ {`extract`, `verified`, `fragile`, `refuted`}. Example:
```
iter	claim	verdict	threat	resolution
0	claims extracted	extract	-	7 claims listed
1	treatment recovery rate > control (70.6 vs 55.0)	verified	reproduced; holds	kept
2	treatment causes higher recovery (+16pp)	fragile	Simpson: control >= treatment within both age groups	rescoped to descriptive; causal claim retracted
3	biomarker correlates with recovery_days (r=0.16)	fragile	one outlier drives it (r=0.16 -> 0.02 without it)	retracted
5	pilot site 100% recovery (superior)	fragile	n=5 subgroup	hedged: too small to conclude
```
Report the **outcome**: the `<report>` path, the per-claim verdicts, and a summary — how many claims
were verified, hedged, or retracted, and the single most important fragility found.

## Constraints
- **Reproduce *and* stress-test — both.** A claim that only reproduces is not verified; it must
  survive the threat most likely to kill it. Skipping the stress test is the failure mode this loop
  exists to prevent.
- **Every verdict is backed by a re-run** recorded in `<report>`; no claim is waved through or
  condemned on intuition.
- **A fragile claim is changed, never left standing** — hedge it to what the data supports or retract
  it, because leaving it as first written is exactly what shipped the unverified draft.
- **Distinguish description from causation** — "treatment arm recovered more" can be true while
  "treatment causes recovery" is refuted by a confound; say exactly what the data supports.
- **Only read `<dataset>`** — never modify it, because it is the ground truth every claim is checked
  against. The sandbox is self-contained (no `../` escapes).
- One claim per iteration, so each verdict is attributable.
- Do not pause the loop to ask whether to continue; run until all claims are resolved or `<budget>`.

## Stops
- **Resolved** — no unverified or unresolved-fragile claims remain.
- **Budget** — `<budget>` iterations reached.

© gaasher · MIT · gaasher/Agent-Loop-Skills

Similar skills

All skills for Data and databases →

From the same set26

The whole set →
red-team red-team finds weaknesses in content filters, classifiers and APIs through adversarial stress-testing — generates attack inputs and builds a catalogue of failures. CX Codex +1 ★ 185repo stars prompt-optimize prompt-optimize is a Claude Code skill that iteratively refines an existing prompt against your evaluation command, keeping only edits that improve the score. CX Codex +1 ★ 185repo stars anomaly-investigation anomaly-investigation is a Claude Code skill that investigates a known spike, drop, or outlier by testing possible causes against data. CX Codex +1 ★ 185repo stars power-analysis power-analysis is a Claude Code skill for sizing a two-group study and drafting its preregistration. It runs during planning, before data collection, for A/B tests, simple RCTs, and evaluations of two models or configurations. CX Codex +1 ★ 185repo stars literature-survey literature-survey is a Claude Code skill that maps research sources against claims. It runs for questions that need a cited survey of agreement, disputes, and gaps. CX Codex +1 ★ 185repo stars optimize-loop optimize-loop is a skill that runs an edit-measure-revert loop for code modules or SQL queries, keeping changes only when correctness holds and a cost metric falls. CX Codex +1 ★ 185repo stars research-proposal Refines an existing research proposal through iterative literature checks, rubric-based scoring, and targeted revisions until it passes. CX Codex +1 ★ 185repo stars research-question Turn a broad topic into a small set of research questions, then score and revise them. Use it when deciding what a study should ask. CX Codex +1 ★ 185repo stars

Project sponsors

Companies that keep this analytics open