Skip to main content
skill CX Codex

consistency-audit — Claude Code skill: audit and install

The consistency-audit Claude Code skill checks a framework’s skills, library code, and dictionary for consistency. Run it on demand, before a release, or after a batch of edits.

Apache-2.0 ★ 89 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add bis-med-it/marco-BIS --skill consistency-audit -a claude-code
Open on GitHub

What consistency-audit does

The skill compares three parts of a framework: workflows in skills/, implementation in scripts/lib/, and specifications in skills/model-dictionary/. It looks for mismatched stage numbers and terminology, checks whether orchestrated skills are reachable, and compares documented behavior with the code.

The audit runs python -m pytest tests/test_doc_consistency.py -q, reviews semantic consistency, and executes python scripts/replications/blanchard_quah_1989/replicate.py. The replication is expected to produce a 6/6 paper-target comparison table and files under outputs/. The commands use Python and pytest.

The skill prepares a prioritized list of findings with file paths and line numbers. It proposes fixes and waits for confirmation before editing. This is a separate maintenance pass, not a stage in the research loop.

What it helps with

  • find drift between skills, code, and the dictionary
  • check stage numbering across a framework
  • compare documented behavior with implementation
  • prepare prioritized findings with file and line references

Example prompts for the agent

  • Audit the framework and check that its stages are consistent.
  • Check whether skills/, scripts/lib/, and skills/model-dictionary/ have drifted after these edits.
  • Run an audit before the release and propose fixes for confirmation.

How to install consistency-audit

1Skills CLI (Vercel)

npx skills add bis-med-it/marco-BIS --skill consistency-audit -a claude-code

2Manually: put the files in place

~/.claude/skills/consistency-audit/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/consistency-audit/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about consistency-audit

How do I install consistency-audit in Claude Code?

Run npx skills add bis-med-it/marco-BIS --skill consistency-audit -a claude-code. The skill file is at ~/.claude/skills/consistency-audit/SKILL.md.

What does consistency-audit do?

It checks agreement between the skills, library implementation, and dictionary, then runs tests, a semantic review, and an end-to-end replication. Findings include priorities and file line references.

Which checks does consistency-audit run?

It runs python -m pytest tests/test_doc_consistency.py -q, reviews semantic consistency, and executes python scripts/replications/blanchard_quah_1989/replicate.py. These commands use Python and pytest.

Source text

SKILL.md · Apache-2.0
---
name: consistency-audit
description: >
  On-demand framework self-audit. Checks the three layers (skills ↔ lib ↔
  dictionary) for drift, internal inconsistency, and doc-vs-practice mismatch.
  Use when the user asks to "audit the framework", "check for consistency /
  drift", "is the workflow consistent", "go through the end-to-end process",
  before a release or promotion, or after a batch of edits to skills, lib, or
  the dictionary. This is NOT a stage of the 7-stage research loop; it is a
  maintenance pass run occasionally, on demand, by a framework developer.
---

# Consistency audit

The framework's three layers (the `skills/` workflow + knowledge, the
`scripts/lib/` implementation, and the `skills/model-dictionary/` specs) must
not silently drift out of agreement.

There are **two layers of defence**, and you run both:

| Layer | Where | Cadence | Catches |
|---|---|---|---|
| **Mechanical** | `tests/test_doc_consistency.py` | every test run (automated) | YAML parse breaks, index↔entries drift, stale module/entry counts, invalid `verified.status`, broken skill→file links, dangling `lib.*` references, replication folders missing core files |
| **Semantic** | this skill | on demand | stage-numbering coherence, propose/confirm choreography, doc-vs-practice, orphaned/overlapping skills, terminology drift (the judgement calls a regex can't make) |

The mechanical layer is necessary but **not sufficient**: a green test suite
with an incoherent workflow is still broken. Run the semantic pass too.

## When to run

- The user asks to audit / check the framework, or to "go through the
  end-to-end process".
- Before promoting work to canonical, or before a release.
- **After a batch of edits to skills, lib, or the dictionary**, especially a
  paper rebuild. Rebuilds can silently break dependent tests and docs.

## Procedure

### Step 1: Mechanical checks (fast, deterministic)

```
python -m pytest tests/test_doc_consistency.py -q
```

Every failure is a concrete drift with a `file:line`. Fix or report each. If
during the audit you find a new class of drift that is *mechanically* checkable
(a count, a reference, a parse, a path), **encode it as a new test here** so it
can never regress. That is how the mechanical layer grows.

### Step 2: Semantic audit (judgement)

A regex can't read meaning. By reading the skills, check:

1. **Stage-numbering coherence.** The orchestrator is the source of truth:
   Stages 0a (knowledge-base initialisation) and 0b (intake & triage) fold
   into the Stage-1 log entry; Stage 1 = mode-specific entry flow, 2 = specification, 3 = data,
   4 = estimation, 5 = sensibility, 6 = results, 7 = disposition. Every
   sub-skill that names a stage must match.
2. **Propose → confirm choreography.** Every decision-point skill must *propose
   a specific choice with rationale and wait for confirmation*, never
   blank-ask (design principle 3). Flag any skill that merely describes.
3. **Doc-vs-practice.** Spot-check that what the skills / CLAUDE.md claim
   matches what the code does: output paths, default hyperparameters, file
   locations, method availability.
4. **Orphaned / overlapping skills.** Is every skill in the orchestrator's
   invocation table reachable from a stage (or clearly marked optional/on
   request)? Do two skills run the same check without a stated division of
   labour?
5. **Terminology / vocabulary.** The same concept uses the same name across
   skills, e.g. the three `verified.status` levels
   (`draft | checked_against_paper | signed_off`), the mode labels (A/B/C).
6. **Source-priority agreement.** Every statement of the knowledge-source
   cascade uses the one canonical order, `user > model dictionary > knowledge
   base > parametric > data-driven`, with economic logic checked at every
   level, defined once in `specification/SKILL.md`.

### Step 3: Live execute smoke (prove the path, not just the prose)

Run one fast verified replication end-to-end and confirm it still passes its
paper targets:

```
python scripts/replications/blanchard_quah_1989/replicate.py
```

Expect a `6/6` paper-target comparison table and output files under the script's
co-located `outputs/`. A clean mechanical + semantic audit with a broken
execute path is still a broken framework.

### Step 4: Report & fix (propose, confirm)

Compile a prioritized findings list (**P1** misleads the agent or user · **P2**
real friction or latent contradiction · **P3** polish), each item with a
`file:line` and a one-line fix. Per design principle 3, **propose the fixes and
wait for the user's confirmation** before editing; then re-run Steps 1–3 to
green. Record the audit (date, findings, dispositions) in your own notes, and
update CONVENTIONS.md or docstrings where a finding changes behaviour.

## Adding new invariants

When the audit surfaces a new class of drift, decide where it belongs:

- **Mechanically checkable** (counts, references, parse-ability, paths, schema)
  → add a test to `tests/test_doc_consistency.py`. **Prefer this**: it runs
  continuously and can't be forgotten.
- **Judgement-only** (semantics, choreography, design coherence) → add a bullet
  to Step 2 above so the next on-demand audit covers it.

© bis-med-it · Apache-2.0 · bis-med-it/marco-BIS

Similar skills

All skills for Testing and QA →

From the same set12

The whole set →
advisor-protocol The advisor-protocol Claude Code skill sends difficult mathematical reasoning to a more capable model for review. It activates when an agent is unsure how to interpret a derivation or translate it into code. CX Codex +1 ★ 89repo stars econometric-diagnostics econometric-diagnostics is a Claude Code skill that checks VAR/BVAR residuals and model adequacy after estimation or on request to test autocorrelation, heteroskedasticity, normality, or structural breaks. CX Codex +1 ★ 89repo stars model-selection The model-selection Claude Code skill matches econometric models to a research question and available data. It runs when a user needs an approach or wants to check a model they have chosen. CX Codex +1 ★ 89repo stars research-log The research-log Claude Code skill records decisions and stages in an econometric analysis. It runs after another MaRCo@BIS skill finishes a stage or when asked for an audit trail, reproducibility log, or session summary. CX Codex +1 ★ 89repo stars knowledge-base The knowledge-base Claude Code skill helps an agent weigh general econometric knowledge against country-specific data and conventions. It activates when setting up an analysis or working with local sources and research. CX Codex +1 ★ 89repo stars model-dictionary The model-dictionary Claude Code skill searches a literature-based dictionary of econometric specifications. It joins model selection and requests to examine a paper for a new entry. CX Codex +1 ★ 89repo stars benchmarking The benchmarking Claude Code skill evaluates forecast performance and checks estimation results. Run it after a model fit or while preparing results for a paper. CX Codex +1 ★ 89repo stars results-presentation results-presentation — a Claude Code skill that turns econometric results into charts, tables, and written interpretation. It activates at the reporting stage or when results need to be presented or prepared for publication. CX Codex +1 ★ 89repo stars

Project sponsors

Companies that keep this analytics open