Skip to main content
skill CX Codex

flaky-test-killer — Claude Code skill for flaky tests

The flaky-test-killer Claude Code skill tracks down flaky-test causes and helps fix them at the root. It stresses the test and compares failure rates before and after the change.

MIT ★ 541 updated 2026-11-09

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add dotslashsimran/simcraft --skill flaky-test-killer -a claude-code
Open on GitHub

What flaky-test-killer does

The skill measures how often a test fails, then varies its conditions: execution order, timezone, system clock, CPU load and temporary directory. When a test fails only in the full suite, a script narrows down the preceding tests to find the one that pollutes shared state. It compares the results with the initial measurements.

The included scripts use only the Python 3.9+ standard library. The skill rules out retries, sleeps and longer timeouts as fixes; it calls for measurements before and after a change. Finding a polluter depends on the order in which tests ran.

What it helps with

  • trace intermittent test failures to their cause
  • find a test that contaminates shared state
  • check whether timezone or execution order affects a test
  • compare failure rates before and after a fix

Example prompts for the agent

  • Find the cause of intermittent failures in test_checkout.
  • This test passes alone but fails in the full suite; find the test that contaminates it.
  • Check whether changing timezones affects this test's failure rate.

How to install flaky-test-killer

1Skills CLI (Vercel)

npx skills add dotslashsimran/simcraft --skill flaky-test-killer -a claude-code

2Manually: put the files in place

~/.claude/skills/flaky-test-killer/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/flaky-test-killer/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about flaky-test-killer

How do I install flaky-test-killer in Claude Code?

Run npx skills add dotslashsimran/simcraft --skill flaky-test-killer -a claude-code. The skill file is placed at ~/.claude/skills/flaky-test-killer/SKILL.md.

What does flaky-test-killer do?

It measures a test's failure rate, investigates conditions that trigger failures and helps verify a root-cause fix with new measurements. Retries and longer timeouts are not treated as fixes.

What does flaky-test-killer need to run?

Its scripts require Python 3.9 or later and use the Python standard library only.

Source text

SKILL.md · MIT
---
name: flaky-test-killer
description: Find why a test is flaky and fix the root cause, never by adding retries, sleeps or bigger timeouts. Measures the flake rate with a confidence interval, runs the test under stressors (random order, timezone, fake clock at month end or DST, CPU load, fresh temp dir, low file limits), bisects for the test that pollutes it, and proves the fix with before and after numbers. Use when the user says "flaky test", "test passes locally fails in CI", "intermittent failure", "random test failure", "works on my machine", "test fails sometimes", "only fails in the full suite", or wants to quarantine or retry a test.
---

# Flaky test killer

A flaky test is a real bug: in the test, in the code, or in the environment. Retries hide it, and the same bug will later show up in production or in another test. The job is to make the failure happen on demand, name the cause, fix it, and prove with numbers that it is gone.

Scripts are Python 3.9+ standard library only. `S=~/simcraft/skills/coding/flaky-test-killer/scripts`.

## Rules

- **No retries, no sleeps, no longer timeouts, no `skip`** as the fix. A wait is fine only when it waits for the actual condition (an event, a join, a poll with a deadline).
- **Measure before you change anything.** Without a "before" rate you cannot prove an "after".
- **One cause at a time.** Many flaky tests have two bugs. Fix one, measure, and look again.
- **Keep an evidence log** (see below). Every claim needs a command and a number.

## Steps

1. **Reproduce and measure.** Run the single test, then the whole suite, many times.
   ```
   python3 $S/flake_rate.py -n 50 --json before.json -- <test command>
   python3 $S/flake_rate.py -n 200 -j 8 --json before.json -- <suite command with shuffle seed {seed}>
   ```
   It prints the failure rate with a 95% Wilson interval, groups distinct failure messages, counts failures per test name, compares pass and fail durations, and gives cause hints. `{seed}` becomes a new random seed every run, and failing seeds are printed so you can replay the exact order. `--until-fail` stops at the first failure and shows the output. If the rate is 0 after 50 runs, go to step 2: the trigger is something your machine does not do.

2. **Run alone vs in the suite vs in parallel.** Fails only in the suite means order or shared state (go to step 4). Fails only with `-j` means a shared resource such as a port, file or database. Fails alone means time, randomness, a race, or the environment.

3. **Sweep the stressors** to see which one moves the rate:
   ```
   python3 $S/stress.py --sweep -k 5 -- <test command>
   ```
   It runs a baseline and then each stressor on its own: five timezones, nine fake clocks (midnight, month end, Feb 28, leap day, year end, both DST switches, 2038), heavy CPU load, a fresh temp dir, a low open-files limit and a Turkish locale. A stressor is flagged only when a Fisher exact test says it really changed the rate. If the baseline always fails, it tells you which setting makes it pass (say `tz=UTC`, the classic "passes in CI, fails locally") and which settings make it fail with a different message (a second, hidden bug). Pin the first cause and sweep again: `stress.py --sweep --tz UTC --only clock -- ...`. Use the same flags without `--sweep` for a single stressed run.

4. **Find the polluter** when the test passes alone but fails after others. Save the order of a failing run (the test ids that ran before the victim, in order) and bisect:
   ```
   python3 $S/find_polluter.py --tests order.txt --victim <id> --cmd "<runner> {tests}" [--fail-pattern "<victim's failure text>"] [--repeat 3]
   ```
   It checks the victim passes alone and fails after the list, then shrinks the list with delta debugging to the one test (or smallest group) that breaks it. If the victim fails alone and passes after the list, it finds the "state setter" the victim secretly depends on. Use `--fail-pattern` so another flaky test in the list does not confuse the search.

5. **Name the cause and fix it at the root.** Look up the failure signature in the cause table in [reference.md](reference.md). Each row gives the signature, the stressor that exposes it, and the right fix.

6. **Prove it.** Run the same command again with enough runs and compare:
   ```
   python3 $S/flake_rate.py -n 300 -j 8 --json after.json --compare before.json -- <same command>
   python3 $S/flake_rate.py --plan 0.01     # clean runs needed to claim rate < 1%
   ```
   "Fixed" means the Fisher test says the drop is real AND the stressor that exposed the bug now passes. To claim a rate below X with 95% confidence you need about 3/X clean runs in a row (299 for 1%). Ten green runs prove almost nothing for a 5% flake.

## Evidence log

Keep a short log in the PR or issue, one line per step:
```
before: 42/60 failed (70%, CI 57-80%) cmd: run_tests.py --seed {seed}
cause 1: polluter tests.test_users...test_enable_eu leaves SETTINGS["symbol"]="€" (find_polluter, 7 probes)
cause 2: test_export sleeps 10ms, worker takes 0-20ms (fails alone, rate 60%, not moved by any stressor)
fix: patch.dict around the polluter; victim sets its own state; worker returns an Event, test waits on it
after: 0/300 failed (rate < 1.0% at 95%), Fisher p = 7e-41; sweep clean on tz, clock, cpu
```

## Practise

`python3 ~/simcraft/skills/coding/flaky-test-killer/examples/make_examples.py --out /some/tmp/dir` makes a unittest project and a node:test project with an order dependency, a month end bug, a timezone bug, a sleep race and an unawaited promise. `--variant fixed` writes the fixed versions to compare against.

Per-ecosystem commands (pytest, jest, vitest, node:test, go, cargo, JUnit), the full cause taxonomy, and the statistics are in [reference.md](reference.md).

© dotslashsimran · MIT · dotslashsimran/simcraft

Similar skills

All skills for Testing and QA →

From the same set24

The whole set →
scripted-screencast The scripted-screencast Claude Code skill turns scripted browser actions into a product demo video with cursor effects, zooms, and captions. It activates for screencast requests and rerenders after interface changes. CX Codex +1 ★ 541repo stars security-threat-model The security-threat-model Claude Code skill maps threats from an app’s code and tests the highest risks locally. It activates for requests about STRIDE, app security reviews, or vulnerabilities before launch. CX Codex +1 ★ 541repo stars vox-explainer The vox-explainer Claude Code skill makes Vox-style explainer graphics or a faceless video. It runs on requests for voiceover-led visuals, animated quotes, or stats. CX Codex +1 ★ 541repo stars benchmark-honestly The benchmark-honestly Claude Code skill checks performance measurements and compares code versions. It activates for benchmark requests, load tests, latency percentiles, or claims that a change made code faster. CX Codex +1 ★ 541repo stars reverse-engineer-video The reverse-engineer-video Claude Code skill breaks down a reference clip into shots, camera moves and editing beats. Invoke it to inspect a cut, plan a remake or compare one with the original. CX Codex +1 ★ 541repo stars system-design-from-scratch system-design-from-scratch — a skill for Claude Code that builds minimal distributed architectures, verifies invariants under simulated node failures, and stress-tests them with real metrics. CX Codex +1 ★ 541repo stars paper-to-runnable The paper-to-runnable Claude Code skill turns a research paper into a minimal implementation of its central idea. It activates for requests to build or reproduce a paper, including from an arXiv link or local PDF. CX Codex +1 ★ 541repo stars codebase-map The codebase-map Claude Code skill maps an unfamiliar repository and traces a real request through it. Ask it to explain the project structure, locate a code path, or onboard a new teammate. CX Codex +1 ★ 541repo stars

Project sponsors

Companies that keep this analytics open