absolute-deflake — Claude Code skill for flaky tests
The absolute-deflake Claude Code skill finds tests with inconsistent results and targets the cause of each failure. It activates on requests about flaky CI or tests that fail intermittently.
MIT 165 installs ★ 224 updated 2026-10-26
npx skills add dhruvjainin/absolute --skill absolute-deflake -a claude-codeWhat absolute-deflake does
The skill repeats test runs, shuffles their order, and compares isolated results with full or parallel runs. It traces failures to a cause and checks the proposed fix; retries, skipped tests, and sleep are not treated as fixes.
For Jest and Vitest, it repeats the suite and randomizes ordering. For pytest, it uses pytest-randomly and pytest-repeat; for Go, it runs with -count=N, -shuffle=on, and -race. When preferences.health.deflakeRuns is unset, the repeat count defaults to 20. The workflow uses the shared process in references/health-engine.md.
It records a failure rate for each suspect test, then reports the suspected cause and fix. After a change, the skill reruns tests many times with randomized runs and checks the full suite. If the failure points to a race in product code, it flags the case for a code fix rather than masking it in the test.
What it helps with
- find tests that fail intermittently
- trace failures to test order or parallel execution
- identify races, leaked resources, or shared state
- remove
retry,skip, andflakyannotations that hide failures
Example prompts for the agent
- Find and fix tests that fail randomly in CI.
- Work out why this test passes alone but fails in the full suite.
- Investigate the unstable tests marked with
retry.
How to install absolute-deflake
1Skills CLI (Vercel)
npx skills add dhruvjainin/absolute --skill absolute-deflake -a claude-code2Manually: put the files in place
~/.claude/skills/absolute-deflake/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/absolute-deflake/
1Skills CLI (Vercel)
npx skills add dhruvjainin/absolute --skill absolute-deflake -a codex2Manually: put the files in place
.agents/skills/absolute-deflake/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add dhruvjainin/absolute --skill absolute-deflake -a cursor2Manually: put the files in place
.cursor/skills/absolute-deflake/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install dhruvjainin/absolute skills/absolute-deflake2Manually: put the files in place
.github/skills/absolute-deflake/SKILL.md1Skills CLI (Vercel)
npx skills add dhruvjainin/absolute --skill absolute-deflake -a gemini-cli2Platform CLI
gemini skills install https://github.com/dhruvjainin/absolute --path skills/absolute-deflakeManually: put the files in place
.devin/skills/absolute-deflake/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add dhruvjainin/absolute --skill absolute-deflake -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about absolute-deflake
How do I install absolute-deflake in Claude Code?
Run npx skills add dhruvjainin/absolute --skill absolute-deflake -a claude-code. The skill file is at ~/.claude/skills/absolute-deflake/SKILL.md.
What does absolute-deflake do?
It reruns inconsistent tests, diagnoses why they fail, and checks whether a fix holds across repeated runs. It addresses the source of nondeterminism instead of adding retries, skipping tests, or inserting delays.
How does absolute-deflake detect flaky tests?
It compares repeated runs with shuffled order, isolated runs, and full or parallel runs. The commands differ for Jest and Vitest, pytest, and Go.
Source text
SKILL.md · MIT
--- name: absolute-deflake version: 0.5.0 description: > Flaky test fixes: detect nondeterministic tests empirically (repeat/shuffle/parallel runs), diagnose the root cause, fix it — never retry/skip/sleep — and verify across many randomized runs. Triggers on "absolute deflake", "fix flaky tests", "CI is flaky", "this test fails randomly/intermittently". category: workflow tags: - workflow - testing - flaky-tests - maintenance platforms: - claude-code - gemini-cli - openai-codex - mcp user-invocable: true argument-hint: "[target]" license: MIT maintainers: - github: maddhruv --- > Start your first response with the 🧪 emoji. ## Absolute Deflake Find tests that pass and fail nondeterministically, diagnose the **root cause** of each, and fix it — not by retrying or skipping, but by removing the source of nondeterminism. Output is evidence (failure rate per test) → cause → fix, verified by repeated runs. Runs the shared engine in **`references/health-engine.md`** — read it for the DETECT → SCAN → TRIAGE → FIX → VERIFY → REPORT loop and the safety contract. This file covers only what's specific to flaky tests. --- ## When to use - "Our CI is flaky", "this test fails randomly", "fix the intermittent failures". - A test passes locally but fails in CI (or vice versa), or fails ~1 in N runs. - Burning down a backlog of `retry`/`skip`-marked tests that mask real flakiness. Not for tests that fail *deterministically* — that's a real bug or a real regression (`/absolute work` for a fix, or just fix it). `deflake` targets *nondeterministic* failures. --- ## What it scans Establish flakiness empirically — a test isn't flaky because someone said so. Use `preferences.health.deflakeRuns` from config as the default N for repeat-runs (else 20): | Ecosystem | Repeat-run / detect | |---|---| | Jest/Vitest | run suite N× (`--run` loop), randomize order (`--shuffle` / `testSequencer`) | | pytest | `pytest-randomly` + `pytest --count=N` (`pytest-repeat`); `-p no:randomly` to A/B | | Go | `go test -count=N -shuffle=on ./...`, `-race` | Also mine signals: existing `retry`/`flaky`/`skip` annotations, CI history if reachable, and run the suite both **in isolation** and **in full/parallel** — order- and concurrency- dependent failures only show one way. Record a **failure rate** per suspect test. --- ## Common root causes (diagnose, don't guess) | Cause | Tell | Fix | |---|---|---| | Test-order / shared state | passes alone, fails in suite (or vice versa) | isolate state; reset/teardown between tests | | Time / clock | fails near midnight, DST, or under load | fake timers / inject clock; no real `sleep` | | Async race / missing await | fails under parallelism or slow CI | await the actual condition; no fixed timeouts | | Randomness | fails ~X% with no pattern | seed the RNG; fix the seed in tests | | Network / external I/O | fails offline or on slow links | mock/stub the boundary | | Unordered collections | fails on map/set iteration order | sort before asserting | | Resource leak / port reuse | fails on repeat or parallel runs | unique resources; clean up | --- ## Risk ranking (TRIAGE) | Wave | Class | Default | |---|---|---| | 1 | clear, isolated cause (seed, await, fake clock, sort) | fix now | | 2 | shared-state / ordering — needs fixture refactor | fix this pass, per test | | 3 | flakiness pointing at a **real product race**, not just the test | gated — surface; may be a genuine bug to fix in code | A flaky test sometimes means the *code* has a race, not the test. Don't "stabilize" the test into hiding a real concurrency bug — flag wave-3 cases for a real fix. --- ## Fix & verify - Fix the **cause**. Then prove it: re-run the test **many times** (and shuffled / parallel / with `-race`) — green once is not deflaked; green across N randomized runs is. - Remove the `retry`/`skip`/`flaky` annotation that was masking it once the cause is fixed. - **Never** "fix" by adding retries, raising timeouts blindly, `sleep`, or skipping the test — that hides flakiness, doesn't remove it. - Re-run the **full** suite to confirm the fix didn't destabilize neighbors. --- ## Gotchas 1. **Retry/skip as a fix.** Masks the flake, ships the nondeterminism. Forbidden here. 2. **`sleep` to dodge a race.** Slows the suite and still flakes under load. Await the condition. 3. **One green run = done.** Flakes are probabilistic — verify with many randomized runs. 4. **Stabilizing a real product race.** If the *code* races, fix the code, not just the assertion. 5. **Ignoring order/parallel dimension.** Run isolated *and* in-suite; the bug hides in whichever you skip. --- ## Companion commands - **`/absolute upgrade`** — a flaky suite makes upgrade verification unreliable; deflake first. - **`/absolute debt`** — flaky-test annotations are test debt; this clears them at the root. - **`/absolute work`** — when the flake is a genuine product-code race needing real design.
© dhruvjainin · MIT · dhruvjainin/absolute