break-it — skill for Claude Code: learn RAG through debugging
A skill that plants bugs in RAG pipelines and asks you to find them. Debugging teaches architecture faster than building from scratch, since you have to understand why things work to spot why they fail.
npx skills add TakaGoto/rag-learning-academy --skill break-it -a claude-codeWhat break-it does
Break-it generates a working-but-broken RAG pipeline and challenges you to locate the bug. Errors scale by difficulty: easy mode shows obviously wrong results; medium mode looks plausible but is incorrect; hard mode pairs two bugs that interact. The skill covers real failure modes: chunk sizes too small or with zero overlap, mismatched embedding models between indexing and search, context placed after the question, overly strict metadata filters, stale indexes.
Once you spot the issue, the skill explains why it causes the observed behavior, how common this bug is in production systems, and how to prevent it. Stuck after two attempts? Ask for hints—from pointing at the general area to spelling out the exact problem. Every session logs to progress/debug-log.md, letting you track which bug types you've mastered.
What it helps with
- Spot how bad chunking fragments context across boundaries
- Catch embedding model mismatches between index and query time
- Diagnose the lost-in-the-middle effect when context is placed after questions
- Identify stale indexes that don't reflect new documents
- Learn to read RAG code critically
Example prompts for the agent
- Give me a medium-difficulty RAG debugging challenge
- /break-it hard — I want two interacting bugs
- Why is my retrieval returning irrelevant results?
How to install break-it
1Skills CLI (Vercel)
npx skills add TakaGoto/rag-learning-academy --skill break-it -a claude-code2Manually: put the files in place
~/.claude/skills/break-it/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/break-it/
1Skills CLI (Vercel)
npx skills add TakaGoto/rag-learning-academy --skill break-it -a codex2Manually: put the files in place
.agents/skills/break-it/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add TakaGoto/rag-learning-academy --skill break-it -a cursor2Manually: put the files in place
.cursor/skills/break-it/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install TakaGoto/rag-learning-academy break-it2Manually: put the files in place
.github/skills/break-it/SKILL.md1Skills CLI (Vercel)
npx skills add TakaGoto/rag-learning-academy --skill break-it -a gemini-cli2Platform CLI
gemini skills install https://github.com/TakaGoto/rag-learning-academy --path .claude/skills/break-itManually: put the files in place
.devin/skills/break-it/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add TakaGoto/rag-learning-academy --skill break-it -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about break-it
How do I install break-it in Claude Code?
Run npx skills add TakaGoto/rag-learning-academy --skill break-it -a claude-code, then open ~/.claude/skills/break-it/SKILL.md. For Cursor, use npx skills add TakaGoto/rag-learning-academy --skill break-it -a cursor. For GitHub Copilot, run gh skill install TakaGoto/rag-learning-academy break-it.
What does break-it do?
It creates a functional but incorrect RAG pipeline and asks you to find the bug. Errors touch chunking, embeddings, retrieval, generation, or pipeline logic. After you identify the issue, it explains the root cause and how to prevent similar problems.
Can I choose the difficulty level?
Yes. Use /break-it easy, /break-it medium, or /break-it hard. Easy has one obvious bug; medium has one subtle one; hard combines two bugs that amplify each other.
Source text
SKILL.md · MIT
--- name: break-it description: "Find the bug — learn RAG by debugging intentionally broken pipelines" --- # Break It: Learn by Debugging Introduce a realistic bug into a RAG pipeline and challenge the learner to find and fix it. Debugging teaches more than building because you have to understand *why* things work to figure out *why they don't*. > **Language awareness:** See `.claude/LANGUAGE_AWARENESS.md`. ## Step 1: Choose a Difficulty If the learner specifies a difficulty (e.g., `/break-it hard`), use that. Otherwise, calibrate based on their progress: - **Easy** — One obvious bug. The pipeline runs but gives clearly wrong results. - **Medium** — One subtle bug. The pipeline runs and results look plausible but are wrong. - **Hard** — Two bugs that interact. Fixing one makes the other more visible. ## Step 2: Select a Bug Category Pick from these common RAG failure modes. Rotate through categories so the learner sees variety: ### Chunking Bugs - Chunk size set way too small (10 tokens) — fragments lose all context - Zero overlap — information at boundaries is lost - Wrong separator for the document type (splitting code on paragraphs) ### Embedding Bugs - Mismatched embedding models between indexing and querying - Missing query prefix for asymmetric models (e.g., no "query: " prefix for E5) - Embeddings not normalized, breaking cosine similarity ### Retrieval Bugs - Top-k set to 1 — missing relevant context spread across chunks - Wrong distance metric (L2 instead of cosine) - Metadata filter too restrictive — filtering out relevant results ### Generation Bugs - Context placed after the question (lost-in-the-middle effect) - No grounding instruction — model ignores context and hallucinates - Context window exceeded — retrieved chunks silently truncated ### Pipeline Bugs - Stale index — new documents added but not re-embedded - Document not chunked before embedding (entire doc as one vector) - Query embedded with a different model than the corpus ## Step 3: Present the Broken Pipeline Show the learner a complete, runnable pipeline with the bug(s) already in place. Include: 1. The code (with the bug hidden in plain sight) 2. Sample data 3. A query that exposes the bug 4. The actual output (showing the wrong behavior) 5. What the correct output should look like Say: "Something's wrong with this pipeline. The query should return [expected], but it's returning [actual]. Can you find the bug?" ## Step 4: Guide the Debugging Let the learner investigate. If they're stuck after 2 attempts, offer graduated hints: - **Hint 1** (gentle): Point to the general area — "The issue is in the retrieval step" or "Look at how the chunks are created" - **Hint 2** (specific): Narrow it down — "Compare the embedding model used for indexing vs querying" - **Hint 3** (direct): Explain the bug and why it causes the behavior ## Step 5: Explain and Connect Once they find the bug (or after hints), explain: 1. **Why** this bug causes the observed behavior 2. **How common** this bug is in real RAG systems 3. **How to prevent** it — what checks or tests would catch this early 4. **The concept** behind the fix — link to the relevant `/lesson` or `/explain` topic ## Step 6: Track and Suggest Log the debugging session to `progress/debug-log.md` with the date, bug type, difficulty, and whether they found it independently. Suggest next steps: - `/break-it [harder difficulty]` to level up - `/lesson` on the concept behind the bug - `/build` to implement the correct pattern from scratch
© TakaGoto · MIT · TakaGoto/rag-learning-academy
