Skip to main content
skill CX Codex

llm-debugger — Claude Code skill: what it does, how to install

llm-debugger diagnoses LLM output failures, patches faulty prompts, refines tool schemas, and generates regression test cases.

MIT 259 installs ★ 64 updated 2026-01-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add patricio0312rev/skills --skill llm-debugger -a claude-code
Open on GitHub

What llm-debugger does

The skill inspects model outputs against a taxonomy of failure types. It spots hallucinations, broken formatting such as invalid JSON, missing required fields, length constraint violations, and improper tool invocations.

Once it isolates the root cause, it generates targeted prompt patches. These insert strict grounding instructions, explicit formatting structures, or step-by-step reasoning guidelines. For tool schemas, it marks mandatory fields and injects sample invocations to stop argument misuse.

It also writes regression tests using the failed payload and similar inputs to confirm the fix works. Without explicit expected criteria like format or required fields, root cause analysis relies solely on basic pattern checks.

What it helps with

  • Classify root causes of corrupted JSON and missing output fields
  • Patch prompts with grounding constraints to suppress hallucinations
  • Update tool schemas with required parameter definitions and examples
  • Create regression test suites from failed model completions

Example prompts for the agent

  • Diagnose why the model produced broken JSON and suggest a prompt fix
  • Check this completion for hallucinations and generate regression tests

How to install llm-debugger

1Skills CLI (Vercel)

npx skills add patricio0312rev/skills --skill llm-debugger -a claude-code

2Manually: put the files in place

~/.claude/skills/llm-debugger/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/llm-debugger/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about llm-debugger

How do I install llm-debugger in Claude Code?

Run npx skills add patricio0312rev/skills --skill llm-debugger -a claude-code in your terminal. This creates the skill configuration at ~/.claude/skills/llm-debugger/SKILL.md.

What failure types does llm-debugger detect?

It categorizes hallucinations, format violations, broken length constraints, reasoning errors, tool misuse, unexpected refusals, and incomplete outputs.

What information is required to run a diagnosis?

You provide the original prompt, the failed completion, and expected requirements such as output format or mandatory keys.

Source text

SKILL.md · MIT
---
name: llm-debugger
description: Diagnoses LLM output failures including hallucinations, constraint violations, format errors, and reasoning issues. Provides root cause classification, prompt fixes, tool improvements, and new test cases. Use for "debugging AI", "fixing prompts", "quality issues", or "output errors".
---

# LLM Debugger

Systematically diagnose and fix LLM output issues.

## Failure Taxonomy

```python
class FailureType(Enum):
    HALLUCINATION = "hallucination"
    FORMAT_VIOLATION = "format_violation"
    CONSTRAINT_BREAK = "constraint_break"
    REASONING_ERROR = "reasoning_error"
    TOOL_MISUSE = "tool_misuse"
    REFUSAL = "unexpected_refusal"
    INCOMPLETE = "incomplete_output"
```

## Root Cause Analysis

```python
def diagnose_failure(input: str, output: str, expected: dict) -> dict:
    """Identify why LLM output failed"""

    issues = []

    # Check format
    if expected.get("format") == "json":
        try:
            json.loads(output)
        except:
            issues.append({
                "type": FailureType.FORMAT_VIOLATION,
                "details": "Invalid JSON output"
            })

    # Check required fields
    if expected.get("required_fields"):
        for field in expected["required_fields"]:
            if field not in output:
                issues.append({
                    "type": FailureType.INCOMPLETE,
                    "details": f"Missing required field: {field}"
                })

    # Check constraints
    if expected.get("max_length"):
        if len(output) > expected["max_length"]:
            issues.append({
                "type": FailureType.CONSTRAINT_BREAK,
                "details": f"Output too long: {len(output)} > {expected['max_length']}"
            })

    # Check for hallucination indicators
    if contains_hallucination_markers(output):
        issues.append({
            "type": FailureType.HALLUCINATION,
            "details": "Contains fabricated information"
        })

    return {
        "has_issues": len(issues) > 0,
        "issues": issues,
        "primary_issue": issues[0] if issues else None
    }

def contains_hallucination_markers(output: str) -> bool:
    """Detect common hallucination patterns"""
    markers = [
        r'According to.*that doesn\'t exist',
        r'In \d{4}.*before that year',
        r'contradicts itself',
    ]
    return any(re.search(marker, output, re.IGNORECASE) for marker in markers)
```

## Prompt Fixes

````python
PROMPT_FIXES = {
    FailureType.FORMAT_VIOLATION: """
Add strict format instructions:
"Output MUST be valid JSON with this exact structure:
```json
{{"field1": "value", "field2": 123}}
````

Do not add any text before or after the JSON."
""",

    FailureType.HALLUCINATION: """

Add grounding instructions:
"Base your response ONLY on the provided context.
If information is not in the context, say 'I don't have that information.'
Never make up facts or details."
""",

    FailureType.CONSTRAINT_BREAK: """

Add explicit constraints:
"Your response must be:

- Maximum 200 words
- No code examples
- Professional tone only"
  """,
      FailureType.REASONING_ERROR: """
  Add step-by-step reasoning:
  "Think through this step by step:

1. First, identify...
2. Then, consider...
3. Finally, conclude..."
   """,
   }

def suggest_prompt_fix(diagnosis: dict, current_prompt: str) -> str:
"""Generate improved prompt based on diagnosis"""
primary_issue = diagnosis["primary_issue"]
if not primary_issue:
return current_prompt

    fix = PROMPT_FIXES[primary_issue["type"]]

    return f"""{current_prompt}

IMPORTANT INSTRUCTIONS:
{fix}
"""

````

## Tool Improvements

```python
def suggest_tool_fixes(diagnosis: dict, tool_schema: dict) -> dict:
    """Improve tool schema to prevent misuse"""

    fixes = {}

    for issue in diagnosis["issues"]:
        if issue["type"] == FailureType.TOOL_MISUSE:
            # Make required parameters more explicit
            fixes["parameters"] = {
                **tool_schema["parameters"],
                "description": "REQUIRED: " + tool_schema["parameters"].get("description", "")
            }

            # Add examples
            fixes["examples"] = [
                {
                    "params": {"query": "example search"},
                    "description": "Use for finding information"
                }
            ]

    return {**tool_schema, **fixes}
````

## Test Case Generation

```python
def generate_test_cases(diagnosis: dict, failed_input: str, failed_output: str) -> List[dict]:
    """Create regression test cases from failures"""

    test_cases = []

    # Test case for the specific failure
    test_cases.append({
        "input": failed_input,
        "expected_behavior": "Should not " + diagnosis["primary_issue"]["details"],
        "validation": lambda output: not has_same_issue(output, diagnosis),
    })

    # Edge cases based on failure type
    if diagnosis["primary_issue"]["type"] == FailureType.FORMAT_VIOLATION:
        test_cases.append({
            "input": failed_input,
            "validation": lambda output: is_valid_json(output),
            "description": "Output must be valid JSON"
        })

    # Similar inputs that might trigger same issue
    similar_inputs = generate_similar_inputs(failed_input)
    for inp in similar_inputs:
        test_cases.append({
            "input": inp,
            "validation": lambda output: not has_same_issue(output, diagnosis),
        })

    return test_cases
```

## Debugging Workflow

```python
def debug_llm_output(
    prompt: str,
    output: str,
    expected: dict,
    context: dict = {}
) -> dict:
    """Complete debugging workflow"""

    # 1. Diagnose issue
    diagnosis = diagnose_failure(prompt, output, expected)

    if not diagnosis["has_issues"]:
        return {"status": "ok", "diagnosis": diagnosis}

    # 2. Generate fixes
    fixed_prompt = suggest_prompt_fix(diagnosis, prompt)

    # 3. Create test cases
    test_cases = generate_test_cases(diagnosis, prompt, output)

    # 4. Test fix
    test_output = llm(fixed_prompt)
    fix_works = diagnose_failure(fixed_prompt, test_output, expected)["has_issues"] == False

    return {
        "status": "issues_found",
        "diagnosis": diagnosis,
        "fixes": {
            "prompt": fixed_prompt,
            "fix_verified": fix_works,
        },
        "test_cases": test_cases,
        "recommendations": generate_recommendations(diagnosis, context)
    }
```

## Interactive Debugging

```python
def interactive_debug():
    """Interactive debugging session"""
    print("LLM Debugger - Interactive Mode")

    prompt = input("Enter prompt: ")
    expected_output = input("Expected output format (json/text): ")

    output = llm(prompt)
    print(f"\nGenerated output:\n{output}\n")

    if input("Does this look correct? (y/n): ").lower() == 'n':
        diagnosis = diagnose_failure(prompt, output, {"format": expected_output})

        print("\nDiagnosis:")
        for issue in diagnosis["issues"]:
            print(f"- {issue['type'].value}: {issue['details']}")

        print("\nSuggested fix:")
        print(suggest_prompt_fix(diagnosis, prompt))
```

## Best Practices

1. **Reproduce consistently**: Multiple runs
2. **Isolate variables**: Test prompt changes one at a time
3. **Check examples**: Are few-shot examples clear?
4. **Validate constraints**: Are they explicit enough?
5. **Test edge cases**: Boundary conditions
6. **Log everything**: Inputs, outputs, issues
7. **Iterate systematically**: Track what fixes work

## Output Checklist

- [ ] Failure taxonomy defined
- [ ] Root cause analyzer
- [ ] Prompt fix suggestions
- [ ] Tool schema improvements
- [ ] Test case generator
- [ ] Fix verification
- [ ] Debugging workflow
- [ ] Recommendations engine
- [ ] Interactive debugger
- [ ] Logging system

© patricio0312rev · MIT · patricio0312rev/skills

Similar skills

All skills for AI and ML →

From the same set148

The whole set →
db-performance-watchlist db-performance-watchlist is a Claude Code skill that tracks PostgreSQL metrics, including slow queries, connection pools, and unused indexes through Prisma. CX Codex +1 260installs api-contract-normalizer The api-contract-normalizer skill aligns API responses across endpoints, covering envelopes, errors, pagination, status codes, and versioning. Use it when API conventions have drifted. CX Codex +1 260installs vscode-rest-client-generator vscode-rest-client-generator generates .http files from Express, Next.js, and Fastify routes for the VS Code REST Client extension. Ideal for testing APIs in the editor without Postman. CX Codex +1 259installs embedding-pipeline-builder embedding-pipeline-builder builds document embedding workflows for loading, cleaning, chunking, indexing, and retrieval. Useful when setting up RAG preprocessing or semantic search. CX Codex +1 259installs tool-function-schema-designer tool-function-schema-designer turns a rough tool idea into a JSON schema, a typed interface, a validation layer and example calls the model can actually follow. CX Codex +1 258installs test-reporting-triage-skill test-reporting-triage-skill sorts test failures into categories, suggests likely owners and lists next steps. Useful when you need a structured readout of a test run. CX Codex +1 257installs mocking-assistant mocking-assistant generates reliable mock setups for APIs, services, and UI components using MSW conventions and Vitest testing patterns. CX Codex +1 256installs service-layer-extractor service-layer-extractor refactors bloated route handlers into dedicated services and repositories. Separates business logic from HTTP transport and simplifies unit testing. CX Codex +1 255installs

Project sponsors

Companies that keep this analytics open