prompt-regression-tester — Claude Code skill: what it does and install
prompt-regression-tester compares prompt revisions across structured test suites, detects regressions, and drafts targeted fixes.
MIT 265 installs ★ 64 updated 2026-01-12
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a claude-codeWhat prompt-regression-tester does
The skill evaluates old and new prompt variants against JSON test suites. Each test case can validate must_include terms, forbidden tokens in must_not_include, output format structure, and target length boundaries.
Under the hood, it calculates quantitative stability indicators: semantic similarity across runs, structural retention, and constraint satisfaction. The breakage logic flags a failure whenever word count deviates by more than 30% from expected_length or required keywords vanish.
Execution concludes with a markdown report that groups regressions by failure pattern and outputs concrete phrasing suggestions to restore expected behavior.
What it helps with
- detect behavioral drift when updating system prompts
- verify adherence to required and prohibited keyword lists in LLM responses
- flag formatting violations and text length deviations across revisions
- generate automated markdown regression reports with prompt fix advice
Example prompts for the agent
- Compare
prompts/v1.txtagainstprompts/v2.txtusing cases intests.json - Analyze breakages in the new summarization prompt and suggest fixes
- Calculate stability metrics and format adherence for the updated prompt
How to install prompt-regression-tester
1Skills CLI (Vercel)
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a claude-code2Manually: put the files in place
~/.claude/skills/prompt-regression-tester/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/prompt-regression-tester/
1Skills CLI (Vercel)
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a codex2Manually: put the files in place
.agents/skills/prompt-regression-tester/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a cursor2Manually: put the files in place
.cursor/skills/prompt-regression-tester/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install patricio0312rev/skills ai-engineering/prompt-regression-tester/SKILL.md2Manually: put the files in place
.github/skills/prompt-regression-tester/SKILL.md1Skills CLI (Vercel)
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a gemini-cli2Platform CLI
gemini skills install https://github.com/patricio0312rev/skills --path ai-engineering/prompt-regression-testerManually: put the files in place
.devin/skills/prompt-regression-tester/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add patricio0312rev/skills --skill prompt-regression-tester -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about prompt-regression-tester
How to install prompt-regression-tester in Claude Code?
Run npx skills add patricio0312rev/skills --skill prompt-regression-tester -a claude-code in your terminal.
What does prompt-regression-tester do?
It benchmarks prompt versions against predefined test cases, tracking semantic consistency, format compliance, and specific failures.
How does the skill detect a test breakage?
It identifies missing required keywords, occurrences of forbidden terms, schema format breaches, or word counts deviating over 30% from target values.
Source text
SKILL.md · MIT
---
name: prompt-regression-tester
description: Compares old vs new prompts across test cases with diff summaries, stability metrics, breakage analysis, and fix suggestions. Use for "prompt testing", "A/B testing prompts", "prompt versioning", or "quality regression".
---
# Prompt Regression Tester
Systematically test prompt changes to prevent regressions.
## Test Case Format
```json
{
"test_cases": [
{
"id": "test_001",
"input": "Summarize this article",
"context": "Article text here...",
"expected_behavior": "Concise 2-3 sentence summary",
"baseline_output": "Output from v1.0 prompt",
"must_include": ["main point", "conclusion"],
"must_not_include": ["opinion", "speculation"]
}
]
}
```
## Comparison Framework
```python
def compare_prompts(old_prompt, new_prompt, test_cases):
results = {
"test_cases": [],
"summary": {
"total": len(test_cases),
"improvements": 0,
"regressions": 0,
"unchanged": 0,
},
"breakages": []
}
for test in test_cases:
old_output = llm(old_prompt.format(**test))
new_output = llm(new_prompt.format(**test))
comparison = {
"test_id": test["id"],
"old_output": old_output,
"new_output": new_output,
"diff": compute_diff(old_output, new_output),
"scores": {
"old": score_output(old_output, test),
"new": score_output(new_output, test),
},
"verdict": classify_change(old_output, new_output, test)
}
results["test_cases"].append(comparison)
results["summary"][comparison["verdict"]] += 1
if comparison["verdict"] == "regressions":
results["breakages"].append(analyze_breakage(comparison, test))
return results
```
## Stability Metrics
```python
def calculate_stability_metrics(results):
return {
"output_stability": measure_output_consistency(results),
"format_stability": check_format_preservation(results),
"constraint_adherence": check_constraints(results),
"behavioral_consistency": measure_behavior_delta(results),
}
def measure_output_consistency(results):
"""How similar are outputs between versions?"""
similarities = []
for result in results["test_cases"]:
sim = semantic_similarity(
result["old_output"],
result["new_output"]
)
similarities.append(sim)
return sum(similarities) / len(similarities)
```
## Breakage Analysis
```python
def analyze_breakage(comparison, test_case):
"""Identify why the new prompt failed"""
reasons = []
new_out = comparison["new_output"]
# Missing required content
for keyword in test_case.get("must_include", []):
if keyword.lower() not in new_out.lower():
reasons.append(f"Missing required content: '{keyword}'")
# Contains forbidden content
for keyword in test_case.get("must_not_include", []):
if keyword.lower() in new_out.lower():
reasons.append(f"Contains forbidden content: '{keyword}'")
# Format violations
if not check_format(new_out, test_case.get("expected_format")):
reasons.append("Output format violation")
# Length issues
expected_length = test_case.get("expected_length")
if expected_length:
actual_length = len(new_out.split())
if abs(actual_length - expected_length) > expected_length * 0.3:
reasons.append(f"Length deviation: expected ~{expected_length}, got {actual_length}")
return {
"test_id": test_case["id"],
"reasons": reasons,
"old_output": comparison["old_output"][:100],
"new_output": comparison["new_output"][:100],
}
```
## Fix Suggestions
```python
def suggest_fixes(breakages):
"""Generate fix suggestions based on breakage patterns"""
suggestions = []
# Group breakages by reason
reason_groups = {}
for breakage in breakages:
for reason in breakage["reasons"]:
if reason not in reason_groups:
reason_groups[reason] = []
reason_groups[reason].append(breakage["test_id"])
# Generate suggestions
for reason, test_ids in reason_groups.items():
if "Missing required content" in reason:
suggestions.append({
"issue": reason,
"affected_tests": test_ids,
"suggestion": "Add explicit instruction in prompt to include this content",
"example": f"Make sure to mention {reason.split(':')[1]} in your response."
})
elif "format violation" in reason:
suggestions.append({
"issue": reason,
"affected_tests": test_ids,
"suggestion": "Add stricter format constraints to prompt",
"example": "Output must follow this exact format: ..."
})
return suggestions
```
## Report Generation
```markdown
# Prompt Regression Report
## Summary
- **Total tests:** 50
- **Improvements:** 5 (10%)
- **Regressions:** 3 (6%)
- **Unchanged:** 42 (84%)
## Stability Metrics
- **Output stability:** 0.87
- **Format stability:** 0.95
- **Constraint adherence:** 0.94
## Regressions (3)
### test_005: Missing required content
**Old output:** "The main benefit is cost savings..."
**New output:** "This approach provides flexibility..."
**Issue:** Missing required keyword 'cost'
**Fix:** Add explicit instruction: "Mention cost implications in your response"
## Recommended Actions
1. Revert changes that caused regressions (tests: 005, 012, 023)
2. Add format constraints for JSON output
3. Run full test suite before deployment
```
## Best Practices
- Test with diverse inputs
- Compare across multiple runs (LLMs are stochastic)
- Track metrics over time
- Automate in CI/CD
- Review all regressions before deploy
- Maintain test case library
## Output Checklist
- [ ] Test cases defined (30+)
- [ ] Comparison runner
- [ ] Stability metrics
- [ ] Breakage analyzer
- [ ] Fix suggestions
- [ ] Diff visualizer
- [ ] Automated report
- [ ] CI integration
© patricio0312rev · MIT · patricio0312rev/skills