Skip to main content
skill CX Codex

evaluation-harness — Claude Code skill for LLM eval and regressions

evaluation-harness — a skill for Claude Code and five other agent platforms that sets up repeatable LLM evaluation: golden datasets, scoring rubrics, pass/fail thresholds, and regression reports.

MIT 275 installs ★ 64 updated 2026-01-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add patricio0312rev/skills --skill evaluation-harness -a claude-code
Open on GitHub

What evaluation-harness does

It runs in Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. The skill provides a framework where test cases live in JSON with input, expected_output, and a rubric assigning a weight to each metric. Scoring functions range from exact string match to embedding similarity and an LLM judge that grades against rubric criteria.

The EvaluationHarness runner feeds the dataset through your model, records results, and checks category thresholds. For code_generation, correctness needs at least 0.9 and style at least 0.7. If any metric falls short, the test fails.

The regression report compares two runs — baseline and current — and flags regressions, improvements, and unchanged tests. ci_evaluation runs on every commit and exits with sys.exit(1) when a regression appears. The output checklist asks for 50+ examples in the dataset.

What it helps with

  • build a golden dataset with expected outputs and rubrics
  • set up several scoring metrics instead of one
  • define pass/fail thresholds per category
  • catch regressions after changing a model or prompt
  • wire evaluation into CI and fail on regressions

Example prompts for the agent

  • Create a golden dataset of 50 examples for my summarization task, split by category, and add rubrics
  • Write a runner that runs the current model against the dataset and compares it with the baseline from baseline_results.json
  • Set up CI so evaluation runs on every commit and exits non-zero when regressions show up

How to install evaluation-harness

1Skills CLI (Vercel)

npx skills add patricio0312rev/skills --skill evaluation-harness -a claude-code

2Manually: put the files in place

~/.claude/skills/evaluation-harness/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/evaluation-harness/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about evaluation-harness

How do I install evaluation-harness in Claude Code?

Run npx skills add patricio0312rev/skills --skill evaluation-harness -a claude-code, then the skill lands in ~/.claude/skills/evaluation-harness/SKILL.md.

What does evaluation-harness do?

It builds an evaluation system for LLM apps: a golden dataset with expected outputs, scoring functions, pass thresholds, a test runner, and a regression report between runs.

Which scoring metrics are included?

Four: exact string match, semantic similarity via cosine distance of embeddings, percentage of required keywords found, and LLM-as-judge scoring against rubric criteria.

Source text

SKILL.md · MIT
---
name: evaluation-harness
description: Builds repeatable evaluation systems with golden datasets, scoring rubrics, pass/fail thresholds, and regression reports. Use for "LLM evaluation", "testing AI systems", "quality assurance", or "model benchmarking".
---

# Evaluation Harness

Build systematic evaluation frameworks for LLM applications.

## Golden Dataset Format

```json
[
  {
    "id": "test_001",
    "category": "code_generation",
    "input": "Write a Python function to reverse a string",
    "expected_output": "def reverse_string(s: str) -> str:\n    return s[::-1]",
    "rubric": {
      "correctness": 1.0,
      "style": 0.8,
      "documentation": 0.5
    },
    "metadata": {
      "difficulty": "easy",
      "tags": ["python", "strings"]
    }
  }
]
```

## Scoring Rubrics

```python
from typing import Dict, Any

def score_exact_match(actual: str, expected: str) -> float:
    """Binary score: 1.0 if exact match, 0.0 otherwise"""
    return 1.0 if actual.strip() == expected.strip() else 0.0

def score_semantic_similarity(actual: str, expected: str) -> float:
    """Cosine similarity of embeddings"""
    actual_emb = get_embedding(actual)
    expected_emb = get_embedding(expected)
    return cosine_similarity(actual_emb, expected_emb)

def score_contains_keywords(actual: str, keywords: List[str]) -> float:
    """Percentage of required keywords present"""
    found = sum(1 for kw in keywords if kw.lower() in actual.lower())
    return found / len(keywords)

def score_with_llm(actual: str, expected: str, rubric: Dict[str, float]) -> Dict[str, float]:
    """Use LLM as judge"""
    prompt = f"""
    Grade this output on a scale of 0-1 for each criterion:

    Expected: {expected}
    Actual: {actual}

    Criteria: {', '.join(rubric.keys())}

    Return JSON with scores.
    """
    return json.loads(llm(prompt))
```

## Test Runner

```python
class EvaluationHarness:
    def __init__(self, dataset_path: str):
        self.dataset = self.load_dataset(dataset_path)
        self.results = []

    def run_evaluation(self, model_fn):
        for test_case in self.dataset:
            # Generate output
            actual = model_fn(test_case["input"])

            # Score
            scores = self.score_output(
                actual,
                test_case["expected_output"],
                test_case["rubric"]
            )

            # Record result
            self.results.append({
                "test_id": test_case["id"],
                "category": test_case["category"],
                "scores": scores,
                "passed": self.check_threshold(scores, test_case),
                "actual_output": actual,
            })

        return self.generate_report()

    def score_output(self, actual, expected, rubric):
        return {
            "exact_match": score_exact_match(actual, expected),
            "semantic_similarity": score_semantic_similarity(actual, expected),
            **score_with_llm(actual, expected, rubric)
        }

    def check_threshold(self, scores, test_case):
        min_scores = test_case.get("min_scores", {})
        for metric, threshold in min_scores.items():
            if scores.get(metric, 0) < threshold:
                return False
        return True
```

## Thresholds & Pass Criteria

```python
# Define thresholds per category
THRESHOLDS = {
    "code_generation": {
        "correctness": 0.9,
        "style": 0.7,
    },
    "summarization": {
        "semantic_similarity": 0.8,
        "brevity": 0.7,
    },
    "classification": {
        "exact_match": 1.0,
    }
}

def check_test_passed(result: Dict) -> bool:
    category = result["category"]
    thresholds = THRESHOLDS.get(category, {})

    for metric, threshold in thresholds.items():
        if result["scores"].get(metric, 0) < threshold:
            return False

    return True
```

## Regression Report

```python
def generate_regression_report(baseline_results, current_results):
    report = {
        "summary": {},
        "regressions": [],
        "improvements": [],
        "unchanged": 0
    }

    for baseline, current in zip(baseline_results, current_results):
        assert baseline["test_id"] == current["test_id"]

        baseline_passed = baseline["passed"]
        current_passed = current["passed"]

        if baseline_passed and not current_passed:
            report["regressions"].append({
                "test_id": baseline["test_id"],
                "category": baseline["category"],
                "baseline_scores": baseline["scores"],
                "current_scores": current["scores"],
            })
        elif not baseline_passed and current_passed:
            report["improvements"].append(baseline["test_id"])
        else:
            report["unchanged"] += 1

    report["summary"] = {
        "total_tests": len(baseline_results),
        "regressions": len(report["regressions"]),
        "improvements": len(report["improvements"]),
        "unchanged": report["unchanged"],
    }

    return report
```

## Continuous Evaluation

```python
# Run evaluation on every commit
def ci_evaluation():
    harness = EvaluationHarness("golden_dataset.json")
    results = harness.run_evaluation(production_model)

    # Check for regressions
    baseline = load_baseline("baseline_results.json")
    report = generate_regression_report(baseline, results)

    # Fail CI if regressions
    if report["summary"]["regressions"] > 0:
        print(f"❌ {report['summary']['regressions']} regressions detected!")
        sys.exit(1)

    print("✅ All tests passed!")
```

## Best Practices

1. **Representative dataset**: Cover edge cases
2. **Multiple metrics**: Don't rely on one score
3. **Human validation**: Review LLM judge scores
4. **Version datasets**: Track changes over time
5. **Automate in CI**: Catch regressions early
6. **Regular updates**: Add new test cases

## Output Checklist

- [ ] Golden dataset created (50+ examples)
- [ ] Multiple scoring functions
- [ ] Pass/fail thresholds defined
- [ ] Test runner implemented
- [ ] Regression comparison
- [ ] Report generation
- [ ] CI integration
- [ ] Baseline established

© patricio0312rev · MIT · patricio0312rev/skills

Similar skills

All skills for AI and ML →

From the same set148

The whole set →
insomnia-collection-generator insomnia-collection-generator scans Express, Fastify, and Next.js routes to generate an Insomnia v4 collection export file with organized workspaces, folders, and environments. CX Codex +1 277installs security-pr-checklist-skill security-pr-checklist-skill builds a repeatable security checklist for pull requests. Use it to structure code reviews, required checks, and CI security gates. CX Codex +1 276installs design-to-component-translator The design-to-component-translator skill turns Figma specs into React components with design tokens, responsive rules, and interaction states for implementing designs in code. CX Codex +1 276installs secrets-env-manager secrets-env-manager validates environment variables in CI workflows, enforces secret masking, and catches leaked keys early. CX Codex +1 276installs backend-latency-profiler-helper backend-latency-profiler-helper tracks down slow API endpoints, lists likely causes, and puts the fixes in priority order. CX Codex +1 275installs rbac-policy-tester rbac-policy-tester generates role permission matrices and automated authorization test suites to catch RBAC regressions in CI. CX Codex +1 274installs etl-sync-job-builder etl-sync-job-builder generates TypeScript ETL pipelines with incremental watermark tracking, idempotent batch upserts, and retry backoff logic. CX Codex +1 274installs structured-logging-standardizer structured-logging-standardizer sets up structured JSON logging with request correlation IDs using AsyncLocalStorage and pino. Keeps backend logs queryable and connected. CX Codex +1 273installs

Project sponsors

Companies that keep this analytics open