Skip to main content
skill CX Codex

golden-dataset — Claude Code skill: what it does, how to install

golden-dataset is a Claude Code skill for evaluation dataset lifecycle patterns, covering curation, versioning, quality thresholds, and Langfuse experiment runs.

MIT 193 installs ★ 305 updated 2026-11-14

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add yonatangross/orchestkit --skill golden-dataset -a claude-code
Open on GitHub

What golden-dataset does

This skill supplies patterns for managing evaluation datasets used in AI and ML testing. It splits the workflow into six rule files across curation, storage management, regression checks, and a 9-phase document intake workflow. The agent loads individual instructions from rules/ on demand.

The framework enforces strict quality gates: a minimum score of 0.70 for entry inclusion, a 0.65 threshold for automatic addition, and duplication blocking at 0.90 similarity (with warnings starting at 0.85). Every document must carry at least two domain tags and three test queries spanning four difficulty tiers. It also specifies how to orchestrate runs using Langfuse client libraries, catching regression errors to fail CI builds.

Keep in mind that the manifest sets user-invocable: false and disables model invocation, which means it guides background agent behavior rather than exposing a direct slash command.

What it helps with

  • validate incoming documents against quality thresholds before dataset inclusion
  • enforce difficulty distribution across trivial, easy, medium, and hard queries
  • structure portable JSON dataset backups excluding regenerated embeddings
  • run regression experiments in CI using Langfuse evaluators

Example prompts for the agent

  • Check this candidate document against the golden dataset intake rules
  • Help balance query difficulties across our evaluation test cases
  • Configure a Langfuse dataset experiment runner with custom evaluators

How to install golden-dataset

1Skills CLI (Vercel)

npx skills add yonatangross/orchestkit --skill golden-dataset -a claude-code

2Manually: put the files in place

~/.claude/skills/golden-dataset/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/golden-dataset/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about golden-dataset

How to install golden-dataset in Claude Code?

Run npx skills add yonatangross/orchestkit --skill golden-dataset -a claude-code in your terminal. The configuration is written to ~/.claude/skills/golden-dataset/SKILL.md.

What does golden-dataset do?

It establishes standard engineering practices for maintaining evaluation datasets, covering multi-agent curation, version control rules, bias detection, and automated regression testing.

Can golden-dataset be invoked directly via a prompt command?

No, the skill configuration sets user-invocable: false and is assigned to the data-pipeline-engineer agent persona to automate data workflows behind the scenes.

Source text

SKILL.md · MIT
---
name: golden-dataset
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Golden dataset lifecycle patterns for curation, versioning, quality validation, and CI integration. Use when building evaluation datasets, managing dataset versions, validating quality scores, or integrating golden tests into pipelines.
user-invocable: false
metadata:
  owner-agent: "data-pipeline-engineer"
  category: document-asset-creation
  version: "2.0.0"
  author: "OrchestKit"
  complexity: "medium"
  tags: "golden-dataset, evaluation, dataset-curation, dataset-validation, quality, llm-testing"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---

# Golden Dataset

Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in `rules/` loaded on-demand.

## Quick Reference

| Category | Rules | Impact | When to Use |
| -------- | ----- | ------ | ----------- |
| [Curation](#curation) | 2 | HIGH | Content collection, annotation pipelines |
| [Management](#management) | 2 | HIGH | Versioning, backup/restore |
| [Validation](#validation) | 1 | CRITICAL | Regression testing |
| [Add Workflow](#add-workflow) | 1 | HIGH | 9-phase curation, quality scoring, bias detection, silver-to-gold |

Total: 6 rules across 4 categories. House thresholds and scars: `references/ork-delta.md`.

## Curation

Content collection, multi-agent annotation, and diversity analysis for golden datasets.

| Rule | File | Key Pattern |
| ---- | ---- | ----------- |
| Collection | `rules/curation-collection.md` | Content type classification, quality thresholds, duplicate prevention |
| Annotation | `rules/curation-annotation.md` | Multi-agent pipeline, consensus aggregation, Langfuse tracing |

Difficulty ladder, coverage floors, and duplicate thresholds: `references/ork-delta.md`.

## Management

Versioning, storage, and CI/CD automation for golden datasets.

| Rule | File | Key Pattern |
| ---- | ---- | ----------- |
| Versioning | `rules/management-versioning.md` | JSON backup format, embedding regeneration, disaster recovery |
| Storage | `rules/management-storage.md` | Backup strategies, URL contract, data integrity checks |

CI automation for backups is upstream's job; see "Upstream coverage" below.

## Validation

Quality scoring, drift detection, and regression testing for golden datasets.

| Rule | File | Key Pattern |
| ---- | ---- | ----------- |
| Regression | `rules/validation-regression.md` | Difficulty distribution, pre-commit hooks, full dataset validation |

Schema validation and duplicate detection are upstream's job (see "Upstream coverage"
below); the house thresholds they must enforce live in `references/ork-delta.md`.

## Add Workflow

Structured workflow for adding new documents to the golden dataset.

| Rule | File | Key Pattern |
| ---- | ---- | ----------- |
| Add Document | `rules/curation-add-workflow.md` | 9-phase curation, parallel quality analysis, bias detection |

## Quick Start Example

```python
async def validate_before_add(document: dict, source_url_map: dict) -> dict:
    """Pre-addition validation for golden dataset entries."""
    errors = []

    # 1. URL contract check
    if "placeholder" in document.get("source_url", ""):
        errors.append("URL must be canonical, not a placeholder")

    # 2. Content quality
    if len(document.get("title", "")) < 10:
        errors.append("Title too short (min 10 chars)")

    # 3. Tag requirements
    if len(document.get("tags", [])) < 2:
        errors.append("At least 2 domain tags required")

    return {"valid": len(errors) == 0, "errors": errors}
```

## Key Decisions

| Decision | Recommendation |
| -------- | -------------- |
| Backup format | JSON (version controlled, portable) |
| Embedding storage | Exclude from backup (regenerate on restore) |
| Quality threshold | >= 0.70 quality score for inclusion |
| Confidence threshold | >= 0.65 for auto-include |
| Duplicate threshold | >= 0.90 similarity blocks, >= 0.85 warns |
| Min tags per entry | 2 domain tags |
| Min test queries | 3 per document |
| Difficulty balance | Trivial 3, Easy 3, Medium 5, Hard 3 minimum |
| CI frequency | Weekly automated backup (Sunday 2am UTC) |

## Common Mistakes

1. Using placeholder URLs instead of canonical source URLs
2. Skipping embedding regeneration after restore
3. Not validating referential integrity between documents and queries
4. Over-indexing on articles (neglecting tutorials, research papers)
5. Missing difficulty distribution balance in test queries
6. Not running verification after backup/restore operations
7. Testing restore procedures in production instead of staging
8. Committing SQL dumps instead of JSON (not version-control friendly)

## Running a dataset as an experiment

Curating a dataset is half the job; the other half is running something against it and scoring the
result. Both Langfuse SDKs ship a runner, and their shapes differ.

**Python (SDK 4.x):** see `monitoring-observability/references/experiments-api.md`.

**JS/TS (SDK 5.x):** `@langfuse/client` exposes the runner directly on a fetched dataset.

```typescript
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();
const dataset = await langfuse.dataset.get("my-evaluation-dataset");

const result = await dataset.runExperiment({
  name: "Retrieval quality",
  task: myTask,               // (params) => Promise<any>
  evaluators: [myEvaluator],  // per-item: (params) => Promise<Evaluation | Evaluation[]>
});
```

| Type | Scores | Use for |
|---|---|---|
| `Evaluator` | one item | Per-example quality (faithfulness, relevance) |
| `RunEvaluator` | the whole run | Aggregate assertions — pass rate, mean score, regression checks |
| `Evaluation` | — | `{ name, value, comment?, metadata?, dataType?, configId? }` |

A per-item `Evaluator` cannot see the other items, so anything comparative belongs in a
`RunEvaluator`. `createEvaluatorFromAutoevals` wraps an autoevals scorer instead of hand-writing
one, and `RegressionError` is thrown when a run regresses against a configured baseline — catch it
to fail CI on a quality drop rather than only on an exception.

Full JS surface: `monitoring-observability/references/langfuse-js-v5.md`.

## Evaluations

See `test-cases.json` for 9 test cases across all categories.

## Upstream coverage (do not restate)

| Topic | First-party source |
| ----- | ------------------ |
| Dataset schema validation (JSON Schema, field constraints) | https://json-schema.org and https://zod.dev |
| Duplicate detection via embeddings, cosine similarity | https://github.com/pgvector/pgvector |
| Scheduled backup automation (cron workflows, commit bots) | https://docs.github.com/actions/using-workflows/events-that-trigger-workflows#schedule |
| Dataset runs, experiment scoring, annotation queues | https://langfuse.com/docs/datasets |
| Backup and restore mechanics for postgres datasets | https://www.postgresql.org/docs/current/backup.html |

House thresholds these must enforce: `references/ork-delta.md`.

## Related Skills

- `ork-extras:rag-retrieval` - Retrieval evaluation using golden dataset
- `ork-extras:monitoring-observability` - Langfuse tracing patterns for curation workflows
- `ork:testing-llm` - Evaluation harnesses that consume golden datasets
- `ork:testing-unit` - Unit testing patterns and strategies

## Capability Details

### curation

**Keywords:** golden dataset, curation, content collection, annotation, quality criteria

**Solves:**

- Classify document content types for golden dataset
- Run multi-agent quality analysis pipelines
- Generate test queries for new documents

### management

**Keywords:** golden dataset, backup, restore, versioning, disaster recovery

**Solves:**

- Backup and restore golden datasets with JSON
- Regenerate embeddings after restore
- Automate backups with CI/CD

### validation

**Keywords:** golden dataset, validation, schema, duplicate detection, quality metrics

**Solves:**

- Validate entries against document schema
- Detect duplicate or near-duplicate entries
- Analyze dataset coverage and distribution gaps

© OrchestKit · MIT · yonatangross/orchestkit

Similar skills

All skills for AI and ML →

From the same set86

The whole set →
demo-producer demo-producer — a skill for Claude Code automates showcase video creation for skills, CLIs, and code. Handles everything from terminal recording via VHS to video composition in Remotion. CX Codex +1 196installs configure Interactive wizard for OrchestKit settings: MCP servers, hook permissions, keybindings, presets. Runs on first install and via /orchestkit:configure. CX Codex +1 196installs async-jobs async-jobs is a Claude Code skill with engineering patterns and configurations for background task processing with Celery, ARQ, Redis, and Temporal. CX Codex +1 195installs product-frameworks This skill provides product frameworks for business analysis, prioritization, metrics, and research. Use it to shape a business case, PRD, or user research plan. CX Codex +1 194installs mcp-patterns mcp-patterns — a skill for Claude Code and Codex that guides Model Context Protocol server design. Activates when configuring transports, adding OAuth, wiring tool handlers, or hardening MCP security. CX Codex +1 191installs browser-tools browser-tools is a Claude Code skill that adds safety rules for browser automation with agent-browser, including scraping ethics, request pacing, local URLs, and authentication. It runs automatically during browser automation. CX Codex +1 191installs scope-appropriate-architecture scope-appropriate-architecture classifies projects into six tiers and picks patterns that fit their scope, preventing microservices for MVPs. Activates during design and complexity checks. CX Codex +1 179installs storybook-mcp-integration The storybook-mcp-integration Claude Code skill connects Storybook MCP tools to work with existing components. The agent can find component docs, open story previews, and run story tests. CX Codex +1 166installs

Project sponsors

Companies that keep this analytics open