Skip to main content
skill CX Codex

scoring-checks — Claude Code & Codex skill: evaluate config quality

Adds deterministic scoring checks that measure AI agent config quality. Evaluates file structure, standards compliance, and configuration patterns — with no randomness or external calls.

MIT ★ 1,319 updated 2026-09-24

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add caliber-ai-org/ai-setup --skill scoring-checks -a claude-code
Open on GitHub

What scoring-checks does

This skill lets you build new checks in src/scoring/checks/ that grade config against a point system. Each check returns a Check[] array with an ID, name, category (existence, quality, grounding, accuracy, freshness, or bonus), and a numeric result. Points come from src/scoring/constants.ts, and checks integrate through filterChecksForTarget() in src/scoring/index.ts.

All checks run locally against the filesystem — no network requests or LLM inference. Each must be deterministic: the same disk state always produces the same result. Use this when adding new scoring criteria or extending the check logic in src/scoring/checks/.

What it helps with

  • add a new config quality check to the skill
  • create a deterministic evaluation for standards compliance
  • extend the point system for agent assessment
  • build a check that counts code blocks or validates references

Example prompts for the agent

  • Add a scoring check that validates CLAUDE.md exists and has tokens budgeted
  • Create a new check to measure code-block density in .cursorrules
  • Modify scoring criteria to grade grounding against actual project files

How to install scoring-checks

1Skills CLI (Vercel)

npx skills add caliber-ai-org/ai-setup --skill scoring-checks -a claude-code

2Manually: put the files in place

~/.claude/skills/scoring-checks/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/scoring-checks/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about scoring-checks

How do I install scoring-checks in Claude Code?

Run npx skills add caliber-ai-org/ai-setup --skill scoring-checks -a claude-code, then open ~/.claude/skills/scoring-checks/SKILL.md.

What does scoring-checks do?

The skill provides templates and guidance for writing new checks that grade AI agent config: file presence, structure quality, standards adherence. All checks are deterministic and filesystem-only.

Why must checks be deterministic?

Because the scoring system needs to produce the same grade for the same config every time. Checks cannot call external APIs, invoke LLMs, or make network requests—they read files only.

Source text

SKILL.md · MIT
---
name: scoring-checks
description: Add a new deterministic scoring check in src/scoring/checks/ that evaluates config quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes or refactoring scoring logic.
paths:
  - src/scoring/checks/**/*.ts
  - src/scoring/constants.ts
  - src/scoring/index.ts
---
# Adding a Scoring Check

Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.

## Critical

- **Check must be deterministic**: Same filesystem state → same result every time. No randomness, no external APIs.
- **Point values come from constants.ts**: Every `earnedPoints` and `maxPoints` must reference `POINTS_*` from `src/scoring/constants.ts`. Do NOT hardcode numbers.
- **Always return `Check[]` array**: Export a function `check<Category>(dir: string): Check[]` where category is one of: `existence`, `quality`, `grounding`, `accuracy`, `freshness`, `bonus`.
- **Every check must have**: `id` (kebab-case, unique), `name`, `category`, `maxPoints`, `earnedPoints`, `passed`, `detail`, and optional `suggestion`/`fix`.
- **Fix object fields**: `action` (string describing what to do), `data` (context for the fix), `instruction` (user-facing guidance).
- **Register in src/scoring/index.ts**: Add the import and spread the result into the `allChecks` array in `computeLocalScore()`.
- **Target filtering**: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate `*_ONLY_CHECKS` set in `constants.ts`.

## Instructions

### Step 1: Define point constants in src/scoring/constants.ts

Verify before proceeding: Is your check measurable with a numeric point value?

Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):

```typescript
// In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical

// If threshold-based, add a companion array:
export const YOUR_THRESHOLD_ARRAY = [
  { minValue: 10, points: 4 },
  { minValue: 5, points: 2 },
] as const;
```

Check existing patterns: Token budgets use `TOKEN_BUDGET_THRESHOLDS`, code blocks use `CODE_BLOCK_THRESHOLDS`, concreteness uses `CONCRETENESS_THRESHOLDS`.

Verify: Review `CATEGORY_MAX` object to ensure your check fits within its category's point budget.

### Step 2: Create or edit check function in src/scoring/checks/

Choose the file based on category. Each file exports a `check<Name>(dir: string): Check[]` function:

- `existence.ts` — files/directories exist (CLAUDE.md, .cursorrules, skills, MCP servers)
- `quality.ts` — config structure, size, clarity (code blocks, token budget, concreteness, duplicates)
- `grounding.ts` — references to actual project files/directory structure
- `accuracy.ts` — validity of references, git-based config drift
- `freshness.ts` — git commit-based staleness, secrets, permissions
- `bonus.ts` — hooks, learned content, OpenSkills format
- `sources.ts` — source configuration and usage

Create the function following this structure:

```typescript
import type { Check } from '../index.js';
import {
  POINTS_YOUR_CHECK,
  YOUR_THRESHOLD_ARRAY,
} from '../constants.js';
import { readFileOrNull } from '../utils.js'; // or other helpers

export function checkYourCategory(dir: string): Check[] {
  const checks: Check[] = [];

  // 1. Measure something concrete
  const yourMetric = /* e.g., countFiles(), validatePaths(), etc. */;
  const threshold = YOUR_THRESHOLD_ARRAY.find(t => yourMetric >= t.minValue);
  const earnedPts = threshold?.points ?? 0;

  checks.push({
    id: 'your_unique_check_id',
    name: 'Human-readable check name',
    category: 'quality', // matches function context
    maxPoints: POINTS_YOUR_CHECK,
    earnedPoints: earnedPts,
    passed: earnedPts >= Math.ceil(POINTS_YOUR_CHECK * 0.6), // or custom logic
    detail: `${earnedPts}/${POINTS_YOUR_CHECK} points — ${yourMetric} items found`,
    suggestion: earnedPts >= POINTS_YOUR_CHECK ? undefined : 'Action to improve',
    fix: earnedPts >= POINTS_YOUR_CHECK ? undefined : {
      action: 'verb_noun', // e.g., 'add_code_blocks', 'fix_references'
      data: { currentValue: yourMetric, targetValue: 10 },
      instruction: 'Specific, actionable guidance for the user.',
    },
  });

  return checks;
}
```

Verify ID uniqueness: Run `grep -r "'your_unique_check_id'" src/scoring/checks/` — should return only your new check.

### Step 3: Handle platform-specific filtering (if applicable)

If your check only applies to certain agents (Claude, Cursor, Codex, GitHub Copilot), register it in `src/scoring/constants.ts`:

```typescript
// Add to the appropriate set:
export const CLAUDE_ONLY_CHECKS = new Set([
  'claude_md_exists',
  'your_new_check_id', // ← add here
  'claude_rules_exist',
]);
```

Available sets (update exactly one if applicable):
- `CLAUDE_ONLY_CHECKS` — Claude Code targets
- `CURSOR_ONLY_CHECKS` — Cursor targets
- `CODEX_ONLY_CHECKS` — Codex/OpenCode targets
- `COPILOT_ONLY_CHECKS` — GitHub Copilot targets
- `BOTH_ONLY_CHECKS` — Both Claude AND Cursor (cross-platform parity)
- `NON_CODEX_CHECKS` — Everything except Codex/OpenCode
- `CLAUDE_OR_CODEX_CHECKS` — Claude OR Codex

Verify filtering: Examine `filterChecksForTarget()` in `src/scoring/index.ts` to ensure your category will work correctly for your target agents.

### Step 4: Register in src/scoring/index.ts

Import your function at the top:

```typescript
import { checkYourCategory } from './checks/your-file.js';
```

Add to `computeLocalScore()` inside the `allChecks` array initialization:

```typescript
export function computeLocalScore(dir: string, targetAgent?: TargetAgent): ScoreResult {
  const target = targetAgent ?? detectTargetAgent(dir);

  const allChecks: Check[] = [
    ...checkExistence(dir),
    ...checkQuality(dir),
    ...checkGrounding(dir),
    ...checkAccuracy(dir),
    ...checkYourCategory(dir), // ← ADD HERE IN ORDER
    ...checkFreshness(dir),
    ...checkBonus(dir),
    ...checkSources(dir),
  ];
  // ... rest of function
}
```

Verify registration: Run `npm test src/scoring/__tests__/accuracy.test.ts` (or similar) — all existing tests should still pass.

### Step 5: Write deterministic unit tests

Create or edit `src/scoring/checks/__tests__/your-file.test.ts`:

```typescript
import { describe, it, expect } from 'vitest';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { checkYourCategory } from '../your-file.js';
import { POINTS_YOUR_CHECK } from '../../constants.js';

describe('checkYourCategory', () => {
  it('awards full points when condition passes', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      // Set up the passing condition
      writeFileSync(join(dir, 'SOME_FILE.md'), 'content that satisfies check');
      
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      
      expect(check).toBeDefined();
      expect(check?.passed).toBe(true);
      expect(check?.earnedPoints).toBe(POINTS_YOUR_CHECK);
    } finally {
      rmSync(dir, { recursive: true });
    }
  });

  it('awards zero points when condition fails', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      // Don't create the required condition
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      
      expect(check?.passed).toBe(false);
      expect(check?.earnedPoints).toBe(0);
    } finally {
      rmSync(dir, { recursive: true });
    }
  });

  it('returns correct detail message', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      expect(check?.detail).toBeTruthy();
    } finally {
      rmSync(dir, { recursive: true });
    }
  });
});
```

Run tests: `npm test src/scoring/checks/__tests__/your-file.test.ts`. All must pass before shipping.

## Examples

### Example 1: Existence Check

**Trigger**: User says "Add a check to verify .claude/rules/ directory exists."

**Actions**:
1. Add `export const POINTS_CLAUDE_RULES = 3;` to constants.ts
2. In existence.ts: `existsSync(join(dir, '.claude', 'rules'))` → true/false
3. Import in index.ts and add `...checkExistence(dir)` (already done)
4. Test with mkdtempSync; verify earnedPoints matches POINTS_CLAUDE_RULES

**Result**: Check `id: 'claude_rules_exist'` returns `earnedPoints: 3, passed: true` when dir exists.

### Example 2: Quality Check with Thresholds

**Trigger**: User says "Verify config has at least 3 code blocks with executable commands."

**Actions**:
1. Add to constants.ts:
   ```typescript
   export const CODE_BLOCK_THRESHOLDS = [
     { minBlocks: 3, points: 8 },
     { minBlocks: 2, points: 6 },
     { minBlocks: 1, points: 3 },
   ] as const;
   ```
2. In quality.ts:
   - Parse CLAUDE.md with regex to count \`\`\` blocks
   - Match against CODE_BLOCK_THRESHOLDS
   - Return points based on threshold match
3. In fix: Suggest which commands to add

**Result**: 3+ blocks = 8 pts, 2 blocks = 6 pts, 1 block = 3 pts, 0 blocks = 0 pts.

### Example 3: Accuracy Check (Reference Validation)

**Trigger**: User says "Check that all file paths mentioned in config actually exist."

**Actions**:
1. Add `export const POINTS_REFERENCES_VALID = 8;` to constants.ts
2. In accuracy.ts:
   - Extract backtick-quoted paths and dir patterns from config
   - Check existence with `existsSync(join(dir, path))`
   - Calculate ratio: `valid / total`
   - Award partial points: `Math.round(ratio * POINTS_REFERENCES_VALID)`
3. In fix: List invalid paths the user should fix

**Result**: 80% valid refs = ~6 pts; 100% valid = 8 pts; 0% valid = 0 pts.

## Common Issues

**Issue**: "My check doesn't appear in the score report."
**Fix**: 1) Verify ID in `*_ONLY_CHECKS` if platform-specific. 2) Verify import and spread in `index.ts` `allChecks` array. 3) Run `npm test` to ensure no tsc errors. 4) Check `detectTargetAgent()` returns your target platform.

**Issue**: "Points are hardcoded but should use constants."
**Fix**: Replace all literal numbers like `earnedPoints: 5` with `earnedPoints: POINTS_YOUR_CHECK`. Constants are in `src/scoring/constants.ts` — use them consistently.

**Issue**: "Check makes an API call or network request."
**Fix**: Scoring MUST be deterministic and offline. Use only: `fs` module (readFileSync, existsSync, readdirSync), `path`, `execSync` for git commands. No HTTP, no LLM calls, no external services.

**Issue**: "Platform-specific check appears for the wrong agent."
**Fix**: 1) Verify check ID is in correct `*_ONLY_CHECKS` set. 2) Double-check `filterChecksForTarget()` handles your platform set. 3) Test with `detectTargetAgent()` on a real project.

**Issue**: "Test fails with 'Module not found' error."
**Fix**: Ensure file is in `src/scoring/checks/` (not nested). Use `.js` extension in imports (TypeScript transpiles to ES modules). Run `npm run build` to check for tsc errors.

**Issue**: "Detail message is confusing or too technical."
**Fix**: Use friendly language: `3 code blocks found (need 3 for full points)` instead of `codeBlockCount=3`. Make it clear WHY they got/lost points.

**Issue**: "Threshold-based check gives wrong points for edge cases."
**Fix**: Test all boundaries: value=0, value=threshold, value>>threshold. Use `.find()` to match highest-to-lowest: `find(t => value >= t.minValue)`.

**Issue**: "Two checks have the same ID."
**Fix**: Run `grep -r "'my_id'" src/scoring/checks/` to find duplicates. IDs must be globally unique across all check files. Use descriptive names like `claude_md_exists`, not `check_1`.

© caliber-ai-org · MIT · caliber-ai-org/ai-setup

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set17

The whole set →
find-skills find-skills discovers community skills from the Caliber registry and installs them directly into your coding session. CX Codex +1 ★ 1,319repo stars save-learning save-learning turns a user's lasting preferences into project learnings. Use it when an agent should carry a rule from one session into the next. CX Codex +1 ★ 1,319repo stars setup-caliber setup-caliber walks a project through Caliber setup: installs the CLI, hooks agent config sync into every commit, and creates the config files your agents are missing. CX Codex +1 ★ 1,319repo stars adding-a-command adding-a-command generates Commander.js CLI commands in src/commands/, connects them to src/cli.ts, and sets up telemetry tracking. CX Codex +1 ★ 1,319repo stars llm-provider The llm-provider skill guides you through adding an LLM backend to a project, from its provider class to configuration and factory registration. Use it when connecting a third-party API or another model. CX Codex +1 ★ 1,319repo stars caliber-testing caliber-testing is a Claude Code skill that generates Vitest tests with process.env resets, vi.hoisted mocks, and test files placed in __tests__ directories. CX Codex +1 ★ 1,319repo stars writers-pattern The writers-pattern skill guides you through adding a module that writes agent config files for a new platform. Use it when extending caliber to another coding tool. CX Codex +1 ★ 1,319repo stars caliber-jev-compaction caliber-jev-compaction compacts Claude Code sessions using TypeSafe's Jev model to score and drop stale tool calls instead of summarizing. CX Codex ★ 1,319repo stars

Project sponsors

Companies that keep this analytics open