Skip to main content
skill CX Codex

cost-counterfactual — Claude Code skill: what it does, how to install

cost-counterfactual evaluates LLM routing by comparing actual session spend against hypothetical always-haiku, always-sonnet, and always-opus baselines. Useful for tracking over-escalation and ROI.

MIT 249 installs ★ 82,234 updated 2026-11-01

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-code
Open on GitHub

What cost-counterfactual does

The skill parses session-* records stored in the cost-tracking namespace. It aggregates token counts per session and recalculates costs against fixed baselines: always-haiku, always-sonnet, or always-opus. The difference between hypothetical spend and actual spend determines whether dynamic routing pays off.

It runs via Bash. The --since flag filters the timeline, while --format json pipes output to tools like jq. This makes it straightforward to build CI gates that fail when savings fall below a chosen threshold after configuration updates.

Negative savings against Haiku highlight over-escalation on simpler prompts. Because the math assumes identical outcomes without retries, the calculated savings serve as a theoretical upper ceiling.

What it helps with

  • measure actual savings against a fixed Sonnet baseline
  • detect over-escalation to expensive models on simple prompts
  • guard against savings regressions in CI checks
  • audit routing cost efficiency during quarterly reviews

Example prompts for the agent

  • Run cost-counterfactual for the last 7 days as a table
  • Compare our spend against always-sonnet using cost-counterfactual
  • Output cost-counterfactual metrics in json format to check routing savings

How to install cost-counterfactual

1Skills CLI (Vercel)

npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-code

2Manually: put the files in place

~/.claude/skills/cost-counterfactual/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/cost-counterfactual/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about cost-counterfactual

How to install cost-counterfactual in Claude Code?

Run npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-code in your terminal.

What does cost-counterfactual measure?

It calculates hypothetical spend if every session had been routed exclusively to Haiku, Sonnet, or Opus. Comparing that figure with actual spend reveals the exact percentage saved by dynamic routing.

What does a negative savings figure mean?

A negative result against always-haiku indicates that the router escalated to pricier models on tasks that Haiku could have handled. This highlights potential over-escalation or an overly strict quality threshold.

Source text

SKILL.md · MIT
---
name: cost-counterfactual
description: Multi-baseline counterfactual cost analysis. Compares actual session spend to hypothetical always-haiku / always-sonnet / always-opus routing baselines. Answers "is the routing earning its keep?" Negative savings flag over-escalation; positive savings quantify the router's win.
argument-hint: "[--since 7d] [--baseline always-haiku|always-sonnet|always-opus|all] [--format table|json]"
allowed-tools: Bash
disable-model-invocation: true
---

Multi-baseline counterfactual cost analysis. Pairs with the existing observability surface:

- **`cost-budget-check`** — "have we crossed a threshold?" (reactive)
- **`cost-projection`** — "when will we cross a threshold?" (predictive)
- **`cost-counterfactual`** — "is the routing earning its keep?" (comparative) ← this one

## Algorithm

1. Read all `session-*` records from the `cost-tracking` namespace.
2. Apply `--since` window filter (default all-time).
3. Sum tokens across `byModel[*]` entries for each session.
4. For each requested baseline (default: all three):
   - `counterfactualUsd = (input × tier.input + output × tier.output + cache_write × tier.cache_write + cache_read × tier.cache_read) / 1M`
5. Compute `savings = counterfactualUsd − actualUsd`.
6. Emit per-baseline totals + savings % across the comparison set.

## Smoke transcript (2 sessions: 50K haiku tokens + 50K sonnet tokens)

```
| Sessions considered | 2 |
| Total input tokens  | 100,000 |
| Actual spend        | $0.162500 |

| Baseline           | Hypothetical | Actual    | Savings    | %       |
| `always-haiku`     | $0.025000    | $0.162500 | -$0.137500 | -550.00% |
| `always-sonnet`    | $0.300000    | $0.162500 | +$0.137500 |   45.83% |
| `always-opus`      | $1.500000    | $0.162500 | +$1.337500 |   89.17% |
```

## How to read negative savings

A negative `always-haiku` result means **the router chose more-expensive models than haiku** on tasks haiku could have handled. That's an over-escalation signal:
- Maybe qualityBar is set too high
- Maybe the sonnet/opus session was warranted by complexity but the baseline doesn't know that
- Run `cost optimize` (or inspect specific sessions via `cost conversation`) to investigate

Positive savings quantify the router's win against that baseline. The most informative number is usually `always-sonnet` — it's the standard "safe default" baseline most teams would pick if they didn't have routing.

## When to use

- **Quarterly cost review**: "We saved $X vs always-Sonnet — here's the proof."
- **CI gate**: `cost counterfactual --format json | jq '.baselines[1].savingsPct > 30'` — fail builds if routing isn't saving ≥30% vs sonnet baseline (workload-shift detector).
- **Routing-config validation**: When introducing a new qualityBar or cost-ceiling, re-run counterfactual to confirm savings didn't regress.

## Stationarity caveat

Like all counterfactual analyses, this assumes the same tokens at the same complexity would have produced the same outcome from the baseline model. That's an upper bound — the baseline might have failed and required retries, which the math doesn't capture. Treat the numbers as a quality-blind ceiling.

© ruvnet · MIT · ruvnet/ruflo

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set155

The whole set →
nested-subagents nested-subagents spawns a tree of sub-agents where each child can spawn its own children. Useful when intermediate state would overflow a single agent's context. CX Codex +1 317installs cost-session cost-session is a Claude Code skill that breaks down costs per message in a session to reveal expensive requests. CX Codex +1 249installs cost-projection The cost-projection Claude Code skill projects future spend from recent session data. It estimates daily costs and shows when configured budget thresholds may be reached. CX Codex +1 249installs cost-health cost-health runs four cost checks at once and returns a single status: budget, burn acceleration, session outliers and days left before the budget is gone. CX Codex +1 249installs cost-diff The cost-diff Claude Code skill compares two JSON snapshots of spending. Use it to detect cost changes between a baseline and a pull request branch. CX Codex +1 248installs cost-burn cost-burn — a skill for Claude Code and other platforms that observes production spend trends. It triggers in CI to detect if daily spend acceleration exceeds a specified threshold. CX Codex +1 248installs cost-anomaly The cost-anomaly Claude Code skill detects individual sessions with unusual spend and reports their modified z-scores. It runs when a request is about pinpointing a costly outlier session rather than tracking an aggregate trend. CX Codex +1 248installs Ruflo The Ruflo MCP server for Claude Code and Codex coordinates agent teams and keeps memory across sessions. After init, hooks route tasks and coordinate agents in the background. CX Codex +1 ★ 82,234repo stars

Project sponsors

Companies that keep this analytics open