cost-counterfactual — Claude Code skill: what it does, how to install
cost-counterfactual evaluates LLM routing by comparing actual session spend against hypothetical always-haiku, always-sonnet, and always-opus baselines. Useful for tracking over-escalation and ROI.
MIT 249 installs ★ 82,234 updated 2026-11-01
npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-codeWhat cost-counterfactual does
The skill parses session-* records stored in the cost-tracking namespace. It aggregates token counts per session and recalculates costs against fixed baselines: always-haiku, always-sonnet, or always-opus. The difference between hypothetical spend and actual spend determines whether dynamic routing pays off.
It runs via Bash. The --since flag filters the timeline, while --format json pipes output to tools like jq. This makes it straightforward to build CI gates that fail when savings fall below a chosen threshold after configuration updates.
Negative savings against Haiku highlight over-escalation on simpler prompts. Because the math assumes identical outcomes without retries, the calculated savings serve as a theoretical upper ceiling.
What it helps with
- measure actual savings against a fixed Sonnet baseline
- detect over-escalation to expensive models on simple prompts
- guard against savings regressions in CI checks
- audit routing cost efficiency during quarterly reviews
Example prompts for the agent
- Run cost-counterfactual for the last 7 days as a table
- Compare our spend against always-sonnet using cost-counterfactual
- Output cost-counterfactual metrics in json format to check routing savings
How to install cost-counterfactual
1Skills CLI (Vercel)
npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-code2Manually: put the files in place
~/.claude/skills/cost-counterfactual/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/cost-counterfactual/
1Skills CLI (Vercel)
npx skills add ruvnet/ruflo --skill cost-counterfactual -a codex2Manually: put the files in place
.agents/skills/cost-counterfactual/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add ruvnet/ruflo --skill cost-counterfactual -a cursor2Manually: put the files in place
.cursor/skills/cost-counterfactual/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install ruvnet/ruflo plugins/ruflo-cost-tracker/skills/cost-counterfactual2Manually: put the files in place
.github/skills/cost-counterfactual/SKILL.md1Skills CLI (Vercel)
npx skills add ruvnet/ruflo --skill cost-counterfactual -a gemini-cli2Platform CLI
gemini skills install https://github.com/ruvnet/ruflo --path plugins/ruflo-cost-tracker/skills/cost-counterfactualManually: put the files in place
.devin/skills/cost-counterfactual/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add ruvnet/ruflo --skill cost-counterfactual -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about cost-counterfactual
How to install cost-counterfactual in Claude Code?
Run npx skills add ruvnet/ruflo --skill cost-counterfactual -a claude-code in your terminal.
What does cost-counterfactual measure?
It calculates hypothetical spend if every session had been routed exclusively to Haiku, Sonnet, or Opus. Comparing that figure with actual spend reveals the exact percentage saved by dynamic routing.
What does a negative savings figure mean?
A negative result against always-haiku indicates that the router escalated to pricier models on tasks that Haiku could have handled. This highlights potential over-escalation or an overly strict quality threshold.
Source text
SKILL.md · MIT
--- name: cost-counterfactual description: Multi-baseline counterfactual cost analysis. Compares actual session spend to hypothetical always-haiku / always-sonnet / always-opus routing baselines. Answers "is the routing earning its keep?" Negative savings flag over-escalation; positive savings quantify the router's win. argument-hint: "[--since 7d] [--baseline always-haiku|always-sonnet|always-opus|all] [--format table|json]" allowed-tools: Bash disable-model-invocation: true --- Multi-baseline counterfactual cost analysis. Pairs with the existing observability surface: - **`cost-budget-check`** — "have we crossed a threshold?" (reactive) - **`cost-projection`** — "when will we cross a threshold?" (predictive) - **`cost-counterfactual`** — "is the routing earning its keep?" (comparative) ← this one ## Algorithm 1. Read all `session-*` records from the `cost-tracking` namespace. 2. Apply `--since` window filter (default all-time). 3. Sum tokens across `byModel[*]` entries for each session. 4. For each requested baseline (default: all three): - `counterfactualUsd = (input × tier.input + output × tier.output + cache_write × tier.cache_write + cache_read × tier.cache_read) / 1M` 5. Compute `savings = counterfactualUsd − actualUsd`. 6. Emit per-baseline totals + savings % across the comparison set. ## Smoke transcript (2 sessions: 50K haiku tokens + 50K sonnet tokens) ``` | Sessions considered | 2 | | Total input tokens | 100,000 | | Actual spend | $0.162500 | | Baseline | Hypothetical | Actual | Savings | % | | `always-haiku` | $0.025000 | $0.162500 | -$0.137500 | -550.00% | | `always-sonnet` | $0.300000 | $0.162500 | +$0.137500 | 45.83% | | `always-opus` | $1.500000 | $0.162500 | +$1.337500 | 89.17% | ``` ## How to read negative savings A negative `always-haiku` result means **the router chose more-expensive models than haiku** on tasks haiku could have handled. That's an over-escalation signal: - Maybe qualityBar is set too high - Maybe the sonnet/opus session was warranted by complexity but the baseline doesn't know that - Run `cost optimize` (or inspect specific sessions via `cost conversation`) to investigate Positive savings quantify the router's win against that baseline. The most informative number is usually `always-sonnet` — it's the standard "safe default" baseline most teams would pick if they didn't have routing. ## When to use - **Quarterly cost review**: "We saved $X vs always-Sonnet — here's the proof." - **CI gate**: `cost counterfactual --format json | jq '.baselines[1].savingsPct > 30'` — fail builds if routing isn't saving ≥30% vs sonnet baseline (workload-shift detector). - **Routing-config validation**: When introducing a new qualityBar or cost-ceiling, re-run counterfactual to confirm savings didn't regress. ## Stationarity caveat Like all counterfactual analyses, this assumes the same tokens at the same complexity would have produced the same outcome from the baseline model. That's an upper bound — the baseline might have failed and required retries, which the math doesn't capture. Treat the numbers as a quality-blind ceiling.
© ruvnet · MIT · ruvnet/ruflo