benchmark-vllm-model — Claude Code skill: cost per token on vLLM
Works out what a self-hosted model costs per token and per request: brings it up on a GPU, loads it to saturation, then divides the hourly price by measured throughput.
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a claude-codeWhat benchmark-vllm-model does
Reach for this skill when you need a real number for running a model yourself instead of paying a hosted API. It starts the model in a Chalk container on a GPU, pushes traffic until throughput flattens, checks that the card is the bottleneck, and only then divides the node's hourly rate by the measured figure. The sequence matters — a throughput number with no saturation check proves nothing.
The bundled scripts/bench.py varies prompt length and how many requests run at once, and takes throughput from vLLM's own Prometheus counters. Measurement runs over a sustained window rather than a fixed request count, because short windows are mostly jitter. The skill suggests driving load from inside the cluster and repeating the run against the public webUrl; matching results mean the ingress is not in the way.
You need HF_TOKEN set before starting, and the GPU pool has to carry the chalk.ai/workload-type=compute label — without it the pod never gets scheduled. Turing cards (T4) have no native bf16, so those runs use --dtype float16. For the hourly rate the skill points at GCP's billing catalog rather than pricing aggregator sites: the accelerator is often a separate line item, and leaving it out understates the node.
What it helps with
- work out cost per million tokens for a self-hosted model
- size the GPU count a given load needs
- compare self-hosting economics against a hosted API
- tell whether the server or the load generator is the bottleneck
- pick vLLM flags that match your request profile
Example prompts for the agent
- Benchmark this model on vLLM and give me $/1M tokens
- What would it cost to serve this model on a single L4?
- Compare my self-hosted setup against a hosted API on cost per request
How to install benchmark-vllm-model
1Skills CLI (Vercel)
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a claude-code2Manually: put the files in place
~/.claude/skills/benchmark-vllm-model/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/benchmark-vllm-model/
1Skills CLI (Vercel)
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a codex2Manually: put the files in place
.agents/skills/benchmark-vllm-model/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a cursor2Manually: put the files in place
.cursor/skills/benchmark-vllm-model/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install chalk-ai/chalk-ai-plugins benchmark-vllm-model2Manually: put the files in place
.github/skills/benchmark-vllm-model/SKILL.md1Skills CLI (Vercel)
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a gemini-cli2Platform CLI
gemini skills install https://github.com/chalk-ai/chalk-ai-plugins --path plugins/chalk-benchmark-model/skills/benchmark-vllm-modelManually: put the files in place
.devin/skills/benchmark-vllm-model/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about benchmark-vllm-model
How do I install benchmark-vllm-model in Claude Code?
Run npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a claude-code. The skill lands at ~/.claude/skills/benchmark-vllm-model/SKILL.md.
What does benchmark-vllm-model do?
It serves the model on vLLM inside a Chalk container, measures throughput under load, and turns the node's hourly price into cost per token and per request.
Why not take the hourly price from a pricing aggregator?
The accelerator is frequently billed as its own line item and aggregators drop it, so the node looks cheaper than it is. The skill pulls rates from GCP's billing catalog.
