# benchmark-vllm-model

Works out what a self-hosted model costs per token and per request: brings it up on a GPU, loads it to saturation, then divides the hourly price by measured throughput.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 837 installs
- Author: chalk-ai
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/chalk-ai/chalk-ai-plugins/tree/HEAD/plugins/chalk-benchmark-model/skills/benchmark-vllm-model/SKILL.md
- Updated: 2026-09-15
- HTML: https://affpapa.org/en/nevibecode/skills/benchmark-vllm-model

## What it does

Reach for this skill when you need a real number for running a model yourself instead of paying a hosted API. It starts the model in a Chalk container on a GPU, pushes traffic until throughput flattens, checks that the card is the bottleneck, and only then divides the node's hourly rate by the measured figure. The sequence matters — a throughput number with no saturation check proves nothing.

The bundled `scripts/bench.py` varies prompt length and how many requests run at once, and takes throughput from vLLM's own Prometheus counters. Measurement runs over a sustained window rather than a fixed request count, because short windows are mostly jitter. The skill suggests driving load from inside the cluster and repeating the run against the public `webUrl`; matching results mean the ingress is not in the way.

You need `HF_TOKEN` set before starting, and the GPU pool has to carry the `chalk.ai/workload-type=compute` label — without it the pod never gets scheduled. Turing cards (T4) have no native bf16, so those runs use `--dtype float16`. For the hourly rate the skill points at GCP's billing catalog rather than pricing aggregator sites: the accelerator is often a separate line item, and leaving it out understates the node.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/benchmark-vllm-model/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/benchmark-vllm-model/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/benchmark-vllm-model/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/benchmark-vllm-model/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install chalk-ai/chalk-ai-plugins benchmark-vllm-model
```

2. **Manually: put the files in place**
- File: `.github/skills/benchmark-vllm-model/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/chalk-ai/chalk-ai-plugins --path plugins/chalk-benchmark-model/skills/benchmark-vllm-model
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/benchmark-vllm-model/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add chalk-ai/chalk-ai-plugins --skill benchmark-vllm-model -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/chalk-ai/chalk-ai-plugins/tree/HEAD/plugins/chalk-benchmark-model/skills/benchmark-vllm-model/SKILL.md

## When to use it
- work out cost per million tokens for a self-hosted model
- size the GPU count a given load needs
- compare self-hosting economics against a hosted API
- tell whether the server or the load generator is the bottleneck
- pick vLLM flags that match your request profile

## Example prompts
- Benchmark this model on vLLM and give me $/1M tokens
- What would it cost to serve this model on a single L4?
- Compare my self-hosted setup against a hosted API on cost per request

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md