# inference-serving-vllm

A Claude skill that walks you through serving LLMs on vLLM: an Anthropic-compatible server, offline batch runs and quantized models.

- skill · `type=skill` · Claude Code · 70 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-vllm
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/inference-serving-vllm

## What it does

The skill covers vLLM, the library for serving large language models. Three workflows are laid out: a production API deployment, offline batch inference over a pile of prompts, and quantized serving when GPU memory is tight. Each step comes with `vllm serve` commands and the flags for memory, context length and port.

Quantization with GPTQ, AWQ and FP8 gets its own section, as do tensor parallelism across several GPUs and Prometheus metrics on port 9090. Checklists keep the order straight: configure the server, run a load test, turn on monitoring, ship it. A troubleshooting part deals with out-of-memory errors during model loading.

The skill assumes `vllm`, `torch` and `transformers` are already installed. Tensor parallelism and quantization want the right hardware — on a single card `--tensor-parallel-size` buys you nothing. There is also a comparison against llama.cpp, TensorRT-LLM and plain transformers, so you can pick the right tool instead of reaching for vLLM every time.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/inference-serving-vllm
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill inference-serving-vllm -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/inference-serving-vllm/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/inference-serving-vllm/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/inference-serving-vllm/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/inference-serving-vllm/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates inference-serving-vllm
```

2. **Manually: put the files in place**
- File: `.github/skills/inference-serving-vllm/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-vllm
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/inference-serving-vllm/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-vllm

## When to use it
- stand up an OpenAI-compatible endpoint in front of a local model
- fit a 70B model into limited VRAM with quantization
- run a batch of prompts from a file without a server
- track latency and KV cache usage with Prometheus metrics
- decide between vLLM, llama.cpp and TensorRT-LLM

## Example prompts
- Serve `meta-llama/Llama-3-8B-Instruct` with vLLM on port 8000 and prefix caching enabled
- Help me run a 70B AWQ model across four GPUs with tensor parallelism
- Run offline inference over the prompts in `prompts.txt` and write the output to `results.jsonl`

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 361 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md