inference-serving-vllm — Claude Code skill: serve LLMs with vLLM
A Claude skill that walks you through serving LLMs on vLLM: an Anthropic-compatible server, offline batch runs and quantized models.
npx claude-code-templates@latest --skill ai-research/inference-serving-vllmWhat inference-serving-vllm does
The skill covers vLLM, the library for serving large language models. Three workflows are laid out: a production API deployment, offline batch inference over a pile of prompts, and quantized serving when GPU memory is tight. Each step comes with vllm serve commands and the flags for memory, context length and port.
Quantization with GPTQ, AWQ and FP8 gets its own section, as do tensor parallelism across several GPUs and Prometheus metrics on port 9090. Checklists keep the order straight: configure the server, run a load test, turn on monitoring, ship it. A troubleshooting part deals with out-of-memory errors during model loading.
The skill assumes vllm, torch and transformers are already installed. Tensor parallelism and quantization want the right hardware — on a single card --tensor-parallel-size buys you nothing. There is also a comparison against llama.cpp, TensorRT-LLM and plain transformers, so you can pick the right tool instead of reaching for vLLM every time.
What it helps with
- stand up an OpenAI-compatible endpoint in front of a local model
- fit a 70B model into limited VRAM with quantization
- run a batch of prompts from a file without a server
- track latency and KV cache usage with Prometheus metrics
- decide between vLLM, llama.cpp and TensorRT-LLM
Example prompts for the agent
- Serve
meta-llama/Llama-3-8B-Instructwith vLLM on port 8000 and prefix caching enabled - Help me run a 70B AWQ model across four GPUs with tensor parallelism
- Run offline inference over the prompts in
prompts.txtand write the output toresults.jsonl
How to install inference-serving-vllm
1claude-code-templates
npx claude-code-templates@latest --skill ai-research/inference-serving-vllm2Skills CLI (Vercel)
npx skills add davila7/claude-code-templates --skill inference-serving-vllm -a claude-code3Manually: put the files in place
~/.claude/skills/inference-serving-vllm/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/inference-serving-vllm/
Manually: put the files in place
.agents/skills/inference-serving-vllm/SKILL.mdFor all projects use ~/.agents/skills/
Manually: put the files in place
.cursor/skills/inference-serving-vllm/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install davila7/claude-code-templates inference-serving-vllm2Manually: put the files in place
.github/skills/inference-serving-vllm/SKILL.mdPlatform CLI
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-vllmManually: put the files in place
.devin/skills/inference-serving-vllm/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about inference-serving-vllm
How do I install inference-serving-vllm in Claude Code?
Use the templates CLI: npx claude-code-templates@latest --skill ai-research/inference-serving-vllm. Or add it with npx skills add davila7/claude-code-templates --skill inference-serving-vllm -a claude-code. The skill then lives at ~/.claude/skills/inference-serving-vllm/SKILL.md.
What does inference-serving-vllm do?
It guides you through deploying models on vLLM: server configuration, load testing, metrics, Docker, offline batches and quantization. It also explains when llama.cpp or TensorRT-LLM is the better pick.
What do I need before using it?
The vllm, torch and transformers packages. Tensor parallelism and quantization need GPUs — a single card will not get you far with those modes.
