inference-serving-tensorrt-llm — Claude Code skill: setup guide
Provides Claude Code with reference guides and recipes to run LLM inference on NVIDIA GPUs using TensorRT-LLM. Ideal for configuring FP8 quantization and multi-GPU serving.
npx claude-code-templates@latest --skill ai-research/inference-serving-tensorrt-llmWhat inference-serving-tensorrt-llm does
This skill adds guidance to Claude Code for serving large models through NVIDIA's TensorRT-LLM library. It teaches the agent how to initialize models, configure sampling options, and expose endpoints using the trtllm-serve CLI utility.
The material covers in-flight batching, paged memory for key-value caches, FP8 or INT4 quantization, and tensor parallelism across 4 or 8 GPUs. Hardware requirements are strict: you need NVIDIA hardware such as A100 or H100 accelerators, Python 3.10 to 3.12, CUDA 13.0.0, and TensorRT 10.13.2. Without NVIDIA acceleration, this stack will not work. When targeting CPUs or non-NVIDIA chips, alternative tools like vLLM or llama.cpp fit better.
What it helps with
- configure
trtllm-servewith tensor parallelism across multiple GPUs - prepare Python inference code with FP8 model weights
- enable dynamic in-flight batching for high-throughput serving
- tune sampling parameters for batched prompt generation
Example prompts for the agent
- Generate a script to launch
trtllm-serveformeta-llama/Meta-Llama-3-8Bacross 4 GPUs. - Show me how to load a model in FP8 precision using
tensorrt_llm.LLM. - Help me configure sampling parameters to process 100 prompts in batches.
How to install inference-serving-tensorrt-llm
1claude-code-templates
npx claude-code-templates@latest --skill ai-research/inference-serving-tensorrt-llm2Skills CLI (Vercel)
npx skills add davila7/claude-code-templates --skill inference-serving-tensorrt-llm -a claude-code3Manually: put the files in place
~/.claude/skills/inference-serving-tensorrt-llm/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/inference-serving-tensorrt-llm/
Manually: put the files in place
.agents/skills/inference-serving-tensorrt-llm/SKILL.mdFor all projects use ~/.agents/skills/
Manually: put the files in place
.cursor/skills/inference-serving-tensorrt-llm/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install davila7/claude-code-templates inference-serving-tensorrt-llm2Manually: put the files in place
.github/skills/inference-serving-tensorrt-llm/SKILL.mdPlatform CLI
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-tensorrt-llmManually: put the files in place
.devin/skills/inference-serving-tensorrt-llm/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about inference-serving-tensorrt-llm
How do I install inference-serving-tensorrt-llm in Claude Code?
Run npx claude-code-templates@latest --skill ai-research/inference-serving-tensorrt-llm in your terminal.
What does inference-serving-tensorrt-llm do?
It equips Claude Code with deployment patterns and code examples for NVIDIA TensorRT-LLM. The skill helps configure distributed serving, batching, and quantization.
What environment is required to use TensorRT-LLM?
You need NVIDIA GPUs such as A100 or H100, Python between 3.10 and 3.12, CUDA 13.0.0, and TensorRT 10.13.2. Projects deployed on CPUs or Apple Silicon should use llama.cpp instead.
