# inference-serving-tensorrt-llm

Provides Claude Code with reference guides and recipes to run LLM inference on NVIDIA GPUs using TensorRT-LLM. Ideal for configuring FP8 quantization and multi-GPU serving.

- skill · `type=skill` · Claude Code · 57 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/inference-serving-tensorrt-llm

## What it does

This skill adds guidance to Claude Code for serving large models through NVIDIA's TensorRT-LLM library. It teaches the agent how to initialize models, configure sampling options, and expose endpoints using the `trtllm-serve` CLI utility.

The material covers in-flight batching, paged memory for key-value caches, FP8 or INT4 quantization, and tensor parallelism across 4 or 8 GPUs. Hardware requirements are strict: you need NVIDIA hardware such as A100 or H100 accelerators, Python 3.10 to 3.12, CUDA 13.0.0, and TensorRT 10.13.2. Without NVIDIA acceleration, this stack will not work. When targeting CPUs or non-NVIDIA chips, alternative tools like vLLM or llama.cpp fit better.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/inference-serving-tensorrt-llm
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill inference-serving-tensorrt-llm -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/inference-serving-tensorrt-llm/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/inference-serving-tensorrt-llm/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/inference-serving-tensorrt-llm/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/inference-serving-tensorrt-llm/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates inference-serving-tensorrt-llm
```

2. **Manually: put the files in place**
- File: `.github/skills/inference-serving-tensorrt-llm/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/inference-serving-tensorrt-llm/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm

## When to use it
- configure `trtllm-serve` with tensor parallelism across multiple GPUs
- prepare Python inference code with FP8 model weights
- enable dynamic in-flight batching for high-throughput serving
- tune sampling parameters for batched prompt generation

## Example prompts
- Generate a script to launch `trtllm-serve` for `meta-llama/Meta-Llama-3-8B` across 4 GPUs.
- Show me how to load a model in FP8 precision using `tensorrt_llm.LLM`.
- Help me configure sampling parameters to process 100 prompts in batches.

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 804 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md