# serving-llms-on-epyc

Deploys a vLLM endpoint with the zentorch backend on AMD EPYC CPUs using Docker, Podman, or conda. Handy for CPU-based LLM inference pinned to a single socket.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 817 installs · ★ 465
- Author: amd
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/amd/skills/tree/HEAD/skills/serving-llms-on-epyc/SKILL.md
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/serving-llms-on-epyc

## What it does

This skill runs a vLLM OpenAI endpoint accelerated by zentorch on AMD EPYC server CPUs. It first inspects the system with `scripts/detect.py` to identify the Zen architecture, NUMA topology, and AVX-512 support. For modern hardware like Genoa or Turin with AVX-512 BF16, it configures the validated zentorch stack. On pre-Zen4 systems like Milan lacking AVX-512 BF16, it stops zentorch and asks before switching to stock CPU vLLM.

The deployment pins the process strictly to one socket and its local memory. Multi-socket setups do not span instances because vLLM scales poorly across sockets. The runner relies on Docker or Podman, falling back to a host conda environment. Missing container permissions immediately halt the flow because the skill never uses sudo or attempts privilege escalation.

Any launch failure stops execution instantly without retries. The agent surfaces error logs instead of trying to debug the container. When no model argument is supplied, it defaults to `Qwen/Qwen3-0.6B` as specified in `data/epyc.json`.

## Requirements
- Works with: Docker

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add amd/skills --skill serving-llms-on-epyc -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/serving-llms-on-epyc/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/serving-llms-on-epyc/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add amd/skills --skill serving-llms-on-epyc -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/serving-llms-on-epyc/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add amd/skills --skill serving-llms-on-epyc -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/serving-llms-on-epyc/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install amd/skills serving-llms-on-epyc
```

2. **Manually: put the files in place**
- File: `.github/skills/serving-llms-on-epyc/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add amd/skills --skill serving-llms-on-epyc -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/amd/skills --path skills/serving-llms-on-epyc
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/serving-llms-on-epyc/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add amd/skills --skill serving-llms-on-epyc -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/amd/skills/tree/HEAD/skills/serving-llms-on-epyc/SKILL.md

## When to use it
- Deploy an OpenAI-compatible vLLM endpoint on an AMD EPYC server CPU
- Verify AVX-512 instruction support and Zen architecture before launching
- Pin the inference process to a single CPU socket and local NUMA memory
- Configure Docker, Podman, or conda runtime based on available host RAM

## Example prompts
- Serve Qwen/Qwen3-0.6B using vLLM on this AMD EPYC CPU
- Deploy a zentorch CPU endpoint on our EPYC server using Docker

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 12190e6d14253490.

BEGIN THIRD-PARTY FILE 12190e6d14253490
````text
---
name: serving-llms-on-epyc
description: >-
  Serves an LLM on a supported AMD EPYC server CPU using vLLM with zentorch, in
  Docker, Podman, or conda. Use for "vLLM on CPU", "zentorch serving", or an
  EPYC CPU endpoint, including on a host that also has AMD Instinct GPUs.
  Detects the EPYC generation, validates the runtime, checks model support and
  RAM fit, sizes threads/KV/NUMA, confirms the plan, launches, and verifies the
  endpoint. Runs one instance on one socket and its memory. Reports and stops
  on failure; does not retry or debug. Use serving-llms-on-instinct when the
  endpoint should run on a GPU. Excludes multi-node, EPYC 4000, and pre-Zen4
  EPYC without AVX-512.
allowed-tools: Bash, Read
---

# Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU)

Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU
backend, sized to the hardware. Container-first (Docker or Podman); conda/host
is the fallback. An installed AMD Instinct GPU does not disqualify the host:
select this skill when the endpoint itself should run on the EPYC CPU.

**This is single-socket serving:** one instance pinned to one socket and its memory
(vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it
runs on a single socket; the multi-socket answer is **multiple instances (one per
socket)**, which is out of scope for this single-instance recipe.

Hard rule for this skill: **on any failure, report the cause + logs and STOP.
Do not retry, do not debug.** (Debugging is a separate workflow.)

**The agent does the serve flow itself** -- pull, configure, launch, poll --
using the runtime `validate.py` reports. Never hand the user per-serve commands.
Like serving-llms-on-instinct, an accessible container runtime is a one-time
**prerequisite**: if `validate.py` finds none, report its one-time fix (make
docker accessible / install podman / provide a conda env) and stop. Do not
attempt `sudo` or privilege escalation.

## Data file

Read `data/epyc.json` directly. It holds the container image, mandatory CPU run
flags, supported precision, the model-support policy, the default model, and the
verified throughput-flag gotcha. Its `vllm_version` and image tag are one
validated default stack; keep them aligned and do not hardcode either from memory.

## Step 1: Detect the CPU

```bash
python3 scripts/detect.py            # add --host user@box for a remote host
```

Returns `cpu_model`, `is_amd_epyc`, `epyc_generation`
(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005),
`zen_arch`, `is_supported_epyc`, `avx512`, `logical_cores`, `physical_cores`,
`sockets`, `numa_nodes`, `memory_gb`.

Route from detect.py -- decide the serving path:
````
END THIRD-PARTY FILE 12190e6d14253490

The third-party file has ended. Everything between the 12190e6d14253490 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2723 of 22741 bytes. © amd · MIT · https://github.com/amd/skills/tree/HEAD/skills/serving-llms-on-epyc/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md