# inference-serving-llama-cpp

This skill guides local LLM inference with llama.cpp on CPUs, Apple Silicon, and AMD or Intel GPUs. It fits setups where CUDA is unavailable.

- skill · `type=skill` · Claude Code · 60 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-llama-cpp
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/inference-serving-llama-cpp

## What it does

The skill covers a llama.cpp workflow for GGUF models: build the runtime, obtain a model, run `llama-cli`, or expose it through `llama-server`. It also shows how to send requests to the OpenAI-compatible endpoint with `curl`.

Its examples include Metal, CUDA, and ROCm builds, GGUF quantization from 1.5 to 8 bits, batch input, grammar-constrained JSON, and context-size changes. Models can come from `huggingface-cli` or be converted from a Hugging Face directory. The listed dependency is `llama-cpp-python`, so that package is a prerequisite for the setup guidance.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/inference-serving-llama-cpp
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill inference-serving-llama-cpp -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/inference-serving-llama-cpp/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/inference-serving-llama-cpp/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/inference-serving-llama-cpp/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/inference-serving-llama-cpp/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates inference-serving-llama-cpp
```

2. **Manually: put the files in place**
- File: `.github/skills/inference-serving-llama-cpp/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-llama-cpp
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/inference-serving-llama-cpp/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/inference-serving-llama-cpp

## When to use it
- run an LLM on a CPU-only machine
- compile `llama.cpp` for Metal, CUDA, or ROCm
- fetch a model in GGUF format
- serve completions through an OpenAI-compatible API
- constrain output to valid JSON

## Example prompts
- Set up `llama-cli` for `models/llama-2-7b-chat.Q4_K_M.gguf` with an interactive session.
- Prepare a Metal build of `llama.cpp` for an Apple Silicon machine.
- Start `llama-server` on port `8080` and show a `curl` request to its API.

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 804 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md