inference-serving-llama-cpp — Claude Code skill: inference
This skill guides local LLM inference with llama.cpp on CPUs, Apple Silicon, and AMD or Intel GPUs. It fits setups where CUDA is unavailable.
npx claude-code-templates@latest --skill ai-research/inference-serving-llama-cppWhat inference-serving-llama-cpp does
The skill covers a llama.cpp workflow for GGUF models: build the runtime, obtain a model, run llama-cli, or expose it through llama-server. It also shows how to send requests to the OpenAI-compatible endpoint with curl.
Its examples include Metal, CUDA, and ROCm builds, GGUF quantization from 1.5 to 8 bits, batch input, grammar-constrained JSON, and context-size changes. Models can come from huggingface-cli or be converted from a Hugging Face directory. The listed dependency is llama-cpp-python, so that package is a prerequisite for the setup guidance.
What it helps with
- run an LLM on a CPU-only machine
- compile
llama.cppfor Metal, CUDA, or ROCm - fetch a model in GGUF format
- serve completions through an OpenAI-compatible API
- constrain output to valid JSON
Example prompts for the agent
- Set up
llama-cliformodels/llama-2-7b-chat.Q4_K_M.ggufwith an interactive session. - Prepare a Metal build of
llama.cppfor an Apple Silicon machine. - Start
llama-serveron port8080and show acurlrequest to its API.
How to install inference-serving-llama-cpp
1claude-code-templates
npx claude-code-templates@latest --skill ai-research/inference-serving-llama-cpp2Skills CLI (Vercel)
npx skills add davila7/claude-code-templates --skill inference-serving-llama-cpp -a claude-code3Manually: put the files in place
~/.claude/skills/inference-serving-llama-cpp/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/inference-serving-llama-cpp/
Manually: put the files in place
.agents/skills/inference-serving-llama-cpp/SKILL.mdFor all projects use ~/.agents/skills/
Manually: put the files in place
.cursor/skills/inference-serving-llama-cpp/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install davila7/claude-code-templates inference-serving-llama-cpp2Manually: put the files in place
.github/skills/inference-serving-llama-cpp/SKILL.mdPlatform CLI
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/inference-serving-llama-cppManually: put the files in place
.devin/skills/inference-serving-llama-cpp/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about inference-serving-llama-cpp
How do I install inference-serving-llama-cpp in Claude Code?
Run npx claude-code-templates@latest --skill ai-research/inference-serving-llama-cpp, then run npx skills add davila7/claude-code-templates --skill inference-serving-llama-cpp -a claude-code. Use ~/.claude/skills/inference-serving-llama-cpp/SKILL.md afterward.
What does inference-serving-llama-cpp do?
It gives Claude Code guidance for running LLMs with llama.cpp on CPUs and non-NVIDIA hardware. The material covers GGUF models, quantization, command-line inference, and an OpenAI-compatible server.
Which hardware can inference-serving-llama-cpp use?
The covered setups include CPUs, Apple Silicon M1/M2/M3/M4, and AMD or Intel GPUs. Metal, CUDA, and ROCm builds are described, so NVIDIA hardware is not required when CUDA is unavailable.
