inference-optimisation — Claude Code skill: what it does, install
inference-optimisation is a Claude Code skill that tunes local LLM servers with measured experiments. It starts when the task is to find a machine’s tok/s limit.
MIT ★ 1,699 updated 2026-11-16
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a claude-codeWhat inference-optimisation does
inference-optimisation measures a local LLM server, estimates a practical ceiling, then tests one configuration change at a time. It compares each candidate with controls, records the result in PLAN.md and ledger.jsonl, stores run configs, and updates best-config.json when a change passes the checks.
The skill covers decode and prefill throughput, TTFT, startup time, KV cache size, memory limits, and full-context stability. Its references explain ceiling calculations, measurement pitfalls, tuning levers, engine-specific traps, and publishing gates. The scripts/ directory provides a benchmark harness, ceiling calculator, memory watchdog, serialized queue, and ledger.
It can tune vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, and TensorRT-LLM on NVIDIA, AMD, Intel, or Apple hardware, including unified-memory and multi-GPU systems. A forced server stop during a batch can trigger GPU driver faults.
What it helps with
- find the highest tok/s for a local model server
- compare decode and prefill performance on one machine
- tune speculative decoding and KV cache within a memory cap
- test full-context stability after a server change
- publish a measured launch recipe
Example prompts for the agent
- Tune this model server and find the highest tok/s on this machine.
- Compare speculative decoding settings with controls between each candidate.
- Prepare and publish a tuned recipe for this server.
How to install inference-optimisation
1Skills CLI (Vercel)
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a claude-code2Manually: put the files in place
~/.claude/skills/inference-optimisation/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/inference-optimisation/
1Skills CLI (Vercel)
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a codex2Manually: put the files in place
.agents/skills/inference-optimisation/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a cursor2Manually: put the files in place
.cursor/skills/inference-optimisation/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install sybil-solutions/inference-optimisation-skill skills/inference-optimisation2Manually: put the files in place
.github/skills/inference-optimisation/SKILL.md1Skills CLI (Vercel)
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a gemini-cli2Platform CLI
gemini skills install https://github.com/sybil-solutions/inference-optimisation-skill --path skills/inference-optimisationManually: put the files in place
.devin/skills/inference-optimisation/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about inference-optimisation
How do I install inference-optimisation in Claude Code?
Run npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -a claude-code. The skill file will be at ~/.claude/skills/inference-optimisation/SKILL.md.
What does inference-optimisation do?
It tunes a local LLM server through measured experiments. The skill calculates a practical ceiling, tests configuration changes against controls, and logs why each change was accepted or rejected.
What does inference-optimisation need to work?
It needs a local model server such as vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, or TensorRT-LLM. The process also uses machine measurements, a memory cap, and a required context length.
Source text
SKILL.md · MIT
--- name: inference-optimisation description: > Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum. --- # Inference optimisation (hill-climbing a model server) You improve how fast a model server runs on one machine, with evidence. The loop is: 1. Measure the baseline. 2. Calculate the ceiling. 3. Change one thing. 4. Measure it against controls. 5. Keep it or reject it, and write down why. 6. Repeat until the remaining gap needs work out of scope, then qualify and publish. Detailed references, read when needed: - `references/ceilings.md`: how to compute theoretical maxima. It includes a worked example. - `references/methodology.md`: measurement pitfalls and statistics. - `references/levers.md`: levers by area, ordered by typical payoff, with measured evidence. - `references/gotchas.md`: engine- and hardware-specific traps. - `references/publishing.md`: qualification gates, pinning, poster and report. - `scripts/`: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger. ## 0. Before touching anything - **Write down the targets and constraints** the user gave. Ask only if one is missing and it changes what you do. - Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup. - Memory cap. - Context length that must stay stable. - Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill. - **Inventory the machine:** - accelerator, memory type and size, and whether memory is unified; - driver, engine version, model, quantization, current launch command. - **Measure, don't trust spec sheets:** - real memory bandwidth (device copy); - real matmul throughput (large GEMM); - clocks under load. - **Read any existing experiment log for this machine and model.** Never rerun a rejected experiment without a new reason. - **Create the working directory:** - `PLAN.md`: a timestamped log of every experiment, with numbers, verdict and reason. - `THEORY.md`: the ceilings. - `ledger.jsonl`: one line per run, from the harness. - `configs/`: one JSON per run. - `best-config.json`. ## 1. Ceilings first Compute the theoretical maximum for each metric before optimising (`scripts/ceiling.py`, `references/ceilings.md`): - **Decode is memory-bandwidth bound.** Count the bytes read per emitted token: - dense weights; - active experts: top-k ÷ experts × expert bytes; - LM head; - KV read at the working context. - Divide by measured bandwidth. - Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part. - **Prefill is compute bound.** FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling. - **TTFT floor:** one forward pass plus tokenize and schedule. - **Startup floor:** weight read from disk or page cache plus allocation, assuming JIT and graph caches persist. - **KV:** bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap. Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic. ## 2. Ground rules (non-negotiable) 1. **One change per experiment.** Give each its own config file. Derive new configs from the current best. 2. **Control for noise.** Run-to-run noise is typically ±5–10%. - Take ≥6 decode samples per run and use the median. - Run controls on both sides of each candidate: control, candidate, control. - Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%. - Repeat any single surprising win. 3. **Keep the environment fixed.** Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs. - Make the harness refuse to run otherwise. - Record the state at the end of each run, and void the run if it changed. - Restore any profile you changed when you finish, and tell the user. 4. **Run one GPU job at a time.** Serialize them with a lock (`scripts/queue.sh`). - Never use `pkill -f`/`pgrep -f` patterns that can match your own shell. - Detach long queues (`setsid nohup … &`) so they survive agent restarts. 5. **Enforce the memory cap with a watchdog** that kills the run. - On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g. `drm-resident-*` in `/proc/<pid>/fdinfo`, `nvidia-smi`, or `ioreg`/`vm_stat` on Apple) plus process anonymous memory. 6. **Stability gates.** A config is only "best" after it passes: - a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly; - a soak with concurrent full-length requests, if the KV pool is sized for N users; - a greedy output equivalence check, if the change claims to be lossless. 7. **Never force-kill a server mid-batch.** Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults. 8. **Verify outputs, not just speed.** Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output. 9. **Log everything in `PLAN.md`,** including failures and why they failed. Negative results are the most reusable output. 10. **Secrets:** never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published. ## 3. The loop ``` baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls ↑ ↓ └──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘ ``` ### Lever order Cheapest first; see `references/levers.md` for evidence. 1. **Persistent JIT, compile and graph caches** (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow. 2. **Decode graphs** (CUDA/XPU graphs): often 2× c1 on their own. 3. **Speculative decoding:** - Choose the method (built-in MTP, EAGLE, a draft model, n-gram). - Sweep draft length per concurrency and context; the best length differs. - Prune the draft LM head's vocabulary. - Keep sampling on the GPU. - Make sure verify runs with decode-style attention. 4. **Concurrency and KV sizing:** - Set max running requests and the KV pool to N × context. - For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency. - Explicitly enable prefix caching and confirm `cached_tokens` > 0. 5. **Prefill:** - Tune chunk size; it trades throughput against decode stalls. - Use a faster attention library. - Use int8/fp8 prefill GEMMs as a labelled quality trade. - Avoid padding cliffs in grouped MoE GEMMs. 6. **Kernel thread and tile sizes:** defaults are often tuned for a bigger card. 7. **Kernel work:** profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau. ### When a lever fails - **If kernel-count reductions (fusion) give nothing** under graphs, the time is inside the main kernels. Stop fusing. - **If higher acceptance doesn't give higher tok/s,** the draft costs too much. Measure per-step time. - **If a weird result appears,** suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery. ### Plateau Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in `PLAN.md` with: - best numbers vs baseline vs ceiling; - what limits further gains, with evidence; - a ranked list of the remaining levers and their estimated cost. Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back. ## 4. Qualify and publish See `references/publishing.md`. - Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree. - Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed. - Then A/B the clean build against your best image to prove they match. - Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable. ## 5. Reporting to the user - **Start with a table:** metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness). - **Then:** what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted). - **Say plainly** what was not verified. - **When asked to free the GPU:** drain, stop the server, confirm no processes and an idle clock, and end any loop.
© sybil-solutions · MIT · sybil-solutions/inference-optimisation-skill