# jetson-speculative-decoding

A skill that adds EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when token generation is the slow part.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1100 installs · ★ 3735
- Author: Jetson Team
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/NVIDIA/skills/tree/HEAD/skills/jetson-speculative-decoding/SKILL.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/skills/jetson-speculative-decoding

## What it does

The skill edits a vLLM server you already run on Jetson: it appends `--speculative-config` to the `vllm serve` command from the `jetson-llm-serve` recipe. A small draft model proposes tokens, the target model verifies them in one forward pass, and per-token latency drops.

On Jetson the outcome depends on VRAM headroom, not on draft quality. That is why the skill allows EAGLE-3 on Thor and AGX Orin only, and tells you not to try it on Orin Nano or NX: there is rarely room for both target and draft, so startup ends in OOM. It suggests starting `num_speculative_tokens` at 5 on Thor and 3 on AGX Orin, and dropping `--gpu-memory-utilization` by about 0.05 against the non-speculative baseline. The gain shows up on single-stream or low-concurrency work, up to 2 requests; at 8 and above batched decode usually wins and the draft model just steals memory.

You need a baseline from `jetson-llm-benchmark` and either a published EAGLE-3 head or a small same-family draft model. After enabling it, the skill has you benchmark at `--concurrency 1` before and after: if the gain is under 10%, or throughput regresses at concurrency 8, turn speculation off.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill jetson-speculative-decoding -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/jetson-speculative-decoding/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/jetson-speculative-decoding/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill jetson-speculative-decoding -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/jetson-speculative-decoding/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill jetson-speculative-decoding -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/jetson-speculative-decoding/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install NVIDIA/skills jetson-speculative-decoding
```

2. **Manually: put the files in place**
- File: `.github/skills/jetson-speculative-decoding/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill jetson-speculative-decoding -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/NVIDIA/skills --path skills/jetson-speculative-decoding
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/jetson-speculative-decoding/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill jetson-speculative-decoding -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/NVIDIA/skills/tree/HEAD/skills/jetson-speculative-decoding/SKILL.md

## When to use it
- speed up token generation on a Jetson vLLM server when TTFT is fine
- pick a speculative token count for Thor or AGX Orin
- check whether a draft model earns back the VRAM it takes
- free memory for the draft model without breaking server startup
- roll back speculation when the benchmark shows a regression

## Example prompts
- Append `--speculative-config` with the eagle3 method to the `vllm serve` command from `jetson-llm-serve` and tune the speculative token count for Thor
- Compare `throughput_tok_s` and `tpot_ms_p50` at `--concurrency 1` before and after enabling speculative decoding
- Startup OOMs after I turned on speculation — what should I lower first?

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 946ff15e971135ce.

BEGIN THIRD-PARTY FILE 946ff15e971135ce
````text
---
name: jetson-speculative-decoding
description: Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.
version: 0.0.1
license: "Apache-2.0"
metadata:
  author: "Jetson Team"
  tags: [jetson, llm, speculative-decoding]
  languages: [markdown]
  data-classification: public
---

# Jetson Speculative Decoding (vLLM)

Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know.

## Purpose

Tune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.

## When to use

- TPOT/ITL is the bottleneck (TTFT is fine, output is just slow).
- Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.
- Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.

## When NOT to use

- High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.
- Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix".
- After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.

## Prerequisites

- A working vLLM server recipe from `jetson-llm-serve`.
- Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.
- A benchmark baseline from `jetson-llm-benchmark` before enabling speculation.
- A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.

## Instructions

Append `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`.

EAGLE-3 (preferred when a head is published for the target model):

```bash
--speculative-config '{
  "method": "eagle3",
  "model": "<eagle3-head-repo-id>",
  "num_speculative_tokens": 5,
  "draft_tensor_parallel_size": 1
}'
```
````
END THIRD-PARTY FILE 946ff15e971135ce

The third-party file has ended. Everything between the 946ff15e971135ce tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2260 of 4832 bytes. © Jetson Team · Apache-2.0 · https://github.com/NVIDIA/skills/tree/HEAD/skills/jetson-speculative-decoding/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md