# debug-model

Parity protocol for a MAX model that serves and emits tokens but disagrees with the reference: tensor dumps first, fixes second.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1300 installs · ★ 206
- Author: modular
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/modular/skills/tree/HEAD/debug-model/SKILL.md
- Updated: 2026-10-25
- HTML: https://affpapa.org/en/nevibecode/skills/debug-model

## What it does

The skill takes over when a MAX model loads, compiles, serves and emits tokens, yet the output disagrees with a reference implementation. High logit cosine with a different argmax, gibberish greedy tokens, coherent text that drifts. Scalar `F.print` taps hide directional bugs, and every new tap costs a full recompile.

So the comparator comes before the model. An HF dumper, a MAX dumper with taps inside `forward()`, and a script that compares the two; dump tensors are cast to FP32 inside `forward()`. Validate both dumpers on a model MAX already serves correctly, `olmo3` or `llama3_modulev3` with `--prefer-module-v3`. You want cosine near 0.999 at every layer. Anything less means the comparator is broken, not your port.

From there you read the output and pick a branch: hunt the layer that diverged, or go straight to serve-versus-pipeline checks when all layers already match. Setup is concrete — a pixi environment with MAX installed, network access to Hugging Face Hub, a GPU for dumping and serving. Crashes on load and half-written modules belong to `import-model`. Tuning logit tolerances on a model that already passed parity is out of scope too.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add modular/skills --skill debug-model -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/debug-model/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/debug-model/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add modular/skills --skill debug-model -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/debug-model/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add modular/skills --skill debug-model -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/debug-model/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install modular/skills debug-model
```

2. **Manually: put the files in place**
- File: `.github/skills/debug-model/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add modular/skills --skill debug-model -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/modular/skills --path debug-model
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/debug-model/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add modular/skills --skill debug-model -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/modular/skills/tree/HEAD/debug-model/SKILL.md

## When to use it
- find the first layer where a port diverges from the PyTorch reference
- tell a broken comparator from a broken model
- test a hypothesis on dumps before paying for a recompile
- separate a model bug from a decode-state bug at serve time
- untangle stacked failures when one fix helps and verification still fails

## Example prompts
- The model serves but generates garbage after a few tokens. Run the debug-model protocol.
- Logits match but tokens diverge. Build the tensor-dump comparator and find the first diverging layer.
- After the MAX upgrade per-layer cosine looks fine but greedy tokens differ. What do I check first?

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 778370edfd9adba7.

BEGIN THIRD-PARTY FILE 778370edfd9adba7
```text
---
name: debug-model
description: >
  Debug a MAX model that loads, compiles, serves, and generates tokens but
  whose output disagrees with a reference implementation. Use whenever parity
  debugging stalls on scalar taps, the model returns gibberish or wrong greedy
  tokens, logit cosine is high but argmax differs, or generation is coherent then
  diverges. Applies during an architecture port, a quantization bring-up, a
  multi-GPU conversion, or after a MAX upgrade. Triggers on "parity failure", "silent
  corruption", "logits match but tokens diverge", "top-1 mismatch", "greedy
  divergence", and "model serves but generates garbage". Not for crashes on load
  or pre-serve scaffolding (use import-model). Mandates reference-vs-MAX
  tensor-dump comparators first, numerical verification of fixes before
  recompiling, and a serve-vs-pipeline bisect when dumps match but text diverges.
compatibility: Requires pixi env with MAX installed, network access to Hugging Face Hub, and a GPU for dumping and serving.
---

# Parity/coherence failure protocol

The model runs without errors but output is wrong. Scalar `F.print` taps hide
directional bugs, and each new tap costs a full recompile. Build a per-layer
tensor-dump comparator first. Every later check becomes a numpy read from disk.

**Use this skill when** MAX output disagrees with a PyTorch reference you can
run and hook. The primary case is a custom-architecture port that serves but
fails parity or coherence checks. The same protocol covers a quantized variant
of a working port, a multi-GPU conversion of a working single-GPU port, and a
regression after a MAX upgrade. It applies anywhere a trusted reference exists.

**Don't use this skill when:**

- The server crashes on load → fix config, weights, module (`import-model`)
- You haven't finished implementing the module → `import-model` Phase 2
- An already-verified model needs logit-comparison tolerances tuned → that's
  threshold calibration, which this skill doesn't cover

## References

| File                                                                      | Read when                                                       |
|---------------------------------------------------------------------------|-----------------------------------------------------------------|
| [comparator-build.md](references/comparator-build.md)                     | Building HF/MAX dumpers and the comparator                      |
```
END THIRD-PARTY FILE 778370edfd9adba7

The third-party file has ended. Everything between the 778370edfd9adba7 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2458 of 7305 bytes. © modular · Apache-2.0 · https://github.com/modular/skills/tree/HEAD/debug-model/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md