Skip to main content
skill CX Codex

diagnose-run — Claude Code skill: diagnosis and installation

The diagnose-run Claude Code skill traces Evolving Infra startup failures, request errors, and benchmark regressions. It activates when evolving-infra serve exits or a run falls behind its reference.

Apache-2.0 ★ 173 updated 2026-11-16

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add mit-han-lab/EvolvingInfra --skill diagnose-run -a claude-code
Open on GitHub

What diagnose-run does

The skill uses run logs and the code of the components they point to, then identifies why Evolving Infra failed to start, serve a request, or match a reference score. For a slower benchmark, it compares runs over matching time windows and examines planner switches, replay settings, and processing stages.

It reports the cause with supporting log lines or measurements before suggesting a fix in the affected component. It leaves code unchanged until the cause is identified. Confirming a fix with another run uses GPUs, so the owner must approve that rerun first.

What it helps with

  • find why serve exited or never printed ready
  • trace a failed request or replay through the logs
  • compare a slow run with its reference over matching windows
  • pinpoint which component produced a startup error

Example prompts for the agent

  • Find why evolving-infra serve exited before printing ready.
  • Trace the replay failure in the logs and locate the request error.
  • Compare this run with its reference and identify the slower stage.

How to install diagnose-run

1Skills CLI (Vercel)

npx skills add mit-han-lab/EvolvingInfra --skill diagnose-run -a claude-code

2Manually: put the files in place

~/.claude/skills/diagnose-run/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/diagnose-run/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about diagnose-run

How do I install diagnose-run in Claude Code?

Run npx skills add mit-han-lab/EvolvingInfra --skill diagnose-run -a claude-code. The skill file is installed at ~/.claude/skills/diagnose-run/SKILL.md.

What does diagnose-run do?

It investigates Evolving Infra startup and request failures, and compares slow benchmark runs with a reference. Its findings include supporting logs or measurements and a proposed fix.

How is diagnose-run different from run-benchmark?

diagnose-run investigates failures or performance gaps in an existing run. Use run-benchmark to run or score a benchmark.

Source text

SKILL.md · Apache-2.0
---
name: diagnose-run
description: Find why an Evolving Infra run failed to start, failed a request, or scored below a reference. Use when `evolving-infra serve` exits or never prints ready, a replay or request errors, or a benchmark is slower than expected. Not for running or scoring a benchmark; use `run-benchmark`.
---

# Diagnose a run

Find the root cause of a run that did not start, failed a request, or scored below a
reference, with the log lines or numbers that show it.

## Scope

- **Reads**: `repository`: the run's logs and the code of every component they point to.
- **Writes**: nothing until the cause is named; then the fix, in the component the cause is in.
- **Never changes**: any code, to make a symptom go away before its cause is named.
- **Outside the repository**: none. A rerun to confirm a fix uses GPUs: confirm it with the
  owner first.

## Inputs

- The work directory of `evolving-infra serve`, or `runs/<job id>/` of `submit.py`.
- What was expected: ready, a reply, or the reference result being compared with.

## Steps

Go to the step that matches the symptom.

### 1. It did not start

`serve` stops when a process it started exits, and names every one that did:
`RuntimeError: engine-w1 exited with -9; see <dir>/logs/engine-w1.log`. Under `submit.py`
that line is in `<run>/serve.log`. Find the first error in each named log (engine logs are
long: weight loading and graph capture print progress lines):

```bash
grep -n -E "Error|Address already in use|Cannot assign" <dir>/logs/<name>.log | grep -v 'File "' | head -5
```

| Message | Cause | Fix |
|---|---|---|
| `Loaded weights leave no GPU memory for the KV cache under --mem-fraction-static=X. Raise ... above Y` | the weights fill the worker's GPUs; typical when a config written for one worker per host runs on one host, where each worker gets half the GPUs | the one-node config, or `--mem-fraction-static` above Y, or more GPUs per worker |
| `ports already in use on this host: ...` (from `serve`) | another process holds a port the layout needs | stop that process, or change the port in `evolving_infra/launch/layout.py` |
| `Address already in use (addr='tcp://<ip>:<port>')`, `OwnerPlane failed to bind` | the port is in another engine's range, or a host service holds it | each engine takes two ports per GPU from its `--disaggregation-bootstrap-port` (in its launch line at the top of the log): if the port is in another engine's range the ranges overlap, else `ss -ltnp \| grep :<port>` names the service; fix in `evolving_infra/launch/layout.py` |
| `Cannot assign requested address (addr='tcp://<ip>:...')` | the engine was given another host's address | compare `<ip>` with this host's `host N: <ip>` line in `serve.log`; they differ when a host's rank does not match its place in `--hosts`: let `serve` find the host by name, do not pass a scheduler's node index |
| router log: a worker is refused, or `engine must enable ...` | the engine and the router run different presets, or engine code read a raw argument | the same preset on both sides; in engine code `resolution_result` (`evolving_infra/engine/AGENTS.md`) |

Nothing exited and no `ready` after 40 minutes: read the tail of each engine log; an engine
still loading weights or capturing graphs is slow, not stuck.

**Check**: the named cause matches the first error of the log, and the fix makes `serve`
print `ready`.

### 2. A request or the replay fails

| Message | Cause | Fix |
|---|---|---|
| replay: `The api_key client option must be set` | the client needs any key | `api_key=EMPTY` |
| `score.py`: `no lifecycle events` | the replay never ran | read `bench.log` |

For a failing request in a running stack, follow it through the logs in this order: `router`
(which P and D it chose), the P's engine log, the D's engine log.

**Check**: the logs of router, P and D show where the request failed, and after the fix the
request returns a reply.

### 3. Slower than the reference

A run can differ from the reference in more than one way: go through items 1 to 5 below for
both runs before concluding. `<run>/bench` is the directory with `trie-lifecycle-*.jsonl`; scripts
are in `skills/diagnose-run/scripts/` and `skills/run-benchmark/scripts/`.

1. **When the planner switched.** `switch_times.py <planner dir> <run>/bench` for both runs
   prints the switches on the replay clock. Pick, for each P/D split, a window in which both
   runs held that split (start 120 s after a switch) and compare `benchmarks/replay/score.py`
   on it. A gap only over the whole run is a gap in when the planner switched.
2. **Same work.** `request_breakdown.py <run>/bench --skip-s S --window-s W` for both. If
   `replay_context_modes`, `uncached_tokens_mean` or `output_tokens_mean` differ, the replay
   settings or the inputs differ. `<run>/bench/replay-settings-0.json` has the resolved
   settings of the run and, under `differs_from_preset`, each one that is not the paper's.
3. **Which stage.** The same output gives `ttft_s_*` (router, prefill, transfer, decode
   admission) and `decode_ms_per_token_*` (decode).
4. **Decode.** `engine_batches.py decode <run>/bench --skip-s S --window-s W <logs>` for both,
   with the logs of the workers that were D in the window: worker `w<i>` logs to
   `engine-w<i>.log`, workers are numbered in the order of `--hosts`, the first
   `--prefill-workers` start as P, and `switch_times.py` shows later changes. Check the
   "window in log time" line it prints against the log's own timestamps. Compare rows with
   the same running requests: equal `ms/step` with a lower `accept len` means speculative
   decoding accepts less (the outputs differ: step 2's replay settings, the draft
   configuration); a higher `ms/step` means the decode step itself is slower (kernels, GPU).
5. **Prefill**, when time to first token differs and decode does not:
   `engine_batches.py prefill ...` on the P workers gives new tokens per second and queue
   lengths.
6. **Noise.** Repeat runs of one configuration differ by about 4 % in requests per second.

**Check**: each of items 1 to 5 has numbers for both runs on the same window.

## Returns

- The root cause, with the log lines or numbers that show it.
- The fix, or the decision that is the owner's.
- For a slower run: each step's numbers for both runs on the same window, and what differs.

© mit-han-lab · Apache-2.0 · mit-han-lab/EvolvingInfra

Similar skills

All skills for DevOps and infrastructure →

From the same set7

Project sponsors

Companies that keep this analytics open