Skip to main content
skill CX Codex

discover-training-system — Claude Code skill: setup and use

The discover-training-system Claude Code skill inspects a training setup before performance changes. It runs at task start and when a setting appears to have no effect.

Apache-2.0 ★ 99 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add TokenRhythm/InfraRSI --skill discover-training-system -a claude-code
Open on GitHub

What discover-training-system does

The skill examines hardware topology, NUMA and pinned-memory limits, the environment actually applied to a job, and the code path that runs. It treats receipts as evidence; a flag value alone does not confirm that its feature was used.

Its diagnostic steps use read_env, read_receipt, and profile_ranges; topology findings can be saved with write_memory. Phase timings and StepTime are compared for the same rank and step. A new execution path needs a kernel-call count or a receipt field written only by that path. On multi-node jobs, NCCL falling back to sockets is a failure signal.

What it helps with

  • identify whether inter-node traffic uses IB
  • trace slow ranks to a NUMA node or socket
  • check whether a setting reaches the executed path
  • compare phase timings before and after a change

Example prompts for the agent

  • Check the GPU topology and find out whether IB is in use.
  • Investigate why changing the flag left StepTime unchanged.
  • Find out why ranks on one node are slower.

How to install discover-training-system

1Skills CLI (Vercel)

npx skills add TokenRhythm/InfraRSI --skill discover-training-system -a claude-code

2Manually: put the files in place

~/.claude/skills/discover-training-system/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/discover-training-system/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about discover-training-system

How do I install discover-training-system in Claude Code?

Run npx skills add TokenRhythm/InfraRSI --skill discover-training-system -a claude-code, then use ~/.claude/skills/discover-training-system/SKILL.md.

What does discover-training-system do?

It examines hardware topology, the job's effective environment, and the code path that actually runs. It uses receipts and phase profiles to check those findings.

How can I confirm that a new code path ran?

Confirm the new kernel's call count per update or check for a receipt field written only by that path. The flag value by itself is not proof.

Source text

SKILL.md · Apache-2.0
---
name: discover-training-system
description: Establish what the training system really is before optimizing it - GPU and interconnect topology, NUMA and pinned-memory limits, effective environment, and which code path actually executes.
license: Apache-2.0
when_to_use: Start of every task; a receipt contradicts the expected hardware behaviour; a flag or env change shows no effect; ranks on some nodes are consistently slower; 'is IB used', 'NVLink topology', 'NUMA', 'pinned memory', 'why did the flag do nothing'.
metadata:
  author: InfraRSI A0 (human-authored)
  source: human-authored (A0)
  version: 1
  tags: [discover, hardware, topology, nvlink, infiniband, numa, single-node, multi-node, h100-80gb]
---

# Discover the training system

The HardwareProfile from `discover` is read-only input (read_env). This skill says how to interpret it and which
facts must be confirmed from receipts rather than assumed.

## Trigger conditions
- First turn of a task (read_env before the first proposal).
- A configuration change produced no StepTime change at all (suspect the setting never reached the executed path).
- Persistent stragglers: `per_step[].slowest_rank` keeps naming ranks on one node or one socket.
- Multi-node tasks, any EP/DP traffic crossing nodes.

## Required observations (receipt fields)
- `env` (versions, hostnames, GPU names, whitelisted env as actually applied), `nodes`, `world_size`.
- `per_step[].slowest_rank`, `memory.per_rank[]` (rank spread), `phases.startup/load/build`.
- HardwareProfile: GPU model and memory, `nvidia-smi topo -m` matrix, IB port state/rate, NUMA nodes per socket, RAM, memlock, software versions.

## Rules: topology
- WINH nodes: 8x H100 80GB, every GPU pair NV18 (18 NVLink links per GPU at 26.562 GB/s each); eight 400 Gb/s IB HCAs Active/LinkUp. Keep TP and the largest all-to-all (EP) inside a node first; cross-node traffic is DP/EDP gradient reduction or EP across IB.
- Measured NCCL bus bandwidth at 256 MiB (inter-node runs, NVLink included, not NIC line rate): all-reduce 234.47 GB/s, all-gather 139.11, reduce-scatter 135.19, all-to-all 44.96 GB/s. All-to-all is the expensive collective; size EP accordingly.
- `ibdev2netdev` may print nothing and NVLink traffic counters may be unsupported by the installed nvidia-smi; absence of a counter is not absence of the link. Confirm IB use from NCCL init lines (NET/IB with GPUDirect RDMA), not from tool output gaps.
- The shell can see fewer CPUs than `lscpu` reports (affinity/cgroups: 168 vs 192 on several WINH nodes). Size OMP_NUM_THREADS from the affinity mask, not from lscpu.

## Rules: NUMA and pinned memory
- Some nodes run sub-NUMA clustering (2 NUMA nodes per socket), others one. Pinned host buffers allocated without binding can land on the remote NUMA node and slow host-side optimizer work or H2D/D2H copies on a subset of ranks.
- SSH `ulimit -l` does not prove the training process can pin; a RAM snapshot is not a reservation. Large core dumps in /dev/shm once consumed about 0.5 TB per node and pushed pinned memory remote; core files must be disabled.

## Rules: installed is not executed
- The effective environment can differ from the job literal: a launch shell set CUDA_DEVICE_MAX_CONNECTIONS=1 over a job value of 8. Read the applied env from the receipt.
- A feature flag without a consumer does nothing ("pseudo" variants passed review because their flags were never read). Prove execution by call counts or a changed kernel trace, not by the flag value.
- A dispatch mode or wrapper can silently route a fused kernel to a dense eager fallback. If a fused path shows exactly the old time, assume a fallback until a count proves otherwise.

## Allowed actions
- read_env; read_receipt and profile_ranges on the start candidate before proposing.
- L0 env: OMP_NUM_THREADS (host threads, relevant with optimizer offload), NCCL_* performance items of the whitelist. Topology selection (NCCL_IB_HCA, NCCL_SOCKET_IFNAME) is owned by discover, not by the agent.
- write_memory: record topology facts that explained a result (with hardware tags).

## Failure signals
- A change with an expected mechanism leaves StepTime and every phase unchanged (flag not consumed).
- `slowest_rank` concentrated on ranks of one node or socket; reserved memory identical but optimizer phase slower on some ranks.
- NCCL init falling back to sockets on a multi-node task.

## Verification
- Same-rank, same-step comparison of phases before and after (profile_ranges), plus the whole-update StepTime beyond the margin.
- For an execution-path claim: a count (calls of the new kernel per update) or a receipt field that only the new path writes.

## Success case (real)
- Flagship (32x H100, real SFT). WINH4/5 run two NUMA nodes per socket, WINH6/7 one; unbound pinned expert state landed on the remote NUMA node and made some ranks' optimizer 1-1.5 s slower. Binding each rank to its GPU's socket (sched_setaffinity) with MPOL_INTERLEAVE before CUDA init: 14.485 -> 14.059 s/step, optimizer phase 3.388 -> 3.082 s, all ranks' optimizer within 2.96-3.06 s, bitwise identical scalars. (reports/flash_next_mfu_20261003/PROGRESS.md, batch 3)

## Failure case (real)
- NeMo CP4 run had `qsa_routes_fast=true`, yet shards carrying a query-position offset or a global length fell back to the original selector (336 nonzero, 179 scalar reads on one 8,192-query shard). The flag alone produced none of the later gain; only fixing the fallback condition did. Lesson: verify the executed path per shard/rank, not the switch. (reports/agentic_moe_sft_paper_20261018/MANUSCRIPT.md 4.1, 6.2)

© InfraRSI A0 (human-authored) · Apache-2.0 · TokenRhythm/InfraRSI

Similar skills

All skills for AI and ML →

From the same set19

The whole set →
validate-training-lifecycle The validate-training-lifecycle Claude Code skill checks a model-training run from checkpoint save through fresh-process resume, HF export, and forward comparison. CX Codex +1 ★ 99repo stars optimize-training-kernels The optimize-training-kernels Claude Code skill adjusts fusion settings and attention backends while checking numerical tolerances. It is triggered by slow kernels, proposed kernel changes, or numerical drift. CX Codex +1 ★ 99repo stars recompute-policy The recompute-policy Claude Code skill helps choose full, selective, or no recomputation based on memory headroom and model structure. It activates for activation-checkpointing changes, OOMs, or recompute_modules decisions. CX Codex +1 ★ 99repo stars moe-ep-communication The moe-ep-communication Claude Code skill configures expert parallelism and token dispatch for MoE in Megatron-Core 0.18.2. It activates for EP sizing, dispatcher choices, and profiling bottlenecks. CX Codex +1 ★ 99repo stars manage-memory-lifetimes The manage-memory-lifetimes Claude Code skill analyzes GPU memory failures and helps select residency, offload, and allocator settings. It activates for OOMs, allocator retries, or tight memory headroom. CX Codex +1 ★ 99repo stars verify-training-semantics The verify-training-semantics Claude Code skill checks training changes against fixed invariants and numerical limits. It runs before run_screen, when reviewing invalid receipts, and when changes touch training behavior. CX Codex +1 ★ 99repo stars choose-parallel-layout The choose-parallel-layout Claude Code skill selects a parallel layout from per-rank memory limits and collective costs. It activates for OOM incidents, long sequences, MoE placement, or multi-node scaling. CX Codex +1 ★ 99repo stars infrarsi-optimize The infrarsi-optimize Claude Code skill tunes an existing GPU training job while keeping its training task unchanged. It tests launch settings and code edits to reduce step time or improve throughput. CX Codex +1 ★ 99repo stars

Project sponsors

Companies that keep this analytics open