Skip to main content
skill CX Codex

validate-training-lifecycle — Claude Code skill: checks and setup

The validate-training-lifecycle Claude Code skill checks a model-training run from checkpoint save through fresh-process resume, HF export, and forward comparison.

Apache-2.0 ★ 99 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add TokenRhythm/InfraRSI --skill validate-training-lifecycle -a claude-code
Open on GitHub

What validate-training-lifecycle does

The skill checks whether a training configuration completes a full lifecycle: train, save a checkpoint, resume in a new process, and compare results. It calls for comparing resumed scalars with an uninterrupted run within tolerance.

For HF export, it requires a real reload and a forward comparison on a fixed batch; a completion flag alone does not verify the weights. It also specifies a collective on the existing grad-stats process group before checkpoint loading, and records applied environment variables, per-rank memory, events, and startup phases. A resume can hit OOM at the first gradient-norm all-reduce even when the loaded state matched byte for byte.

A passing check requires successful resume, scalar agreement within tolerance, verified export, a passing forward comparison, and zero OOMs. The project needs an existing training setup and grad-stats process group; the specified collective does not create a new group.

What it helps with

  • check checkpoint resume in a fresh process
  • compare resumed scalars with an uninterrupted run
  • verify HF weights by reloading and running a forward comparison
  • identify OOM at the first post-resume NCCL all-reduce
  • record applied environment variables and per-rank memory

Example prompts for the agent

  • Check the full training lifecycle before promoting this configuration.
  • Compare the checkpoint resume with the uninterrupted run.
  • Verify the HF export by reloading it and comparing a fixed batch.

How to install validate-training-lifecycle

1Skills CLI (Vercel)

npx skills add TokenRhythm/InfraRSI --skill validate-training-lifecycle -a claude-code

2Manually: put the files in place

~/.claude/skills/validate-training-lifecycle/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/validate-training-lifecycle/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about validate-training-lifecycle

How do I install validate-training-lifecycle in Claude Code?

Run npx skills add TokenRhythm/InfraRSI --skill validate-training-lifecycle -a claude-code. The skill file is placed at ~/.claude/skills/validate-training-lifecycle/SKILL.md.

What does validate-training-lifecycle do?

It checks checkpoint saving, resume in a fresh process, HF export, and forward results against a fixed batch. It also records memory and environment details.

What does validate-training-lifecycle need to run?

The training setup must have an existing grad-stats process group, which the skill uses for its initial NCCL collective. Export verification requires reloading the HF weights and comparing a fixed batch.

Source text

SKILL.md · Apache-2.0
---
name: validate-training-lifecycle
description: Make a fast configuration survive the real job - checkpoint, fresh-process resume, HF export and forward compare - including first communication allocations, common startup, NCCL and allocator environment.
license: Apache-2.0
when_to_use: Before promoting a candidate; any change to memory residency, NCCL env, allocator config or startup order; OOM or NCCL failure after resume; 'checkpoint', 'resume', 'export', 'NCCL env', 'CUDA_DEVICE_MAX_CONNECTIONS', 'expandable segments'.
metadata:
  author: InfraRSI A0 (human-authored)
  source: human-authored (A0)
  version: 1
  tags: [lifecycle, checkpoint, resume, export, nccl, allocator, startup, env, dense, moe, multi-node]
---

# Validate the training lifecycle

Promotion requires the lifecycle check: train -> save -> new process resume -> +N updates -> HF export through the
bridge -> forward compare on a fixed batch. A configuration that only works from a fresh start is not deliverable.

## Trigger conditions
- A candidate is about to be finished/promoted, especially one near the memory cap.
- Changes to PYTORCH_CUDA_ALLOC_CONF, NCCL_* env, CUDA_DEVICE_MAX_CONNECTIONS, optimizer offload or state placement.

## Required observations (receipt fields)
- `lifecycle` (saved_at, resumed_ok, resume_scalars_match, export_ok, forward_max_abs_logit_diff, forward_ok) for lifecycle runs; `memory.per_rank[]`, `events`, `phases` (startup/load/save/export), `env`.

## Rules
- There are two "first steps": the first update of this process (local first-use gates and lazily built references run again after resume) and global update 1 (optimizer moments are created lazily). A resumed process meets both at once, with checkpoint-loaded state already resident: its memory at the first collective differs from a fresh run.
- NCCL allocates communicator buffers lazily at a group's first collective, outside the PyTorch allocator. Common startup: after the optimizer and norm groups are installed and before the checkpoint load, issue one FP32 [1] zero SUM on the existing grad-stats process group on the current stream (no new group, not WORLD, no empty_cache, no env change). Fresh-start cost measured 0.84-1.57 s per rank with zero retries/OOMs.
- Load-point exact is not trajectory exact: a resume where every byte matched at load still produced non-bitwise later state, and CPU RNG differed by one DataLoader draw. Compare scalars after resume against the uninterrupted run within the tolerance.
- HF export writes inference weights, not optimizer state; verify it by a real reload and a forward comparison, not by a "complete" flag.
- CUDA_DEVICE_MAX_CONNECTIONS=1 is what Megatron relies on to order TP/SP communication before dependent compute; FR uses 1. Launch shells may override the job value: read the applied env.
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True was the production setting on the flagship; with it, retries can succeed without an OOM event (retries still count).
- NCCL env recorded on the flagship: NCCL_IB_DISABLE=0, NCCL_BUFFSIZE=1048576, NCCL_NVLS_ENABLE=0. An NCCL_BUFFSIZE change was non-bitwise and not adopted. Topology variables are owned by discover; only performance variables are actions.

## Allowed actions
- L0 env: PYTORCH_CUDA_ALLOC_CONF, CUDA_DEVICE_MAX_CONNECTIONS, NCCL performance variables of the whitelist; memory levers that restore headroom for resume.
- Prefer, at equal StepTime, the candidate with more headroom on the tightest rank: resume adds pressure that a fresh screen does not show.

## Failure signals
- Resume OOM inside NCCL at the first gradient-norm all-reduce; retries only after resume; export reload mismatch; resumed scalars outside tolerance.

## Verification
- Lifecycle receipt: resumed_ok, resume_scalars_match, export_ok and forward_ok all true, 0 OOMs, peak within cap.

## Success case (real)
- Real-SFT baseline (10/13): native checkpoint after update 2, resume for the remaining 1,022 updates; dev CE max relative difference 0.0418666% over 7 evaluation points, both runs best at update 768; step-1024 HF export verified by reload; mean 10.6186055854 s per update (updates 5-1024). (MANUSCRIPT.md 6.8; project_learning_handoff_20261018/LATER_PROJECT_NOTES.md 2)

## Failure case (real)
- Q resume v4 (flagship, 32 GPUs): continuous updates 1-6 with a save at 3 succeeded; resuming from 3 loaded on all ranks and then failed at absolute update 4 inside NCCL at the first gradient-norm all-reduce ("NCCL WARN Cuda failure 2 'out of memory'" on ranks 1, 4, 16, 28) while torch counted 0 OOMs. On rank 28, allocated memory was equal to the fresh run but reserved was 72.332 vs 70.943 GiB and free 0.167 vs 1.156 GiB. The fix direction was the common-startup collective above. (rsi_infra_execution_20261018/diagnostics/q_resume_nccl_oom_v1/SUMMARY.md; MANUSCRIPT.md 6.7)

© InfraRSI A0 (human-authored) · Apache-2.0 · TokenRhythm/InfraRSI

Similar skills

All skills for AI and ML →

From the same set19

The whole set →
weak-model-protocol The weak-model-protocol Claude Code skill sets rules for an optimizer driven by a weaker or cheaper language model. It activates when actions are malformed, context runs short, or the provider returns errors. CX Codex +1 ★ 99repo stars optimize-data-path The optimize-data-path Claude Code skill optimizes data and state transfers between the host and GPU without changing the data contract. It is triggered by costly loading, host synchronization, or optimizer-state transfers. CX Codex +1 ★ 99repo stars learn-optimization-skills The learn-optimization-skills Claude Code skill guides an optimizer’s learning between development tasks and records reusable lessons. It runs for post-task reflection or when a search turns up a reusable finding. CX Codex +1 ★ 99repo stars profile-whole-update The profile-whole-update Claude Code skill measures full-update time and checks whether a change improved the end-to-end update. It comes into play when reviewing profiles, comparing candidates, or tracing delays between steps. CX Codex +1 ★ 99repo stars optimize-training-kernels The optimize-training-kernels Claude Code skill adjusts fusion settings and attention backends while checking numerical tolerances. It is triggered by slow kernels, proposed kernel changes, or numerical drift. CX Codex +1 ★ 99repo stars recompute-policy The recompute-policy Claude Code skill helps choose full, selective, or no recomputation based on memory headroom and model structure. It activates for activation-checkpointing changes, OOMs, or recompute_modules decisions. CX Codex +1 ★ 99repo stars moe-ep-communication The moe-ep-communication Claude Code skill configures expert parallelism and token dispatch for MoE in Megatron-Core 0.18.2. It activates for EP sizing, dispatcher choices, and profiling bottlenecks. CX Codex +1 ★ 99repo stars manage-memory-lifetimes The manage-memory-lifetimes Claude Code skill analyzes GPU memory failures and helps select residency, offload, and allocator settings. It activates for OOMs, allocator retries, or tight memory headroom. CX Codex +1 ★ 99repo stars

Project sponsors

Companies that keep this analytics open