Skip to main content
skill CX Codex

choose-parallel-layout — Claude Code skill for parallelism

The choose-parallel-layout Claude Code skill selects a parallel layout from per-rank memory limits and collective costs. It activates for OOM incidents, long sequences, MoE placement, or multi-node scaling.

Apache-2.0 ★ 99 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add TokenRhythm/InfraRSI --skill choose-parallel-layout -a claude-code
Open on GitHub

What choose-parallel-layout does

The skill chooses TP, PP, CP, EP, ETP, DP, micro-batch size, and gradient accumulation for a fixed global batch. Its calculations account for rank memory, collective costs, and legal process groups. Triggers include OOM, less than 4 GiB of headroom, sequences of 32K tokens or more, and multi-node runs.

Its rules cover parameter, gradient, optimizer, and activation memory, along with layout constraints. For example, TP is limited to 8 within a single NVLink node; SFT permits CP values of 1, 2, or 4. The suggested order is to fit the workload with the least time cost, then remove memory-saving settings where headroom allows.

The skill uses report fields for peak reserved memory, per-rank usage, allocator retries, and step time. A full run passes the memory check with zero OOMs, preferably zero allocator retries, and at least 4 GiB free on the tightest rank.

What it helps with

  • choose TP and DP for each rank's memory budget
  • reduce OOM risk when placing MoE experts
  • weigh PP against gradient accumulation costs
  • validate process-group factorizations

Example prompts for the agent

  • Choose TP, PP, and DP for this global batch using the per-rank memory budget.
  • Analyze the OOM at step two and estimate optimizer memory for expert parameters.
  • Compare CP layouts for a sequence of 32K tokens.

How to install choose-parallel-layout

1Skills CLI (Vercel)

npx skills add TokenRhythm/InfraRSI --skill choose-parallel-layout -a claude-code

2Manually: put the files in place

~/.claude/skills/choose-parallel-layout/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/choose-parallel-layout/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about choose-parallel-layout

How do I install choose-parallel-layout in Claude Code?

Run npx skills add TokenRhythm/InfraRSI --skill choose-parallel-layout -a claude-code. The skill file is at ~/.claude/skills/choose-parallel-layout/SKILL.md.

What does choose-parallel-layout do?

It selects parallel dimensions and micro-batch settings for a fixed global batch, accounting for rank memory, collective costs, and legal process groups.

Which parallelism constraints does choose-parallel-layout check?

Among other rules, TP is capped at 8 within one NVLink node. SFT allows CP values of 1, 2, or 4; PT checks that sequence length is divisible by 2*CP.

Source text

SKILL.md · Apache-2.0
---
name: choose-parallel-layout
description: Choose TP/PP/CP/EP/ETP/DP, micro-batch and gradient accumulation for a fixed global batch from a per-rank memory budget and the cost of each collective; know which process groups are legal.
license: Apache-2.0
when_to_use: First proposals of a task; OOM or a tight rank (<4 GiB headroom); ample headroom with recompute still on; MoE expert placement; long sequences; multi-node scaling; 'which TP', 'EP or DP', 'PP bubble', 'micro batch', 'out of memory'.
metadata:
  author: InfraRSI A0 (human-authored)
  source: human-authored (A0)
  version: 1
  tags: [parallel, layout, memory, tp, pp, cp, ep, dp, micro-batch, dense, moe, long-context, multi-node, single-node]
---

# Choose a parallel layout

GBS (packs per update), sequence length and the data order are task constants. You choose how the same update is
split: `DP = world / (TP*PP*CP)`, `GA = packs_per_update / (DP*MBS)` (must be an integer), experts use
`EDP = world / (ETP*EP*PP)`.

## Trigger conditions
- Start config (ED) receipt shows OOM, allocator retries, or < 4 GiB headroom on the tightest rank.
- Headroom > ~20 GiB while full recompute or offload is still on (memory is being bought with time).
- MoE model, or sequence length >= 32K, or more than one node.

## Required observations (receipt fields)
- `memory.max_peak_reserved_gib`, `memory.per_rank[]` (tightest rank), `events.alloc_retries`, `events.alloc_ooms`.
- `step_time.mean_window_max_rank_s`, `profile` phases (fwd_bwd_s vs optim_s), `derived` dp/ga from propose_config feedback.

## Rules: memory budget per rank (bf16 params, fp32 grads, distributed optimizer)
- Weights 2 B/param + fp32 main grads 4 B/param on every rank holding the shard: dense parameters divided by TP*PP, expert parameters by EP*ETP*PP.
- Optimizer state (fp32 master + exp_avg + exp_avg_sq) 12 B/param, sharded over DP for dense and over EDP for experts. Large EP shrinks EDP and raises per-rank expert optimizer state and work.
- Activations without recompute are of order 34*s*b*h bytes per layer for a standard block (Korthikanti et al. 2022), more for wide SwiGLU MLPs; divided by TP when sequence parallel is on, and by CP. Full recompute keeps roughly the layer input (2*s*b*h) per layer plus one layer's activations during backward.
- Logits: s*b*V*4 bytes in fp32 for the loss, divided by TP (vocab-parallel). A 152K vocabulary at 32K tokens is about 19.9 GB per sequence before TP.
- Keep >= 4 GiB free on the tightest rank and ~8% for NCCL/cuBLAS workspaces; component peaks are not additive, so trust receipts over the ledger. An ideal shard ledger alone could not decide capacity in this project.

## Rules: cost of each dimension
- TP: two all-gathers and two reduce-scatters per layer per pass with SP; only inside an NVLink node (TP <= 8). Use the smallest TP that fits; turn on sequence_parallel whenever TP > 1 (MCore warns when EP and TP > 1 run without SP).
- PP: 1F1B without VPP idles (PP-1)/(GA+PP-1) of the update; with GA of 2-8 it is rarely worth it unless nothing else fits. PP needs num_layers % PP == 0 here.
- CP: splits the sequence; worth it when activations dominate at >= 32K. On SFT tasks (data D-SFT-*) every packed segment is padded to a multiple of 8 tokens and THD context parallelism needs segment lengths divisible by 2*CP, so CP must be 1, 2 or 4; CP=8 is rejected by the L0 validator before any screen. PT tasks (data D-PT-*) allow any CP the layout admits (seq_len % (2*CP) == 0).
- DP with the distributed optimizer: cheapest scaling; overlap_grad_reduce hides the reduce-scatter (see moe-ep-communication and profile-whole-update for verification).
- EP: keep the all-to-all on NVLink (EP <= 8 on one node) unless per-rank expert memory forces more; num_experts % EP == 0. ETP > 1 splits each expert GEMM and is usually slower than larger EP for small experts.
- MBS > 1 raises GEMM efficiency but multiplies activations; GA must stay an integer.

## Rules: legal process groups
- TP*CP*DP*PP = world for attention/dense, ETP*EP*EDP*PP = world for experts. These are two factorizations of the same ranks; EP is not multiplied with DP. world % (TP*PP*CP) == 0 and world % (ETP*EP*PP) == 0.
- Query groups (KV heads) and heads must divide TP, or TP must be a multiple of the query groups when groups < TP (KV heads are replicated).

## Allowed actions
- L0: tensor_model_parallel_size, sequence_parallel, pipeline_model_parallel_size, context_parallel_size (+ cp_comm_type), expert_model_parallel_size, expert_tensor_parallel_size, micro_batch_size; memory levers from manage-memory-lifetimes and recompute-policy.
- Order: make it fit with the least time cost (layout and selective savings before full recompute and offload), then remove savings while headroom allows.

## Failure signals
- OOM at update 2 although update 1 passed (optimizer state is allocated lazily after the first step).
- Allocator retries with tiny free memory: the layout is at the edge even if it completes.
- TP increase that cuts memory but raises StepTime more than the recompute it allowed you to drop.

## Verification
- Fit: a full screen with 0 OOMs, 0 retries preferred, >= 4 GiB headroom on the tightest rank.
- Speed: whole-update StepTime vs parent beyond the margin; check that GA/DP in the feedback match your intent.

## Success case (real)
- NeMo, 4 nodes, GBS 64, MoE: EP32 and EP8 without activation checkpointing both OOMed at step 2. With full activation checkpointing, EP32 reached 22,217.02 tok/s at 44.98 GiB while EP8 reached 7,491.88 tok/s at 73.11 GiB: more expert parallelism cut both memory and time. (reports/winh8_20260925/REPORT.md, completed experiments)
- Fine-grained variant: the QKV all-gather of a narrow projection (width 1,664 gathered over TP8 to 13,312, then sliced) was replaced by two fixed size-4 subgroups inside TP8, while weights, DDP, optimizer and SP stayed TP8: 7.390488 -> 7.317272 s (+1.00%), exact. (MANUSCRIPT.md 4.5, Table 6)

## Failure case (real)
- flash-next MCore on 32 GPUs with expert-DP 1: the shipped config OOMed at step 2 in the MTP forward. Each rank owned 1,568 expert weights and the Muon FP32 master+momentum (~28.7 GiB) stayed resident; EDP1 also doubled per-rank expert optimizer work compared with EDP2 on 64 GPUs. It ran only with lossless state offload, at 20.572 s/step. Lesson: count expert optimizer state per EDP shard before shrinking EDP. (reports/flash_next_mfu_20261003/PROGRESS.md 1; EVIDENCE_PRIOR64.md)

© InfraRSI A0 (human-authored) · Apache-2.0 · TokenRhythm/InfraRSI

Similar skills

All skills for Finance and business →

From the same set19

The whole set →
validate-training-lifecycle The validate-training-lifecycle Claude Code skill checks a model-training run from checkpoint save through fresh-process resume, HF export, and forward comparison. CX Codex +1 ★ 99repo stars optimize-training-kernels The optimize-training-kernels Claude Code skill adjusts fusion settings and attention backends while checking numerical tolerances. It is triggered by slow kernels, proposed kernel changes, or numerical drift. CX Codex +1 ★ 99repo stars recompute-policy The recompute-policy Claude Code skill helps choose full, selective, or no recomputation based on memory headroom and model structure. It activates for activation-checkpointing changes, OOMs, or recompute_modules decisions. CX Codex +1 ★ 99repo stars moe-ep-communication The moe-ep-communication Claude Code skill configures expert parallelism and token dispatch for MoE in Megatron-Core 0.18.2. It activates for EP sizing, dispatcher choices, and profiling bottlenecks. CX Codex +1 ★ 99repo stars manage-memory-lifetimes The manage-memory-lifetimes Claude Code skill analyzes GPU memory failures and helps select residency, offload, and allocator settings. It activates for OOMs, allocator retries, or tight memory headroom. CX Codex +1 ★ 99repo stars discover-training-system The discover-training-system Claude Code skill inspects a training setup before performance changes. It runs at task start and when a setting appears to have no effect. CX Codex +1 ★ 99repo stars verify-training-semantics The verify-training-semantics Claude Code skill checks training changes against fixed invariants and numerical limits. It runs before run_screen, when reviewing invalid receipts, and when changes touch training behavior. CX Codex +1 ★ 99repo stars infrarsi-optimize The infrarsi-optimize Claude Code skill tunes an existing GPU training job while keeping its training task unchanged. It tests launch settings and code edits to reduce step time or improve throughput. CX Codex +1 ★ 99repo stars

Project sponsors

Companies that keep this analytics open