Skip to main content
skill CX Codex

moe-ep-communication — Claude Code skill: MoE setup

The moe-ep-communication Claude Code skill configures expert parallelism and token dispatch for MoE in Megatron-Core 0.18.2. It activates for EP sizing, dispatcher choices, and profiling bottlenecks.

Apache-2.0 ★ 99 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add TokenRhythm/InfraRSI --skill moe-ep-communication -a claude-code
Open on GitHub

What moe-ep-communication does

The skill helps select an expert-parallel size and a token dispatcher: allgather, alltoall, or flex with DeepEP. It also covers grouped GEMM, fused permute operations, communication overlap, and numerical checks for combine. Its diagnostics use step time, profile phases, the slowest rank, per-rank memory, and NCCL errors.

A larger EP can reduce expert parameter memory, yet raise step time when all-to-all crosses nodes. DeepEP must import in the frozen virtual environment; moe_permute_fusion requires TE 2.16 to be available. After a change, the skill checks combine against the reference and verifies router invariants and NCCL errors.

What it helps with

  • choose an EP size for the expert layout across nodes
  • compare allgather, alltoall, and flex dispatch
  • trace delays caused by overloaded experts
  • check combine numerics after changing its communication path

Example prompts for the agent

  • Configure EP and token dispatch for this MoE model in Megatron-Core 0.18.2.
  • Inspect this profile: token exchange dominates step time.
  • Check combine numerics after switching to flex with DeepEP.

How to install moe-ep-communication

1Skills CLI (Vercel)

npx skills add TokenRhythm/InfraRSI --skill moe-ep-communication -a claude-code

2Manually: put the files in place

~/.claude/skills/moe-ep-communication/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/moe-ep-communication/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about moe-ep-communication

How do I install moe-ep-communication in Claude Code?

Run npx skills add TokenRhythm/InfraRSI --skill moe-ep-communication -a claude-code. The skill file is at ~/.claude/skills/moe-ep-communication/SKILL.md.

What does moe-ep-communication do?

It guides EP sizing, token dispatch, expert communication, and combine numerical checks for MoE workloads. It applies to MoE tasks and profiles dominated by dispatch, combine, or expert compute.

What does moe-ep-communication require?

Its configuration guidance targets Megatron-Core 0.18.2. Flex with DeepEP requires DeepEP to import in the frozen virtual environment; permute fusion requires TE 2.16.

Source text

SKILL.md · Apache-2.0
---
name: moe-ep-communication
description: Configure MoE expert parallelism and token dispatch in Megatron-Core 0.18.2 - EP sizing, allgather/alltoall/flex(DeepEP) dispatchers, grouped GEMM, permute fusion, communication overlap - and verify combine numerics.
license: Apache-2.0
when_to_use: Any MoE task; dispatch/combine or expert compute dominates the profile; choosing EP; DeepEP availability; router imbalance stragglers; 'alltoall', 'DeepEP', 'grouped gemm', 'EP overlap', 'expert imbalance'.
metadata:
  author: InfraRSI A0 (human-authored)
  source: human-authored (A0)
  version: 1
  tags: [moe, expert-parallel, dispatch, deepep, all-to-all, communication, overlap, multi-node, single-node]
---

# MoE expert parallelism and communication

Router top-k, capacity (dropless), aux-loss coefficient and router dtype are task constants. You choose where
experts live and how tokens move.

## Trigger conditions
- MoE model; `profile` shows fwd_bwd dominated by dispatch/combine or by expert GEMMs.
- `slowest_rank` follows expert load (hot experts).
- EP crossing nodes on multi-node tasks.

## Required observations (receipt fields)
- `step_time`, `profile` phases, `per_step[].slowest_rank`, `memory.per_rank[]`, `events.nccl_errors`, `invariants.router_topk`, `invariants.num_experts`.

## Rules: dispatcher and kernels
- allgather (MCore default) gathers all tokens on every EP*TP rank: simple, but traffic and memory grow with EP; it does not support variable sequence lengths.
- alltoall: permute + all-to-all to the owning expert rank; the standard choice for EP > 1 (the Megatron-LM MoE quick start uses alltoall + moe_grouped_gemm; FR uses it).
- flex: DeepEP kernels (NVLink + RDMA) via moe_enable_deepep or the default flex backend; DeepEP must import in the frozen venv or the screen fails; moe_deepep_num_sms must be even (default 20 SMs taken from compute).
- moe_grouped_gemm: one GroupedLinear kernel for all local experts; almost always on when a rank holds several experts. moe_permute_fusion: TE fused permute/unpermute (TE 2.16 available).
- moe_shared_expert_overlap: only with shared experts, alltoall/flex; overlaps shared-expert compute with dispatch.
- overlap_moe_expert_parallel_comm (+ delay_wgrad_compute): overlaps EP all-to-all with another micro-batch's compute; needs EP > 1, alltoall/flex, PP = 1 here, no full recompute, no 'moe' recompute, no shared-expert overlap.

## Rules: EP sizing
- num_experts % EP == 0; experts per rank = num_experts / EP. EP inside one node keeps the all-to-all on NVLink. Measured all-to-all bus bandwidth (256 MiB, NVLink included) is 44.96 GB/s versus 234.47 GB/s for all-reduce: cross-node all-to-all is the most expensive traffic here.
- Larger EP lowers expert parameter memory but shrinks EDP, raising per-rank expert optimizer state (choose-parallel-layout).
- Router imbalance is real: load CV 1.59873, hottest expert 34.62x the mean token count in one run. Hot experts make stragglers; dispatcher choice cannot change routing (frozen), but overlap and grouped GEMM reduce the exposed cost.

## Rules: numerics
- Combine accumulation order matters: an unspecified atomic order made runs diverge bitwise after step 14 in one case. FP32 accumulation of BF16 rows in a fixed order is exact and was faster. Order changes are within E1 tolerance only if the reference comparison passes; verify, do not assume.

## Allowed actions
- L0: expert_model_parallel_size, expert_tensor_parallel_size, moe_token_dispatcher_type, moe_enable_deepep, moe_deepep_num_sms, moe_grouped_gemm, moe_permute_fusion, moe_shared_expert_overlap, overlap_moe_expert_parallel_comm, delay_wgrad_compute; NCCL env for all-to-all (NCCL_NCHANNELS_PER_NET_PEER, NCCL_IB_QPS_PER_CONNECTION).
- L1: comm_groups.py (template comm_groups.nccl_group_config: high-priority streams for ep/ep_dp groups, CTA limits).

## Failure signals
- DeepEP import or RDMA errors (`events.exceptions`), NCCL errors, hangs at the first dispatch.
- Numerics drift beyond tolerance after changing dispatcher or combine path.
- More EP with lower memory but higher StepTime (all-to-all crossing nodes).

## Verification
- StepTime beyond the margin, numerics within E1 tolerance vs FR plan A, invariants (router_topk, num_experts, capacity) unchanged, 0 NCCL errors.

## Success case (real)
- Flagship: an FP32 segmented local sum for the combine (stable token sort, per-token segments, a Triton kernel reading BF16 rows in order and accumulating FP32 in registers; covers the combine forward and the dispatch-backward merge; DeepEP itself unchanged): 10.678406 -> 10.131331 s (+5.40%), exact. On flash-next the same idea took the combine from 5.05 to 0.64 ms per call (-0.77 s/step) and removed an atomic-order divergence after step 14. (MANUSCRIPT.md 4.2, Table 6; flash_next_mfu_20261003/FINAL_REPORT.md 2)
- NeMo 2K, EP64: DeepEP 68,709.58 tok/s versus the torch dispatcher 49,237.07 tok/s, no token drop. (winh8_20260925/REPORT.md)

## Failure case (real)
- Tuning DeepEP SMs: num_sms 20 -> 16 on NeMo 64 GPUs was -1.3010% with neither half-window improving, and on flash-next 40 and 24 SMs were both slower than 20. The default was kept. Taking SMs from compute or giving them back is not a free win. (MFU_PROGRESS_REPORT_20260928.md; flash_next_mfu_20261003/FINAL_REPORT.md 4)

© InfraRSI A0 (human-authored) · Apache-2.0 · TokenRhythm/InfraRSI

Similar skills

All skills for AI and ML →
nemo-mbridge-perf-expert-parallel-overlap NVIDIA nemo-mbridge-perf-expert-parallel-overlap is a Claude Code skill for configuring and checking expert-parallel communication overlap in Megatron-Bridge. CX Codex +1 1,800installs nemo-mbridge-perf-moe-optimization-workflow NVIDIA nemo-mbridge-perf-moe-optimization-workflow guides MoE throughput tuning and regression diagnosis in Megatron Bridge. Useful when diagnosing speed drops after commit or configuration changes. CX Codex +1 1,900installs flash-moe-inference reason-machines flash-moe-inference runs the Qwen3.5-397B Mixture-of-Experts model on Apple Silicon by streaming expert weights directly from an SSD in pure C and Metal. Designed for Macs with 48GB of memory. CX Codex +1 1,200installs emerging-techniques-moe-training Orchestra Research emerging-techniques-moe-training provides guidance and code templates for training Mixture of Experts models with DeepSpeed and Transformers. Scales model capacity without matching compute growth. CX Codex +1 59installs distributed-training-megatron-core Orchestra Research distributed-training-megatron-core — a Claude skill that plans large-scale LLM training on NVIDIA Megatron-Core: picks a parallelism layout and builds the launch command. CX Codex +1 88installs miles-rl-training Orchestra Research miles-rl-training — a skill that walks your agent through RL training of large MoE models on miles: FP8, INT4, speculative rollout. Useful when you're setting up training on H100/H200. CX Codex +1 1,100installs

From the same set19

The whole set →
profile-whole-update The profile-whole-update Claude Code skill measures full-update time and checks whether a change improved the end-to-end update. It comes into play when reviewing profiles, comparing candidates, or tracing delays between steps. CX Codex +1 ★ 99repo stars validate-training-lifecycle The validate-training-lifecycle Claude Code skill checks a model-training run from checkpoint save through fresh-process resume, HF export, and forward comparison. CX Codex +1 ★ 99repo stars optimize-training-kernels The optimize-training-kernels Claude Code skill adjusts fusion settings and attention backends while checking numerical tolerances. It is triggered by slow kernels, proposed kernel changes, or numerical drift. CX Codex +1 ★ 99repo stars recompute-policy The recompute-policy Claude Code skill helps choose full, selective, or no recomputation based on memory headroom and model structure. It activates for activation-checkpointing changes, OOMs, or recompute_modules decisions. CX Codex +1 ★ 99repo stars manage-memory-lifetimes The manage-memory-lifetimes Claude Code skill analyzes GPU memory failures and helps select residency, offload, and allocator settings. It activates for OOMs, allocator retries, or tight memory headroom. CX Codex +1 ★ 99repo stars discover-training-system The discover-training-system Claude Code skill inspects a training setup before performance changes. It runs at task start and when a setting appears to have no effect. CX Codex +1 ★ 99repo stars verify-training-semantics The verify-training-semantics Claude Code skill checks training changes against fixed invariants and numerical limits. It runs before run_screen, when reviewing invalid receipts, and when changes touch training behavior. CX Codex +1 ★ 99repo stars choose-parallel-layout The choose-parallel-layout Claude Code skill selects a parallel layout from per-rank memory limits and collective costs. It activates for OOM incidents, long sequences, MoE placement, or multi-node scaling. CX Codex +1 ★ 99repo stars

Project sponsors

Companies that keep this analytics open