# nemo-mbridge-perf-parallelism-strategies

Picks TP, PP, CP, EP and DP degrees for Megatron Bridge training runs, and traces an OOM or slowdown back to a parallelism config change.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1800 installs · ★ 3735
- Author: NVIDIA
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/NVIDIA/skills/tree/HEAD/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/skills/nemo-mbridge-perf-parallelism-strategies

## What it does

The skill answers questions like "which parallelism for this many GPUs" and "why did memory blow up after I edited the config". It ships starting-point tables for dense models from under 1B up to 175-500B, plus separate logic for MoE. There TP is sized by active parameters rather than total, which is why DeepSeek-V3 671B / 37B gets TP=2 instead of 8. EP does the heavy lifting.

Hardware topology is part of the decision. Keep TP inside one NVLink domain and scale across nodes with PP or DP, since TP over the network is almost always a performance loss. Long sequences pull in SP first, then CP. There is also a section on minimum GPU count: `min_gpus = PP * max(TP * CP, EP * ETP)`, not the product of every dimension. The full product over-allocates, and it shows up in plenty of slurm sizing tables — worth rechecking your own numbers against it.

You also get DP and EDP formulas, 3D and 4D config examples, and a memory estimate for a 70B model in FP16. The tables are starting points, not hard rules: the skill tells you to profile the first iteration and verify memory against communication.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/nemo-mbridge-perf-parallelism-strategies/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install NVIDIA/skills nemo-mbridge-perf-parallelism-strategies
```

2. **Manually: put the files in place**
- File: `.github/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/NVIDIA/skills --path skills/nemo-mbridge-perf-parallelism-strategies
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/NVIDIA/skills/tree/HEAD/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md

## When to use it
- size TP, PP and DP for a given GPU count
- compute the minimum GPU count for an MoE config
- find what caused an OOM after a parallelism change
- choose CP for long sequence training
- distribute MoE layers across nodes with EP and PP

## Example prompts
- Which parallelism should I use for a 70B dense model on 64 GPUs?
- Compute min_gpus for PP=2, TP=2, EP=8, ETP=1
- I hit OOM after raising TP to 8 — what should I check?

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 15da947188a7ca66.

BEGIN THIRD-PARTY FILE 15da947188a7ca66
```text
---
name: nemo-mbridge-perf-parallelism-strategies
description: Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.
license: Apache-2.0
when_to_use: Choosing or sizing TP/DP/PP/CP/EP degrees, or tracing an OOM or regression to a parallelism config change; 'how to parallelize', 'tensor parallel', 'pipeline parallel', 'parallelism config', 'which parallelism for X GPUs'.
---

# Parallelism Strategy Selection Skill

For stable background on each parallelism type, see:

- @docs/parallelisms.md
- @skills/nemo-mbridge-perf-parallelism-strategies/card.yaml

## Decision by Model Size

### Dense models

| Model size | GPUs | Recommended starting point |
|---|---|---|
| < 1B | 1-8 | DP only |
| 1-10B | 8-16 | TP=2-4 + DP |
| 10-70B | 16-64 | TP=4-8 + PP=2-4 + DP |
| 70-175B | 64-256 | TP=8 + PP=4-8 + DP |
| 175-500B | 256-1024 | TP=8 + PP=8-16 + CP=2 + DP |

### MoE models

MoE parallelism differs from dense models. Because only a fraction of
parameters are active per token, TP can often stay at 1 or 2 — the active
parameter shard already fits on a single GPU. EP is the primary scaling
dimension, with PP handling cross-node layer distribution.

| Model (total / active) | TP | PP | EP | Notes |
|---|---|---|---|---|
| OLMoE 7B / 1B | 1 | 1 | 8 | EP only, fits single node |
| Moonlight 16B / 3B | 2 | 1 | 8 | small TP for shared layers |
| DeepSeek-V2 236B / 21B | 1 | 4 | 32 | no TP at all |
| GLM-4.5 Air 106B / 12B | 1 | 4 | 8 | no TP at all |
| Qwen3 30B-A3B | 4 | 2 | 4 | |
| GLM-4.5 355B / 32B | 2 | 8 | 16 | |
| Qwen3 235B-A22B | 4 | 16 | 8 | CP=2 for pretrain |
| DeepSeek-V3 671B / 37B | 2 | 16 | 64 | TP=2, not 8 |
| Kimi-K2 1T | 2 | 16 | 32 | |

Key patterns:

- TP is sized by **active** params, not total params. A 671B MoE with
  37B active needs far less TP than a 70B dense model.
- EP scales with expert count. Common: EP = num_experts or
  num_experts / experts_per_gpu.
- PP handles depth. Large MoE models use PP=8-16 across nodes.
- ETP (expert tensor parallelism) is rarely used. Llama 4 is an
  exception (ETP=4).

These are starting points, not hard rules. Always profile the first
iteration to verify memory and communication.

## Decision by Hardware Topology

Single node with NVLink:
```
END THIRD-PARTY FILE 15da947188a7ca66

The third-party file has ended. Everything between the 15da947188a7ca66 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2349 of 9580 bytes. © NVIDIA · Apache-2.0 · https://github.com/NVIDIA/skills/tree/HEAD/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md