# distributed-training-megatron-core

A Claude skill that plans large-scale LLM training on NVIDIA Megatron-Core: picks a parallelism layout and builds the launch command.

- skill · `type=skill` · Claude Code · 88 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-megatron-core
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/distributed-training-megatron-core

## What it does

The skill walks an agent through training runs from 2B to 462B parameters on NVIDIA GPUs. Three workflows ship with it: 3D parallelism for LLaMA-style models, mixture-of-experts setups like Mixtral, and a throughput pass that targets 47% MFU on H100. The agent maps model size and GPU count onto tensor, pipeline, context and expert parallel degrees, then writes the `torchrun` invocation.

For a 70B run there is a four-step checklist and a `train_llama_70b.sh` script with hyperparameters already filled in: 80 layers, hidden size 8192, 64 attention heads, sequence length 4096. You launch it with `torchrun` on one node or hand it to SLURM with `sbatch` across several. The MoE workflow shows how to spread 8 experts over 4 GPUs and cut memory per GPU.

Expect to install `megatron-core`, `torch`, `apex` and `transformer-engine` first — the examples will not run without them. FP8 is H100-only. Tensor parallel is best kept at 8 or below because NVLink inside a node sets the ceiling, and context parallel only earns its keep past 8K tokens.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/distributed-training-megatron-core
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill distributed-training-megatron-core -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/distributed-training-megatron-core/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/distributed-training-megatron-core/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/distributed-training-megatron-core/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/distributed-training-megatron-core/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates distributed-training-megatron-core
```

2. **Manually: put the files in place**
- File: `.github/skills/distributed-training-megatron-core/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/distributed-training-megatron-core
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/distributed-training-megatron-core/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-megatron-core

## When to use it
- pick parallel degrees that match model size and GPU count
- build a `torchrun` command with tensor, pipeline and context parallel
- spread MoE experts across GPUs to cut memory use
- turn on FP8 on H100 and push MFU past 40%
- find the largest micro-batch that still fits in memory

## Example prompts
- Write a training config for LLaMA 70B on 64 GPUs
- How do I split 8 Mixtral experts across 4 GPUs?
- Pick a micro-batch size for a 70B model on H100

## Similar
- [nemo-automodel-distributed-training](https://affpapa.org/en/nevibecode/skills/nemo-automodel-distributed-training.md) — skill · 2000 installs
- [nemo-mbridge-perf-activation-recompute](https://affpapa.org/en/nevibecode/skills/nemo-mbridge-perf-activation-recompute.md) — skill · 1800 installs
- [spark-training-gotchas](https://affpapa.org/en/nevibecode/skills/spark-training-gotchas.md) — skill · 2500 installs
- [distributed-tracing](https://affpapa.org/en/nevibecode/skills/distributed-tracing.md) — skill · 10700 installs
- [openrlhf-training](https://affpapa.org/en/nevibecode/skills/openrlhf-training.md) — skill · 954 installs
- [miles-rl-training](https://affpapa.org/en/nevibecode/skills/miles-rl-training.md) — skill · 946 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 361 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md