distributed-training-megatron-core — Claude Code skill

What distributed-training-megatron-core does

The skill walks an agent through training runs from 2B to 462B parameters on NVIDIA GPUs. Three workflows ship with it: 3D parallelism for LLaMA-style models, mixture-of-experts setups like Mixtral, and a throughput pass that targets 47% MFU on H100. The agent maps model size and GPU count onto tensor, pipeline, context and expert parallel degrees, then writes the torchrun invocation.

For a 70B run there is a four-step checklist and a train_llama_70b.sh script with hyperparameters already filled in: 80 layers, hidden size 8192, 64 attention heads, sequence length 4096. You launch it with torchrun on one node or hand it to SLURM with sbatch across several. The MoE workflow shows how to spread 8 experts over 4 GPUs and cut memory per GPU.

Expect to install megatron-core, torch, apex and transformer-engine first — the examples will not run without them. FP8 is H100-only. Tensor parallel is best kept at 8 or below because NVLink inside a node sets the ceiling, and context parallel only earns its keep past 8K tokens.