# distributed-training-deepspeed

Guides distributed training setups with DeepSpeed, ZeRO memory offloading, and DeepNVMe I/O. Built for debugging ML pipelines and mixed-precision runs.

- skill · `type=skill` · Claude Code · 102 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-deepspeed
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/distributed-training-deepspeed

## What it does

This skill provides structured assistance for Microsoft DeepSpeed workflows. It covers ZeRO memory partitioning stages, pipeline parallelism, sparse attention, and mixed-precision formats including FP16, BF16, and FP8, along with 1-bit Adam optimization.

A central section explains the DeepNVMe subsystem. It explains how to instantiate `aio_handle` and `gds_handle` using builder classes, execute blocking writes, and coordinate non-blocking disk transfers via `.wait()` synchronization.

Using these workflows requires DeepSpeed 0.15.0 or newer alongside PyTorch, Accelerate, and Transformers. Without the host `libaio-dev` package, the core `async_io` operator will fail to build, which reveals itself as an error in `ds_report`.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/distributed-training-deepspeed
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill distributed-training-deepspeed -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/distributed-training-deepspeed/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/distributed-training-deepspeed/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/distributed-training-deepspeed/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/distributed-training-deepspeed/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates distributed-training-deepspeed
```

2. **Manually: put the files in place**
- File: `.github/skills/distributed-training-deepspeed/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/distributed-training-deepspeed
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/distributed-training-deepspeed/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-deepspeed

## When to use it
- configure ZeRO optimization stages to reduce GPU memory footprint
- implement non-blocking tensor I/O with DeepNVMe handles
- diagnose missing C++ kernel extensions through ds_report
- set up FP8, BF16, or FP16 mixed precision for large runs

## Example prompts
- Show me how to save host and CUDA tensors using aio_handle in DeepNVMe
- How do I resolve missing libaio-dev warnings shown by ds_report?
- Configure ZeRO Stage 3 optimization for my PyTorch training script

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 804 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md