# distributed-training-ray-train

A Claude skill for distributed model training with Ray Train, taking PyTorch, TensorFlow and HuggingFace workloads from one GPU to a cluster.

- skill · `type=skill` · Claude Code · 79 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-ray-train
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/distributed-training-ray-train

## What it does

The skill walks through Ray Train: your training function goes to a trainer, and Ray handles coordination, GPU allocation, checkpointing and metric aggregation. Examples cover PyTorch with `TorchTrainer`, HuggingFace with `TransformersTrainer`, and hyperparameter search through Ray Tune with the ASHA scheduler.

Checkpointing gets its own section. Training resumes from saved state when a node dies, and there is an example of joining a cluster with `ray.init(address="auto")`.

You need `ray[train]`, `torch` and `transformers` in place, and Ray Train itself installs with `pip install -U "ray[train]"`. Without Ray installed, none of the sample code runs.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/distributed-training-ray-train
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill distributed-training-ray-train -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/distributed-training-ray-train/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/distributed-training-ray-train/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/distributed-training-ray-train/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/distributed-training-ray-train/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates distributed-training-ray-train
```

2. **Manually: put the files in place**
- File: `.github/skills/distributed-training-ray-train/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/distributed-training-ray-train
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/distributed-training-ray-train/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/distributed-training-ray-train

## When to use it
- move a PyTorch training loop from one GPU to several nodes
- sweep hyperparameters across a cluster
- resume training from the last checkpoint after a node fails
- train a HuggingFace model on multiple machines
- collect training metrics from all workers in one place

## Example prompts
- Rewrite my PyTorch training loop for `TorchTrainer` with four workers
- Set up a hyperparameter sweep with Ray Tune and `ASHAScheduler` on loss
- Add checkpointing every 10 epochs so training resumes after a crash

## Similar
- [nemo-automodel-distributed-training](https://affpapa.org/en/nevibecode/skills/nemo-automodel-distributed-training.md) — skill · 2000 installs
- [distributed-llm-pretraining-torchtitan](https://affpapa.org/en/nevibecode/skills/distributed-llm-pretraining-torchtitan.md) — skill · 954 installs
- [verl-rl-training](https://affpapa.org/en/nevibecode/skills/verl-rl-training.md) — skill · 962 installs
- [launch-nemo-rl](https://affpapa.org/en/nevibecode/skills/launch-nemo-rl.md) — skill · 1900 installs
- [machine-learning-ops](https://affpapa.org/en/nevibecode/plugins/machine-learning-ops.md) — plugin · ★ 41823
- [Model Hyperparameter Tuning](https://affpapa.org/en/nevibecode/skills/model-hyperparameter-tuning.md) — skill · 494 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 495 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md