# data-processing-ray-data

A Claude skill that tells the agent how to process large datasets with Ray Data: streaming, GPU transforms, batch inference.

- skill · `type=skill` · Claude Code · 134 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-processing-ray-data
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/data-processing-ray-data

## What it does

The skill covers Ray Data, a distributed processing library for ML workloads. The agent gets working patterns for reading Parquet, CSV, JSON and images from cloud storage, transforming with `map_batches` and `map`, filtering, grouping, and writing results back. Streaming execution handles datasets larger than memory, and `num_gpus=1` on `map_batches` turns on GPU preprocessing.

Inside is a code-heavy guide: quick start, Ray Train integration, PyTorch and TensorFlow conversion through `to_torch` and `to_tf`, parallelism control with `repartition`. A separate section walks through batch inference — the model loads once per worker, predictions land in Parquet.

You need `ray[data]`, `pyarrow` and `pandas` installed, otherwise the examples will not run. The author points to Ray Data for datasets above 100GB and cluster-wide preprocessing, and suggests Pandas for tables under 1GB on one machine, Dask or Spark for SQL-style work. Worth checking whether your pipeline actually needs to leave a single box.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/data-processing-ray-data
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill data-processing-ray-data -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/data-processing-ray-data/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-processing-ray-data/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/data-processing-ray-data/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/data-processing-ray-data/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates data-processing-ray-data
```

2. **Manually: put the files in place**
- File: `.github/skills/data-processing-ray-data/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/data-processing-ray-data
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-processing-ray-data/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-processing-ray-data

## When to use it
- process a dataset too large for one machine's memory
- run batch inference over millions of records on several GPUs
- build a preprocessing pipeline from cleaning through tokenization
- load images and other multi-modal data for training
- feed a dataset straight into Ray Train, PyTorch or TensorFlow

## Example prompts
- read Parquet from s3://bucket/train/ and run batch inference on 4 GPUs
- build a pipeline: clean, tokenize, write results to Parquet
- convert the dataset to PyTorch with to_torch for training

## Similar
- [cli-batches](https://affpapa.org/en/nevibecode/skills/cli-batches.md) — skill · 964 installs
- [ohlcv-processing](https://affpapa.org/en/nevibecode/skills/ohlcv-processing.md) — skill · 472 installs
- [mongodb-atlas-stream-processing](https://affpapa.org/en/nevibecode/skills/mongodb-atlas-stream-processing.md) — skill · 3600 installs
- [video-processing-editing](https://affpapa.org/en/nevibecode/skills/video-processing-editing.md) — skill · 1400 installs
- [x-ray-a-company](https://affpapa.org/en/nevibecode/skills/x-ray-a-company.md) — skill · 2100 installs
- [ray-so-code-snippet](https://affpapa.org/en/nevibecode/skills/ray-so-code-snippet.md) — skill · 518 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 205 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md