# data-processing-nemo-curator

Instructs Claude Code on creating data preparation pipelines for neural network training using NVIDIA NeMo Curator on GPUs.

- skill · `type=skill` · Claude Code · 123 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-processing-nemo-curator
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/data-processing-nemo-curator

## What it does

The skill helps Claude Code generate curation workflows for machine learning datasets. It covers text, audio, images, and video. You get ready patterns for exact deduplication, MinHash LSH fuzzy matching, and vector comparison. NVIDIA benchmarks show GPU fuzzy deduplication running 16 times faster than CPU processing.

Pipelines rely on `rapids`, `cudf`, and `dask` to distribute execution across GPU nodes. Running these scripts requires NVIDIA hardware with CUDA 12 support. The skill guides you through masking personal data like names and phone numbers, running 30 heuristic checks, or scoring texts with DeBERTa classifiers.

Multimodal handling is built into the workflow. The generated scripts can detect video scenes, compute audio word error rates, or discard unsuitable images based on aesthetic scores.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/data-processing-nemo-curator
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill data-processing-nemo-curator -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/data-processing-nemo-curator/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-processing-nemo-curator/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/data-processing-nemo-curator/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/data-processing-nemo-curator/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates data-processing-nemo-curator
```

2. **Manually: put the files in place**
- File: `.github/skills/data-processing-nemo-curator/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/data-processing-nemo-curator
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-processing-nemo-curator/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-processing-nemo-curator

## When to use it
- remove duplicate documents using GPU MinHash algorithms
- mask personal identities, phone numbers, and emails
- filter audio recordings by speech recognition error rates
- detect scene transitions and extract video segments
- scale batch preprocessing across multiple graphics cards

## Example prompts
- Build a NeMo Curator pipeline to run fuzzy deduplication on a parquet dataset
- Add PII redaction and line repetition filters to the text processing script
- Create an image curation workflow that discards NSFW files and scores aesthetics

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 804 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md