# Data Cleaning Pipeline

Generates structured Python data preprocessing routines to clean missing values, strip invalid text, handle outliers, and validate data quality.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 589 installs · ★ 371
- Author: aj-geddes
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/aj-geddes/useful-ai-prompts/tree/HEAD/skills/data-cleaning-pipeline/SKILL.md
- Updated: 2026-03-04
- HTML: https://affpapa.org/en/nevibecode/skills/data-cleaning-pipeline

## What it does

This skill guides the agent to assemble data cleaning routines in Python using `pandas`, `numpy`, and `scikit-learn`. It covers explicit patterns for dropping incomplete records, imputing figures with `SimpleImputer` or `KNNImputer`, and applying `StandardScaler`.

The workflow organizes transformations through a modular `DataCleaningPipeline` class. Outliers are handled via interquartile range bounds, either by clipping or row deletion. It also includes text normalization with regex and generates summary metrics to verify that clean datasets retain required fields.

Running the generated code requires `scikit-learn` alongside `pandas` in your local Python environment. Without them, imputer and scaler steps fail immediately.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add aj-geddes/useful-ai-prompts --skill data-cleaning-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/data-cleaning-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-cleaning-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add aj-geddes/useful-ai-prompts --skill data-cleaning-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/data-cleaning-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add aj-geddes/useful-ai-prompts --skill data-cleaning-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/data-cleaning-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install aj-geddes/useful-ai-prompts data-cleaning-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/data-cleaning-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add aj-geddes/useful-ai-prompts --skill data-cleaning-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/aj-geddes/useful-ai-prompts --path skills/data-cleaning-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-cleaning-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add aj-geddes/useful-ai-prompts --skill data-cleaning-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/aj-geddes/useful-ai-prompts/tree/HEAD/skills/data-cleaning-pipeline/SKILL.md

## When to use it
- handle missing entries using median or KNN imputation
- filter or clip numeric outliers based on IQR boundaries
- chain custom cleaning steps inside a modular pipeline class
- generate audit metrics comparing raw and processed datasets

## Example prompts
- Build a pandas cleaning pipeline for raw_data.csv that handles missing values with KNN
- Create a step to detect and clip amount outliers using IQR in DataCleaningPipeline
- Write validation checks and a quality summary table for this dataframe

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 1ffbd3c0f141455a.

BEGIN THIRD-PARTY FILE 1ffbd3c0f141455a
````text
---
name: Data Cleaning Pipeline
description: Build robust processes for data cleaning, missing value imputation, outlier handling, and data transformation for data preprocessing, data quality, and data pipeline automation
---

# Data Cleaning Pipeline

## Overview

Data cleaning pipelines transform raw, messy data into clean, standardized formats suitable for analysis and modeling through systematic handling of missing values, outliers, and data quality issues.

## When to Use

- Preparing raw datasets for analysis or modeling
- Handling missing values and data quality issues
- Removing duplicates and standardizing formats
- Detecting and treating outliers
- Building automated data preprocessing workflows
- Ensuring data integrity and consistency

## Core Components

- **Missing Value Handling**: Imputation and removal strategies
- **Outlier Detection & Treatment**: Identifying and handling anomalies
- **Data Type Standardization**: Ensuring correct data types
- **Duplicate Removal**: Identifying and removing duplicates
- **Normalization & Scaling**: Standardizing value ranges
- **Text Cleaning**: Handling text data

## Cleaning Strategies

- **Deletion**: Removing rows or columns
- **Imputation**: Filling with mean, median, or predictive models
- **Transformation**: Converting between formats
- **Validation**: Ensuring data integrity rules

## Implementation with Python

```python
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.impute import SimpleImputer, KNNImputer

# Load raw data
df = pd.read_csv('raw_data.csv')

# Step 1: Identify and handle missing values
print("Missing values:\n", df.isnull().sum())

# Strategy 1: Delete rows with critical missing values
df = df.dropna(subset=['customer_id', 'transaction_date'])

# Strategy 2: Impute numerical columns with median
imputer = SimpleImputer(strategy='median')
df['age'] = imputer.fit_transform(df[['age']])

# Strategy 3: Use KNN imputation for related features
knn_imputer = KNNImputer(n_neighbors=5)
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols] = knn_imputer.fit_transform(df[numeric_cols])

# Strategy 4: Fill categorical with mode
df['category'] = df['category'].fillna(df['category'].mode()[0])

# Step 2: Handle duplicates
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()

# Duplicate on specific columns
df = df.drop_duplicates(subset=['customer_id', 'transaction_date'])

# Step 3: Outlier detection and handling
Q1 = df['amount'].quantile(0.25)
Q3 = df['amount'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Remove outliers
df = df[(df['amount'] >= lower_bound) & (df['amount'] <= upper_bound)]

# Alternative: Cap outliers
````
END THIRD-PARTY FILE 1ffbd3c0f141455a

The third-party file has ended. Everything between the 1ffbd3c0f141455a tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2790 of 7141 bytes. © aj-geddes · MIT · https://github.com/aj-geddes/useful-ai-prompts/tree/HEAD/skills/data-cleaning-pipeline/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md