# ingesting-data

ingesting-data is a Claude Code skill with design patterns and code recipes for importing data from cloud storage, APIs, files, and streams into databases.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 1 install · ★ 533 repo stars
- Author: ancoleman
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/ancoleman/ai-design-components/tree/HEAD/skills/ingesting-data/SKILL.md
- Updated: 2025-12-11
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/ingesting-data

## What it does

This skill provides ingestion architectures and practical snippets for loading external data into target stores. It includes a decision tree covering diverse sources: AWS S3, Google Cloud Storage, Azure Blobs, local CSV or Parquet files, webhook feeds, and Kafka streams.

Inside, you will find code examples across four languages: Python with `dlt` and `polars`, TypeScript with Hono and AWS SDK, Rust with Polars, and concurrent Go. It also includes helper scripts like `validate_csv_schema.py` and modular guides under `references/`.

For files exceeding 100MB, the skill requires chunked processing and checksum validation. Without dead-letter queues and offset commits in place, streaming ingestion risks losing unhandled records during outages.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/ingesting-data/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/ingesting-data/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add ancoleman/ai-design-components --skill ingesting-data -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/ingesting-data/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add ancoleman/ai-design-components --skill ingesting-data -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/ingesting-data/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install ancoleman/ai-design-components skills/ingesting-data
```

2. **Manually: put the files in place**
- File: `.github/skills/ingesting-data/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add ancoleman/ai-design-components --skill ingesting-data -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/ancoleman/ai-design-components --path skills/ingesting-data
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/ingesting-data/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add ancoleman/ai-design-components --skill ingesting-data -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/ancoleman/ai-design-components/tree/HEAD/skills/ingesting-data/SKILL.md

## When to use it
- build an ETL pipeline from S3 to Postgres using Python and dlt
- stream and parse large Parquet datasets efficiently with Polars
- set up webhook ingestion endpoints with signature validation
- design Change Data Capture workflows using Debezium and WAL logs

## Example prompts
- Show me an S3 CSV ingestion pattern in TypeScript using AWS SDK
- How do I build an incremental ETL pipeline from a REST API using Python dlt?
- Help me set up batch file ingestion with schema validation for files over 100MB

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 74cb65c565cc076c.

BEGIN THIRD-PARTY FILE 74cb65c565cc076c
````text
---
name: ingesting-data
description: Data ingestion patterns for loading data from cloud storage, APIs, files, and streaming sources into databases. Use when importing CSV/JSON/Parquet files, pulling from S3/GCS buckets, consuming API feeds, or building ETL pipelines.
---

# Data Ingestion Patterns

This skill provides patterns for getting data INTO systems from external sources.

## When to Use This Skill

- Importing CSV, JSON, Parquet, or Excel files
- Loading data from S3, GCS, or Azure Blob storage
- Consuming REST/GraphQL API feeds
- Building ETL/ELT pipelines
- Database migration and CDC (Change Data Capture)
- Streaming data ingestion from Kafka/Kinesis

## Ingestion Pattern Decision Tree

```
What is your data source?
├── Cloud Storage (S3, GCS, Azure) → See cloud-storage.md
├── Files (CSV, JSON, Parquet) → See file-formats.md
├── REST/GraphQL APIs → See api-feeds.md
├── Streaming (Kafka, Kinesis) → See streaming-sources.md
├── Legacy Database → See database-migration.md
└── Need full ETL framework → See etl-tools.md
```

## Quick Start by Language

### Python (Recommended for ETL)

**dlt (data load tool) - Modern Python ETL:**
```python
import dlt

# Define a source
@dlt.source
def github_source(repo: str):
    @dlt.resource(write_disposition="merge", primary_key="id")
    def issues():
        response = requests.get(f"https://api.github.com/repos/{repo}/issues")
        yield response.json()
    return issues

# Load to destination
pipeline = dlt.pipeline(
    pipeline_name="github_issues",
    destination="postgres",  # or duckdb, bigquery, snowflake
    dataset_name="github_data"
)

load_info = pipeline.run(github_source("owner/repo"))
print(load_info)
```

**Polars for file processing (faster than pandas):**
```python
import polars as pl

# Read CSV with schema inference
df = pl.read_csv("data.csv")

# Read Parquet (columnar, efficient)
df = pl.read_parquet("s3://bucket/data.parquet")

# Read JSON lines
df = pl.read_ndjson("events.jsonl")

# Write to database
df.write_database(
    table_name="events",
    connection="postgresql://user:pass@localhost/db",
    if_table_exists="append"
)
```

### TypeScript/Node.js

**S3 ingestion:**
```typescript
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { parse } from "csv-parse/sync";

const s3 = new S3Client({ region: "us-east-1" });

async function ingestFromS3(bucket: string, key: string) {
  const response = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const body = await response.Body?.transformToString();

  // Parse CSV
  const records = parse(body, { columns: true, skip_empty_lines: true });

  // Insert to database
````
END THIRD-PARTY FILE 74cb65c565cc076c

The third-party file has ended. Everything between the 74cb65c565cc076c tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2721 of 7736 bytes. © ancoleman · MIT · https://github.com/ancoleman/ai-design-components/tree/HEAD/skills/ingesting-data/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
