ingesting-data — Claude Code skill: what it does, how to install
ingesting-data is a Claude Code skill with design patterns and code recipes for importing data from cloud storage, APIs, files, and streams into databases.
MIT 1 installs ★ 533 updated 2025-12-11
npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-codeWhat ingesting-data does
This skill provides ingestion architectures and practical snippets for loading external data into target stores. It includes a decision tree covering diverse sources: AWS S3, Google Cloud Storage, Azure Blobs, local CSV or Parquet files, webhook feeds, and Kafka streams.
Inside, you will find code examples across four languages: Python with dlt and polars, TypeScript with Hono and AWS SDK, Rust with Polars, and concurrent Go. It also includes helper scripts like validate_csv_schema.py and modular guides under references/.
For files exceeding 100MB, the skill requires chunked processing and checksum validation. Without dead-letter queues and offset commits in place, streaming ingestion risks losing unhandled records during outages.
What it helps with
- build an ETL pipeline from S3 to Postgres using Python and dlt
- stream and parse large Parquet datasets efficiently with Polars
- set up webhook ingestion endpoints with signature validation
- design Change Data Capture workflows using Debezium and WAL logs
Example prompts for the agent
- Show me an S3 CSV ingestion pattern in TypeScript using AWS SDK
- How do I build an incremental ETL pipeline from a REST API using Python dlt?
- Help me set up batch file ingestion with schema validation for files over 100MB
How to install ingesting-data
1Skills CLI (Vercel)
npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code2Manually: put the files in place
~/.claude/skills/ingesting-data/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/ingesting-data/
1Skills CLI (Vercel)
npx skills add ancoleman/ai-design-components --skill ingesting-data -a codex2Manually: put the files in place
.agents/skills/ingesting-data/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add ancoleman/ai-design-components --skill ingesting-data -a cursor2Manually: put the files in place
.cursor/skills/ingesting-data/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install ancoleman/ai-design-components skills/ingesting-data2Manually: put the files in place
.github/skills/ingesting-data/SKILL.md1Skills CLI (Vercel)
npx skills add ancoleman/ai-design-components --skill ingesting-data -a gemini-cli2Platform CLI
gemini skills install https://github.com/ancoleman/ai-design-components --path skills/ingesting-dataManually: put the files in place
.devin/skills/ingesting-data/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add ancoleman/ai-design-components --skill ingesting-data -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about ingesting-data
How do I install ingesting-data in Claude Code?
Run npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code in your terminal. This creates the skill file at ~/.claude/skills/ingesting-data/SKILL.md.
What sources can ingesting-data handle?
It covers cloud buckets (S3, GCS, Azure), flat files (CSV, JSON, Parquet, Excel), REST and GraphQL endpoints, CDC logs, and event streams like Kafka.
What languages and libraries are included?
It contains snippets for Python (dlt, polars), TypeScript (@aws-sdk/client-s3, hono), Rust (polars), and Go (aws-sdk-go-v2).
Source text
SKILL.md · MIT
---
name: ingesting-data
description: Data ingestion patterns for loading data from cloud storage, APIs, files, and streaming sources into databases. Use when importing CSV/JSON/Parquet files, pulling from S3/GCS buckets, consuming API feeds, or building ETL pipelines.
---
# Data Ingestion Patterns
This skill provides patterns for getting data INTO systems from external sources.
## When to Use This Skill
- Importing CSV, JSON, Parquet, or Excel files
- Loading data from S3, GCS, or Azure Blob storage
- Consuming REST/GraphQL API feeds
- Building ETL/ELT pipelines
- Database migration and CDC (Change Data Capture)
- Streaming data ingestion from Kafka/Kinesis
## Ingestion Pattern Decision Tree
```
What is your data source?
├── Cloud Storage (S3, GCS, Azure) → See cloud-storage.md
├── Files (CSV, JSON, Parquet) → See file-formats.md
├── REST/GraphQL APIs → See api-feeds.md
├── Streaming (Kafka, Kinesis) → See streaming-sources.md
├── Legacy Database → See database-migration.md
└── Need full ETL framework → See etl-tools.md
```
## Quick Start by Language
### Python (Recommended for ETL)
**dlt (data load tool) - Modern Python ETL:**
```python
import dlt
# Define a source
@dlt.source
def github_source(repo: str):
@dlt.resource(write_disposition="merge", primary_key="id")
def issues():
response = requests.get(f"https://api.github.com/repos/{repo}/issues")
yield response.json()
return issues
# Load to destination
pipeline = dlt.pipeline(
pipeline_name="github_issues",
destination="postgres", # or duckdb, bigquery, snowflake
dataset_name="github_data"
)
load_info = pipeline.run(github_source("owner/repo"))
print(load_info)
```
**Polars for file processing (faster than pandas):**
```python
import polars as pl
# Read CSV with schema inference
df = pl.read_csv("data.csv")
# Read Parquet (columnar, efficient)
df = pl.read_parquet("s3://bucket/data.parquet")
# Read JSON lines
df = pl.read_ndjson("events.jsonl")
# Write to database
df.write_database(
table_name="events",
connection="postgresql://user:pass@localhost/db",
if_table_exists="append"
)
```
### TypeScript/Node.js
**S3 ingestion:**
```typescript
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { parse } from "csv-parse/sync";
const s3 = new S3Client({ region: "us-east-1" });
async function ingestFromS3(bucket: string, key: string) {
const response = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
const body = await response.Body?.transformToString();
// Parse CSV
const records = parse(body, { columns: true, skip_empty_lines: true });
// Insert to database
await db.insert(eventsTable).values(records);
}
```
**API feed polling:**
```typescript
import { Hono } from "hono";
// Webhook receiver for real-time ingestion
const app = new Hono();
app.post("/webhooks/stripe", async (c) => {
const event = await c.req.json();
// Validate webhook signature
const signature = c.req.header("stripe-signature");
// ... validation logic
// Ingest event
await db.insert(stripeEventsTable).values({
eventId: event.id,
type: event.type,
data: event.data,
receivedAt: new Date()
});
return c.json({ received: true });
});
```
### Rust
**High-performance file ingestion:**
```rust
use polars::prelude::*;
use aws_sdk_s3::Client;
async fn ingest_parquet(client: &Client, bucket: &str, key: &str) -> Result<DataFrame> {
// Download from S3
let resp = client.get_object()
.bucket(bucket)
.key(key)
.send()
.await?;
let bytes = resp.body.collect().await?.into_bytes();
// Parse with Polars
let df = ParquetReader::new(Cursor::new(bytes))
.finish()?;
Ok(df)
}
```
### Go
**Concurrent file processing:**
```go
package main
import (
"context"
"encoding/csv"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func ingestCSV(ctx context.Context, client *s3.Client, bucket, key string) error {
resp, err := client.GetObject(ctx, &s3.GetObjectInput{
Bucket: &bucket,
Key: &key,
})
if err != nil {
return err
}
defer resp.Body.Close()
reader := csv.NewReader(resp.Body)
records, err := reader.ReadAll()
if err != nil {
return err
}
// Batch insert to database
return batchInsert(ctx, records)
}
```
## Ingestion Patterns
### 1. Batch Ingestion (Files/Storage)
For periodic bulk loads:
```
Source → Extract → Transform → Load → Validate
↓ ↓ ↓ ↓ ↓
S3 Download Clean/Map Insert Count check
```
**Key considerations:**
- Use chunked reading for large files (>100MB)
- Implement idempotency with checksums
- Track file processing state
- Handle partial failures
### 2. Streaming Ingestion (Real-time)
For continuous data flow:
```
Source → Buffer → Process → Load → Ack
↓ ↓ ↓ ↓ ↓
Kafka In-memory Transform DB Commit offset
```
**Key considerations:**
- At-least-once vs exactly-once semantics
- Backpressure handling
- Dead letter queues for failures
- Checkpoint management
### 3. API Polling (Feeds)
For external API data:
```
Schedule → Fetch → Dedupe → Load → Update cursor
↓ ↓ ↓ ↓ ↓
Cron API call By ID Insert Last timestamp
```
**Key considerations:**
- Rate limiting and backoff
- Incremental loading (cursors, timestamps)
- API pagination handling
- Retry with exponential backoff
### 4. Change Data Capture (CDC)
For database replication:
```
Source DB → Capture changes → Transform → Target DB
↓ ↓ ↓ ↓
Postgres Debezium/WAL Map schema Insert/Update
```
**Key considerations:**
- Initial snapshot + streaming changes
- Schema evolution handling
- Ordering guarantees
- Conflict resolution
## Library Recommendations
| Use Case | Python | TypeScript | Rust | Go |
|----------|--------|------------|------|-----|
| **ETL Framework** | dlt, Meltano, Dagster | - | - | - |
| **Cloud Storage** | boto3, gcsfs, adlfs | @aws-sdk/*, @google-cloud/* | aws-sdk-s3, object_store | aws-sdk-go-v2 |
| **File Processing** | polars, pandas, pyarrow | papaparse, xlsx, parquetjs | polars-rs, arrow-rs | encoding/csv, parquet-go |
| **Streaming** | confluent-kafka, aiokafka | kafkajs | rdkafka-rs | franz-go, sarama |
| **CDC** | Debezium, pg_logical | - | - | - |
## Reference Documentation
- `references/cloud-storage.md` - S3, GCS, Azure Blob patterns
- `references/file-formats.md` - CSV, JSON, Parquet, Excel handling
- `references/api-feeds.md` - REST polling, webhooks, GraphQL subscriptions
- `references/streaming-sources.md` - Kafka, Kinesis, Pub/Sub
- `references/database-migration.md` - Schema migration, CDC patterns
- `references/etl-tools.md` - dlt, Meltano, Airbyte, Fivetran
## Scripts
- `scripts/validate_csv_schema.py` - Validate CSV against expected schema
- `scripts/test_s3_connection.py` - Test S3 bucket connectivity
- `scripts/generate_dlt_pipeline.py` - Generate dlt pipeline scaffold
## Chaining with Database Skills
After ingestion, chain to appropriate database skill:
| Destination | Chain to Skill |
|-------------|----------------|
| PostgreSQL, MySQL | `databases-relational` |
| MongoDB, DynamoDB | `databases-document` |
| Qdrant, Pinecone | `databases-vector` (after embedding) |
| ClickHouse, TimescaleDB | `databases-timeseries` |
| Neo4j | `databases-graph` |
For vector databases, chain through `ai-data-engineering` for embedding:
```
ingesting-data → ai-data-engineering → databases-vector
```
© ancoleman · MIT · ancoleman/ai-design-components