Skip to main content
skill CX Codex

ingesting-data — Claude Code skill: what it does, how to install

ingesting-data is a Claude Code skill with design patterns and code recipes for importing data from cloud storage, APIs, files, and streams into databases.

MIT 1 installs ★ 533 updated 2025-12-11

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code
Open on GitHub

What ingesting-data does

This skill provides ingestion architectures and practical snippets for loading external data into target stores. It includes a decision tree covering diverse sources: AWS S3, Google Cloud Storage, Azure Blobs, local CSV or Parquet files, webhook feeds, and Kafka streams.

Inside, you will find code examples across four languages: Python with dlt and polars, TypeScript with Hono and AWS SDK, Rust with Polars, and concurrent Go. It also includes helper scripts like validate_csv_schema.py and modular guides under references/.

For files exceeding 100MB, the skill requires chunked processing and checksum validation. Without dead-letter queues and offset commits in place, streaming ingestion risks losing unhandled records during outages.

What it helps with

  • build an ETL pipeline from S3 to Postgres using Python and dlt
  • stream and parse large Parquet datasets efficiently with Polars
  • set up webhook ingestion endpoints with signature validation
  • design Change Data Capture workflows using Debezium and WAL logs

Example prompts for the agent

  • Show me an S3 CSV ingestion pattern in TypeScript using AWS SDK
  • How do I build an incremental ETL pipeline from a REST API using Python dlt?
  • Help me set up batch file ingestion with schema validation for files over 100MB

How to install ingesting-data

1Skills CLI (Vercel)

npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code

2Manually: put the files in place

~/.claude/skills/ingesting-data/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/ingesting-data/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about ingesting-data

How do I install ingesting-data in Claude Code?

Run npx skills add ancoleman/ai-design-components --skill ingesting-data -a claude-code in your terminal. This creates the skill file at ~/.claude/skills/ingesting-data/SKILL.md.

What sources can ingesting-data handle?

It covers cloud buckets (S3, GCS, Azure), flat files (CSV, JSON, Parquet, Excel), REST and GraphQL endpoints, CDC logs, and event streams like Kafka.

What languages and libraries are included?

It contains snippets for Python (dlt, polars), TypeScript (@aws-sdk/client-s3, hono), Rust (polars), and Go (aws-sdk-go-v2).

Source text

SKILL.md · MIT
---
name: ingesting-data
description: Data ingestion patterns for loading data from cloud storage, APIs, files, and streaming sources into databases. Use when importing CSV/JSON/Parquet files, pulling from S3/GCS buckets, consuming API feeds, or building ETL pipelines.
---

# Data Ingestion Patterns

This skill provides patterns for getting data INTO systems from external sources.

## When to Use This Skill

- Importing CSV, JSON, Parquet, or Excel files
- Loading data from S3, GCS, or Azure Blob storage
- Consuming REST/GraphQL API feeds
- Building ETL/ELT pipelines
- Database migration and CDC (Change Data Capture)
- Streaming data ingestion from Kafka/Kinesis

## Ingestion Pattern Decision Tree

```
What is your data source?
├── Cloud Storage (S3, GCS, Azure) → See cloud-storage.md
├── Files (CSV, JSON, Parquet) → See file-formats.md
├── REST/GraphQL APIs → See api-feeds.md
├── Streaming (Kafka, Kinesis) → See streaming-sources.md
├── Legacy Database → See database-migration.md
└── Need full ETL framework → See etl-tools.md
```

## Quick Start by Language

### Python (Recommended for ETL)

**dlt (data load tool) - Modern Python ETL:**
```python
import dlt

# Define a source
@dlt.source
def github_source(repo: str):
    @dlt.resource(write_disposition="merge", primary_key="id")
    def issues():
        response = requests.get(f"https://api.github.com/repos/{repo}/issues")
        yield response.json()
    return issues

# Load to destination
pipeline = dlt.pipeline(
    pipeline_name="github_issues",
    destination="postgres",  # or duckdb, bigquery, snowflake
    dataset_name="github_data"
)

load_info = pipeline.run(github_source("owner/repo"))
print(load_info)
```

**Polars for file processing (faster than pandas):**
```python
import polars as pl

# Read CSV with schema inference
df = pl.read_csv("data.csv")

# Read Parquet (columnar, efficient)
df = pl.read_parquet("s3://bucket/data.parquet")

# Read JSON lines
df = pl.read_ndjson("events.jsonl")

# Write to database
df.write_database(
    table_name="events",
    connection="postgresql://user:pass@localhost/db",
    if_table_exists="append"
)
```

### TypeScript/Node.js

**S3 ingestion:**
```typescript
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { parse } from "csv-parse/sync";

const s3 = new S3Client({ region: "us-east-1" });

async function ingestFromS3(bucket: string, key: string) {
  const response = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const body = await response.Body?.transformToString();

  // Parse CSV
  const records = parse(body, { columns: true, skip_empty_lines: true });

  // Insert to database
  await db.insert(eventsTable).values(records);
}
```

**API feed polling:**
```typescript
import { Hono } from "hono";

// Webhook receiver for real-time ingestion
const app = new Hono();

app.post("/webhooks/stripe", async (c) => {
  const event = await c.req.json();

  // Validate webhook signature
  const signature = c.req.header("stripe-signature");
  // ... validation logic

  // Ingest event
  await db.insert(stripeEventsTable).values({
    eventId: event.id,
    type: event.type,
    data: event.data,
    receivedAt: new Date()
  });

  return c.json({ received: true });
});
```

### Rust

**High-performance file ingestion:**
```rust
use polars::prelude::*;
use aws_sdk_s3::Client;

async fn ingest_parquet(client: &Client, bucket: &str, key: &str) -> Result<DataFrame> {
    // Download from S3
    let resp = client.get_object()
        .bucket(bucket)
        .key(key)
        .send()
        .await?;

    let bytes = resp.body.collect().await?.into_bytes();

    // Parse with Polars
    let df = ParquetReader::new(Cursor::new(bytes))
        .finish()?;

    Ok(df)
}
```

### Go

**Concurrent file processing:**
```go
package main

import (
    "context"
    "encoding/csv"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func ingestCSV(ctx context.Context, client *s3.Client, bucket, key string) error {
    resp, err := client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: &bucket,
        Key:    &key,
    })
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    reader := csv.NewReader(resp.Body)
    records, err := reader.ReadAll()
    if err != nil {
        return err
    }

    // Batch insert to database
    return batchInsert(ctx, records)
}
```

## Ingestion Patterns

### 1. Batch Ingestion (Files/Storage)

For periodic bulk loads:

```
Source → Extract → Transform → Load → Validate
  ↓         ↓          ↓         ↓        ↓
 S3      Download   Clean/Map  Insert   Count check
```

**Key considerations:**
- Use chunked reading for large files (>100MB)
- Implement idempotency with checksums
- Track file processing state
- Handle partial failures

### 2. Streaming Ingestion (Real-time)

For continuous data flow:

```
Source → Buffer → Process → Load → Ack
  ↓        ↓         ↓        ↓      ↓
Kafka   In-memory  Transform  DB   Commit offset
```

**Key considerations:**
- At-least-once vs exactly-once semantics
- Backpressure handling
- Dead letter queues for failures
- Checkpoint management

### 3. API Polling (Feeds)

For external API data:

```
Schedule → Fetch → Dedupe → Load → Update cursor
   ↓         ↓        ↓       ↓         ↓
 Cron     API call  By ID   Insert   Last timestamp
```

**Key considerations:**
- Rate limiting and backoff
- Incremental loading (cursors, timestamps)
- API pagination handling
- Retry with exponential backoff

### 4. Change Data Capture (CDC)

For database replication:

```
Source DB → Capture changes → Transform → Target DB
    ↓             ↓               ↓            ↓
 Postgres    Debezium/WAL      Map schema   Insert/Update
```

**Key considerations:**
- Initial snapshot + streaming changes
- Schema evolution handling
- Ordering guarantees
- Conflict resolution

## Library Recommendations

| Use Case | Python | TypeScript | Rust | Go |
|----------|--------|------------|------|-----|
| **ETL Framework** | dlt, Meltano, Dagster | - | - | - |
| **Cloud Storage** | boto3, gcsfs, adlfs | @aws-sdk/*, @google-cloud/* | aws-sdk-s3, object_store | aws-sdk-go-v2 |
| **File Processing** | polars, pandas, pyarrow | papaparse, xlsx, parquetjs | polars-rs, arrow-rs | encoding/csv, parquet-go |
| **Streaming** | confluent-kafka, aiokafka | kafkajs | rdkafka-rs | franz-go, sarama |
| **CDC** | Debezium, pg_logical | - | - | - |

## Reference Documentation

- `references/cloud-storage.md` - S3, GCS, Azure Blob patterns
- `references/file-formats.md` - CSV, JSON, Parquet, Excel handling
- `references/api-feeds.md` - REST polling, webhooks, GraphQL subscriptions
- `references/streaming-sources.md` - Kafka, Kinesis, Pub/Sub
- `references/database-migration.md` - Schema migration, CDC patterns
- `references/etl-tools.md` - dlt, Meltano, Airbyte, Fivetran

## Scripts

- `scripts/validate_csv_schema.py` - Validate CSV against expected schema
- `scripts/test_s3_connection.py` - Test S3 bucket connectivity
- `scripts/generate_dlt_pipeline.py` - Generate dlt pipeline scaffold

## Chaining with Database Skills

After ingestion, chain to appropriate database skill:

| Destination | Chain to Skill |
|-------------|----------------|
| PostgreSQL, MySQL | `databases-relational` |
| MongoDB, DynamoDB | `databases-document` |
| Qdrant, Pinecone | `databases-vector` (after embedding) |
| ClickHouse, TimescaleDB | `databases-timeseries` |
| Neo4j | `databases-graph` |

For vector databases, chain through `ai-data-engineering` for embedding:
```
ingesting-data → ai-data-engineering → databases-vector
```

© ancoleman · MIT · ancoleman/ai-design-components

Similar skills

All skills for Data and databases →

From the same set10

The whole set →
building-ai-chat building-ai-chat guides the design of AI chat interfaces with streamed replies, context controls, and image, file, or voice input. Use it when building an assistant or coding copilot. CX Codex +1 1installs architecting-networks architecting-networks is a Claude Code skill for choosing cloud network topologies: flat, multi-VPC, hub-and-spoke, or full mesh. It supports VPC, segmentation, and hybrid connectivity planning. CX Codex +1 1installs implementing-navigation implementing-navigation implements frontend navigation patterns and Python backend routing with built-in WCAG 2.1 AA accessibility guidelines. CX Codex +1 1installs security-hardening The security-hardening skill tightens OS, container, cloud, network, and database settings. Use it before production deployment, audits, or vulnerability review. CX Codex +1 1installs implementing-observability The implementing-observability skill wires metrics, logs and traces through OpenTelemetry and shows how to run Loki, Grafana, Tempo and Mimir yourself. Useful once one request crosses several services. CX Codex +1 1installs configuring-firewalls The configuring-firewalls skill guides firewall configuration across Linux hosts, cloud security controls, and Kubernetes. Use it when opening a service or narrowing network access. CX Codex +1 1installs building-ci-pipelines building-ci-pipelines — a skill for constructing secure CI/CD pipelines with SLSA provenance, monorepo optimization, caching strategies, and test parallelization. Works with GitHub Actions, GitLab CI, and Argo Workflows. CX Codex +1 1installs implementing-realtime-sync implementing-realtime-sync guides the selection and implementation of real-time patterns: progressive LLM streaming, bidirectional chats, offline sync, and multi-user editing with CRDTs. CX Codex +1 1installs

Project sponsors

Companies that keep this analytics open