# harvard-artifacts-etl-pipeline

A skill that walks your coding agent through an ETL pipeline from the Harvard Art Museums API to Streamlit charts.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1700 installs · ★ 5
- Author: reason-machines
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-artifacts-etl-pipeline/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/harvard-artifacts-etl-pipeline

## What it does

The skill covers the whole path: paged requests to the Harvard Art Museums API, flattening nested JSON into tables, loading into MySQL or TiDB Cloud, analytical SQL, and Plotly charts in Streamlit. Your agent gets working code templates for each stage.

Inside are Python examples. `fetch_artifacts` pulls pages of 100 records and stops once the pages run out; `extract_metadata`, `extract_media` and `extract_colors` split the API response into three tables. The schema ships too: `artifactmetadata`, `artifactmedia` and `artifactcolors`, joined by `artifact_id`.

You need a `HARVARD_API_KEY` and a database — host, port, user and password go into `.env`. Without the API key every request fails, so check that first. Dependencies come from `requirements.txt`: `streamlit`, `pandas`, `requests`, `mysql-connector-python`, `plotly`, `python-dotenv`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-etl-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/harvard-artifacts-etl-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/harvard-artifacts-etl-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-etl-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/harvard-artifacts-etl-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-etl-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/harvard-artifacts-etl-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills harvard-artifacts-etl-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/harvard-artifacts-etl-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-etl-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/harvard-artifacts-etl-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/harvard-artifacts-etl-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-etl-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-artifacts-etl-pipeline/SKILL.md

## When to use it
- pull museum objects from the Harvard Art Museums API page by page
- turn nested museum JSON into relational tables
- load artifact metadata and colors into MySQL or TiDB Cloud
- write SQL queries over the collection and its media
- build an interactive Streamlit dashboard with Plotly charts

## Example prompts
- Help me write an ETL pipeline that fetches objects from the Harvard Art Museums API and splits them into tables
- Write SQL queries against the artifactmetadata table and chart the results with Plotly in Streamlit
- Set up paged fetching from the Harvard API and load the data into TiDB Cloud

## Similar
- [ETL Pipeline](https://affpapa.org/en/nevibecode/skills/etl-pipeline.md) — skill · 4800 installs
- [etl-connector-strategy](https://affpapa.org/en/nevibecode/skills/etl-connector-strategy.md) — skill · 8000 installs
- [pyspark-etl-best-practices-cursorrules-prompt-file](https://affpapa.org/en/nevibecode/rules/pyspark-etl-best-practices-cursorrules-prompt-file.md) — rule · ★ 41011
- [data-pipeline](https://affpapa.org/en/nevibecode/skills/data-pipeline.md) — skill · 4900 installs
- [checking-freshness](https://affpapa.org/en/nevibecode/skills/checking-freshness.md) — skill · 1000 installs
- [streamlit-to-marimo](https://affpapa.org/en/nevibecode/skills/streamlit-to-marimo.md) — skill · 2300 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 22 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md