# harvard-art-museums-data-engineering-pipeline

A skill for building an ETL pipeline on Harvard Art Museums API data: JSON into normalized SQL tables, analytical queries, and a Streamlit dashboard.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1800 installs · ★ 5
- Author: reason-machines
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/harvard-art-museums-data-engineering-pipeline

## What it does

The skill walks through the whole chain, from a Harvard Art Museums API request to an interactive dashboard. Records are pulled page by page with rate limiting in mind, then the nested JSON is split into normalized tables: object metadata, images, and colors each get their own table, joined by `objectid`. The schema is created during the load step.

Code samples are Python. `requests` handles fetching, `pandas` does the reshaping, `mysql-connector-python` writes to the database, and `plotly` plus `streamlit` cover the visuals. Analytics rest on a set of ready-made SQL queries against the loaded tables.

You need a Harvard Art Museums API key and a MySQL or TiDB Cloud database before anything runs. Credentials live in `.env`: `HARVARD_API_KEY`, `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`. Leave that file empty and both the fetch and the load fail. The dashboard starts with `streamlit run app.py` and serves at `http://localhost:8501`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-art-museums-data-engineering-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/harvard-art-museums-data-engineering-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-art-museums-data-engineering-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-art-museums-data-engineering-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills harvard-art-museums-data-engineering-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-art-museums-data-engineering-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/harvard-art-museums-data-engineering-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-art-museums-data-engineering-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-art-museums-data-engineering-pipeline/SKILL.md

## When to use it
- pull objects from the Harvard Art Museums API across pages
- flatten nested JSON into normalized SQL tables
- design a relational schema with foreign keys for a collection
- write analytical SQL queries over the collected data
- put together a Streamlit dashboard with Plotly charts

## Example prompts
- Build an ETL pipeline that pulls objects from the Harvard Art Museums API into MySQL
- Set up a normalized schema for object metadata, images, and colors
- Create a Streamlit dashboard with Plotly charts for the collection data

## Similar
- [xlsx](https://affpapa.org/en/nevibecode/skills/xlsx.md) — skill · 187223 installs
- [google-agents-cli-observability](https://affpapa.org/en/nevibecode/skills/google-agents-cli-observability.md) — skill · 511300 installs
- [caveman-evidence-review](https://affpapa.org/en/nevibecode/skills/caveman-evidence-review.md) — skill · 104500 installs
- [migration](https://affpapa.org/en/nevibecode/skills/migration.md) — skill · 103900 installs
- [performance-optimization](https://affpapa.org/en/nevibecode/skills/performance-optimization.md) — skill · 65200 installs
- [supabase](https://affpapa.org/en/nevibecode/skills/supabase.md) — skill · 8 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 31 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md