# harvard-artifacts-data-pipeline

A skill that walks your agent from Harvard Art Museums API extraction to a Streamlit dashboard backed by SQL analytics.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1800 installs · ★ 5
- Author: reason-machines
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-artifacts-data-pipeline/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/harvard-artifacts-data-pipeline

## What it does

The skill covers the whole route: pulling objects from the Harvard Art Museums API with pagination and pauses between calls, reshaping nested JSON into three tables, loading them into MySQL or TiDB Cloud, and rendering charts in Streamlit with Plotly.

The schema has three tables. `artifactmetadata` holds `objectid`, `title`, `culture`, `century`, `division`, `classification`, `dated`, `accessionyear`, `peoplecount` and `totalpageviews`. `artifactmedia` carries `imagecount`, `videocount`, `hasimage` and `primaryimageurl`. `artifactcolors` stores `color`, `spectrum`, `hue` and `percent`. Foreign keys tie them together on `objectid`. Code samples inside cover fetching, transformation and batch inserts with `ON DUPLICATE KEY UPDATE`, plus a set of analytical queries.

You need a Harvard Art Museums API key and a reachable database before anything runs. The skill reads `HARVARD_API_KEY`, `DB_HOST`, `DB_USER`, `DB_PASSWORD` and `DB_NAME` from the environment, so a missing key stops extraction at the first request. Dependencies come from `requirements.txt`: `streamlit`, `pandas`, `requests`, `mysql-connector-python`, `plotly`, `python-dotenv`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-data-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/harvard-artifacts-data-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/harvard-artifacts-data-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-data-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/harvard-artifacts-data-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-data-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/harvard-artifacts-data-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills harvard-artifacts-data-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/harvard-artifacts-data-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-data-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/harvard-artifacts-data-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/harvard-artifacts-data-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill harvard-artifacts-data-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/harvard-artifacts-data-pipeline/SKILL.md

## When to use it
- pull museum objects from the Harvard API with pagination
- split nested JSON into three related tables
- batch-load metadata rows into MySQL
- write SQL queries over culture, century and classification
- build a Streamlit dashboard with Plotly charts

## Example prompts
- Build the ETL pipeline: fetch 100 objects from the Harvard Art Museums API and split them across `artifactmetadata`, `artifactmedia` and `artifactcolors`
- Write a SQL query counting objects per culture in the database
- Create a Streamlit dashboard charting how objects are distributed across centuries

## Similar
- [data-pipeline](https://affpapa.org/en/nevibecode/skills/data-pipeline.md) — skill · 4900 installs
- [indicator-setup](https://affpapa.org/en/nevibecode/skills/indicator-setup.md) — skill · 551 installs
- [ETL Pipeline](https://affpapa.org/en/nevibecode/skills/etl-pipeline.md) — skill · 4800 installs
- [indicator-dashboard](https://affpapa.org/en/nevibecode/skills/indicator-dashboard.md) — skill · 570 installs
- [sql-pro](https://affpapa.org/en/nevibecode/skills/httpsgithubcomjeffallan-sql-pro.md) — skill · 6400 installs
- [database-optimizer](https://affpapa.org/en/nevibecode/skills/httpsgithubcomjeffallan-database-optimizer.md) — skill · 6000 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 74 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md