# google-cloud-data-engineering-hub

A skill that hands your coding agent a reference set of working GCP data projects: BigQuery, Dataflow, Beam, Composer, Pub/Sub, Dataproc, Vertex AI.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1100 installs · ★ 5
- Author: reason-machines
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/google-cloud-data-engineering-hub/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/google-cloud-data-engineering-hub

## What it does

This skill points the agent at a collection of runnable data engineering projects for Google Cloud. Each one ships as modular Python rather than loose scripts, with ASCII architecture diagrams, sample fixtures and a `deploy.sh` that automates the GCP setup. The examples were run against live cloud projects.

Coverage is grouped by service. BigQuery entries handle CSV ingestion, the UPSERT/MERGE pattern, and training plus prediction with BigQuery ML. Storage projects cover file operations and v4 signed URLs with lifecycle rules. Further out you get Dataflow and Apache Beam jobs, Cloud Composer workflows, Dataproc Serverless, Pub/Sub streaming, and pipelines wired to Gemini AI and Vertex AI.

Before you start: Python 3.10 or newer, the `gcloud` CLI, a GCP project with billing turned on, and the right IAM permissions. No billing-enabled project means nothing runs, so check that first. Dependencies are installed per project from its own `requirements.txt`, and credentials come from `gcloud auth application-default login`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill google-cloud-data-engineering-hub -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/google-cloud-data-engineering-hub/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/google-cloud-data-engineering-hub/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill google-cloud-data-engineering-hub -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/google-cloud-data-engineering-hub/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill google-cloud-data-engineering-hub -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/google-cloud-data-engineering-hub/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills google-cloud-data-engineering-hub
```

2. **Manually: put the files in place**
- File: `.github/skills/google-cloud-data-engineering-hub/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill google-cloud-data-engineering-hub -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/google-cloud-data-engineering-hub
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/google-cloud-data-engineering-hub/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill google-cloud-data-engineering-hub -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/google-cloud-data-engineering-hub/SKILL.md

## When to use it
- load a CSV file into BigQuery
- write a MERGE statement for upserts
- train and query a BigQuery ML model
- stream records through Pub/Sub
- run a workflow in Cloud Composer

## Example prompts
- Build a BigQuery pipeline that loads a CSV file
- Show me how to upsert rows in BigQuery with MERGE
- Set up Pub/Sub streaming into Dataflow

## Similar
- [caveman-manage](https://affpapa.org/en/nevibecode/skills/caveman-manage.md) — skill · 104200 installs
- [workers-best-practices](https://affpapa.org/en/nevibecode/skills/cloudflare-workers-best-practices.md) — skill · 126100 installs
- [cloudflare-email-service](https://affpapa.org/en/nevibecode/skills/cloudflare-email-service.md) — skill · 102300 installs
- [firebase-basics](https://affpapa.org/en/nevibecode/skills/firebase-basics.md) — skill · 14100 installs
- [cloudflare-one](https://affpapa.org/en/nevibecode/skills/cloudflare-one.md) — skill · 93100 installs
- [cloudflare-one-migrations](https://affpapa.org/en/nevibecode/skills/cloudflare-one-migrations.md) — skill · 89500 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 31 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md