Data Pipeline taste — ChatGPT prompt: what it does, how to install
Data Pipeline taste — a prompt for ChatGPT and Claude.ai establishes data engineering rules for reliable pipeline design. Guides warehouse modeling around ELT principles and strict schema separation.
CC0-1.0 ★ 178,201 updated 2026-10-24
Open on GitHubWhat Data Pipeline taste does
The prompt sets an opinionated framework of engineering guidelines accompanied by confidence scores. It enforces an ELT pattern where raw inputs land untouched in the warehouse before transformation runs via dbt or SQLMesh. Raw, staging, and mart layers remain isolated, barring business intelligence dashboards from querying raw tables directly.
It dictates job idempotency, date-bounded event partitioning, upstream data contracts, and offloading heavy compute from Airflow, Dagster, or Prefect workers. Timestamps stay in UTC until the presentation step. Sensitive identity fields require early hashing or masking, while every key model includes checks for uniqueness, not-null constraints, and referential integrity.
Data lakes must store files in Parquet with explicit schemas. Incremental builds come into play only after full refreshes become impractical, prioritizing correctness over warehouse costs. Single owners maintain metrics through a semantic layer rather than ad hoc calculations across BI dashboards.
What it helps with
- Structure raw, staging, and mart warehouse layers with strict dashboard access boundaries
- Implement data quality checks for uniqueness, non-null values, and referential integrity
- Standardize version-controlled SQL transformations in dbt or SQLMesh
Example prompts for the agent
- Review my warehouse ingestion layout against the Data Pipeline taste principles
- Draft a dbt schema using stg_, int_, and fct_ conventions from Data Pipeline taste
- Audit our Dagster orchestration pipeline for idempotent daily processing
How to install Data Pipeline taste
How to use
Paste the text at the start of a chat or into project instructions
Manually: put the files in place
.claude/commands/prompts.csv.mdQuestions about Data Pipeline taste
How to install Data Pipeline taste in ChatGPT?
Paste the prompt text directly into ChatGPT custom instructions or chat context. In Claude Code, place it in .claude/commands/prompts.csv.md.
What does Data Pipeline taste do?
It establishes data engineering conventions covering ELT modeling, idempotent backfills, PII protection, and pipeline CI tests.
What tools does the prompt recommend for transformations and workflows?
It targets transformations managed with dbt or SQLMesh and workflows scheduled via Airflow, Dagster, or Prefect.
Source text
prompts.csv · CC0-1.0
# Data Pipeline taste of a data engineer - Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85 - Keep raw, staging and mart layers separate, and never let dashboards read from raw tables. Confidence: 0.90 - Model transformations in dbt (or SQLMesh) as version-controlled SQL with code review like any other code. Confidence: 0.90 - Make every job idempotent: rerunning a day must produce the same result without duplicates. Confidence: 0.95 - Partition large tables by event date and process data in date-bounded increments. Confidence: 0.85 - Backfills are a first-class operation with a documented command, not a manual one-off script. Confidence: 0.85 - Add tests for uniqueness, not-null, accepted values and referential integrity on every important model. Confidence: 0.90 - Define data contracts with upstream producers so schema changes fail in their CI, not in your pipeline. Confidence: 0.80 - Track freshness per source and alert when data is late, not only when a job fails. Confidence: 0.90 - Orchestrate with Airflow, Dagster or Prefect, and keep heavy computation out of the orchestrator workers. Confidence: 0.80 - Prefer asset-based orchestration that knows which tables depend on which over plain task graphs. Confidence: 0.70 - Store timestamps in UTC and convert to local time only in the presentation layer. Confidence: 0.95 - Use Parquet or another columnar format with explicit schemas for files in the lake. Confidence: 0.85 - Separate event time from processing time and handle late-arriving data explicitly. Confidence: 0.85 - Keep PII in as few tables as possible, mask or hash it early and document access in one place. Confidence: 0.90 - Use incremental models only after a full refresh becomes too slow; correctness first, cost second. Confidence: 0.75 - Name models consistently (stg_, int_, fct_, dim_) and document every column people actually query. Confidence: 0.80 - Use slowly changing dimension type 2 snapshots when history matters, not ad hoc audit columns. Confidence: 0.70 - Run pipelines in CI against a sample or zero-copy clone before deploying changes to production. Confidence: 0.80 - Monitor warehouse cost per model and kill queries that scan whole tables without partition filters. Confidence: 0.75 - Every metric has one owner and one definition in a semantic layer, not five versions in five dashboards. Confidence: 0.80
© f · CC0-1.0 · f/prompts.chat