sre-data-reliability — Claude Code skill for data reliability
sre-data-reliability is a Claude Code skill for reviewing data pipelines, scheduled jobs, backups, and recovery plans. It activates on SRE work involving data integrity and reliability.
Apache-2.0 ★ 67 updated 2026-10-22
npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a claude-codeWhat sre-data-reliability does
The skill helps assess failure risks in data pipelines, recurring jobs, and recovery procedures. It covers pipeline SLOs for freshness and correctness, idempotent stages, late records, and input validation. Replication copies corrupted data too, so it is not treated as a substitute for backups.
For recovery planning, it recommends setting RPO and RTO per dataset, testing restores from real backups, and supporting partial recovery. Scheduled-job guidance covers leader election, durable launch state, and monitoring missed or overrunning runs. Slow corruption can reach recent backups before anyone notices, so the material addresses separate backup domains and early validation.
What it helps with
- set freshness and correctness SLOs for a data pipeline
- make pipeline stages safe to rerun
- plan dataset recovery around RPO and RTO
- review a scheduler for missed or duplicate runs
Example prompts for the agent
- Review this data pipeline and propose freshness and correctness SLOs.
- Assess the backup and recovery plan for the user table.
- Check how this distributed scheduler handles missed and repeated runs.
How to install sre-data-reliability
1Skills CLI (Vercel)
npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a claude-code2Manually: put the files in place
~/.claude/skills/sre-data-reliability/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/sre-data-reliability/
1Skills CLI (Vercel)
npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a codex2Manually: put the files in place
.agents/skills/sre-data-reliability/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a cursor2Manually: put the files in place
.cursor/skills/sre-data-reliability/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install paulip114/sre-practice-skills sre-data-reliability2Manually: put the files in place
.github/skills/sre-data-reliability/SKILL.md1Skills CLI (Vercel)
npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a gemini-cli2Platform CLI
gemini skills install https://github.com/paulip114/sre-practice-skills --path sre-data-reliabilityManually: put the files in place
.devin/skills/sre-data-reliability/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about sre-data-reliability
How do I install sre-data-reliability in Claude Code?
Run npx skills add paulip114/sre-practice-skills --skill sre-data-reliability -a claude-code. The skill file is at ~/.claude/skills/sre-data-reliability/SKILL.md.
What does sre-data-reliability do?
It helps review data pipelines, scheduled jobs, data integrity safeguards, backups, and recovery plans using SRE practices.
What backup and recovery topics does the skill cover?
It addresses dataset RPO and RTO, restore tests, partial recovery, and keeping backups separate from the primary system. It also covers early detection of data corruption.
Source text
SKILL.md · Apache-2.0
---
name: sre-data-reliability
description: Use when designing, operating or reviewing data pipelines, scheduled/cron jobs, backups and restores, or data integrity protections — pipeline SLOs, idempotency, handling stragglers and hot spots, distributed cron, defense in depth against data loss, and recovery testing. Based on Google SRE practice.
---
# Data Pipelines, Scheduled Jobs and Data Integrity
## Data integrity: what users need is restores, not backups
Users judge you by whether their data is there and correct when they ask for it. A backup that has never been restored is a hope, not a plan.
### Threat model
Data loss comes from many combinations of:
- **Root cause**: user error, application bugs, operator/automation error, infrastructure failure, malicious action, migration bugs.
- **Scope**: wide (many entities) vs. narrow (one user/object).
- **Rate**: sudden (big bang) vs. slow and creeping (a bug corrupting a little data each day for months).
Slow, creeping corruption is the most dangerous: by the time it's noticed, it's in every recent backup and replica.
### Defense in depth
1. **First layer — soft deletion / undo**: deleted data goes to a trash/tombstone state for a retention period (days to weeks) before purge; users and support can undo. Protects against user and application errors.
2. **Second layer — backups and recovery**: versioned backups with retention long enough to catch slow corruption, stored in a separate failure domain and administrative domain (so the same bug or operator mistake can't delete both). Replication is **not** a backup — it faithfully replicates corruption and deletes.
3. **Third layer — early detection**: continuous **out-of-band validation** that checks data invariants (referential integrity, counts, checksums, business rules) and alerts on drift. Detection time is what bounds how far back you need to restore.
### Recovery
- Define **RPO** (how much data loss is tolerable) and **RTO** (how fast you must recover) per dataset, and design backups to meet them.
- **Test restores continuously** and automatically, end to end, from real backups; measure time to restore. Alert when a restore test fails.
- Support **partial/point restores** (one user, one table, one time range), not just "restore everything".
- Keep tooling and runbooks for recovery as production-grade as serving code.
## Data pipelines
### Design
- Prefer **continuous/streaming or frequent small batches** over huge periodic batches when freshness matters; big periodic jobs create load spikes ("thundering herds") and long recovery after failure.
- Make every stage **idempotent** and re-runnable; write outputs atomically (write then rename/commit).
- Define pipeline SLOs: **freshness**, **correctness**, **coverage/completeness**, and end-to-end latency (`sre-slos-and-error-budgets`).
- Handle **stragglers and hot spots**: split work into many small shards, rebalance dynamically, avoid skewed keys.
- **Schema changes and migrations**: versioned schemas, backward/forward compatible readers, dual-writes and backfills staged and verified.
- Isolate pipelines by priority so a large low-priority backfill can't starve a critical job.
- Plan dependencies: know upstream sources, their SLOs, and what happens when they are late or bad; validate inputs and quarantine bad records instead of failing whole runs or silently passing garbage.
### Operating
- Monitor: freshness/lag at each stage, throughput, backlog, error/quarantine counts, data quality checks.
- Alert on SLO (e.g. "data older than X for > Y% of reads"), not on every task failure.
- Keep runbooks for: reprocessing a time range, skipping/quarantining poison inputs, backfills, and rolling back a bad output version.
- Canary pipeline code and config changes on a subset of data and compare output against the previous version before full rollout.
## Distributed cron / scheduled jobs
- Run the scheduler itself redundantly with leader election via a consensus system, so one machine's failure doesn't skip or double-run jobs.
- Persist launch state durably: record intent **before** launching and success **after**, so a new leader can tell whether a job ran.
- Make jobs **idempotent** or have them check for prior completion; decide explicitly whether a missed run should be skipped or run late, and whether a double run is acceptable.
- **Spread start times** (jitter, hashed minutes) to avoid everyone running at :00 and overloading shared systems.
- Monitor for missed runs, overruns (job still running at next start), and failures.
## Review checklist
- [ ] RPO/RTO defined per dataset; backups meet them?
- [ ] Backups in separate failure + administrative domain; retention covers slow corruption?
- [ ] Soft-delete / undo for user-facing deletions?
- [ ] Automated, end-to-end restore tests with alerting?
- [ ] Out-of-band data validation running?
- [ ] Pipeline stages idempotent, atomic outputs, freshness/correctness SLOs?
- [ ] Scheduled jobs survive scheduler failure without skipping or double-running?
## Sources
- SRE book ch. 24 Distributed Periodic Scheduling with Cron — https://sre.google/sre-book/distributed-periodic-scheduling/
- SRE book ch. 25 Data Processing Pipelines — https://sre.google/sre-book/data-processing-pipelines/
- SRE book ch. 26 Data Integrity — https://sre.google/sre-book/data-integrity/
- Workbook ch. 13 Data Processing Pipelines — https://sre.google/workbook/data-processing/
© paulip114 · Apache-2.0 · paulip114/sre-practice-skills