Skip to main content
skill CX Codex

schema-e2e-validation — Claude Code skill for schema checks

The schema-e2e-validation skill runs YAML schema generation and Earthly E2E checks. Use it before committing schema changes or to compare a contract with ClickHouse Cloud.

MIT 123 installs ★ 3 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add terrylica/cc-skills --skill schema-e2e-validation -a claude-code
Open on GitHub

What schema-e2e-validation does

The skill runs schema generation for Python types, DDL, and documentation, or a full E2E workflow. Generation uses earthly +test-schema-generate; the E2E command is ./scripts/earthly-with-doppler.sh +test-schema-e2e. Generated files go to ./earthly-artifacts/, with E2E outputs in e2e/types/, e2e/ddl/, and e2e/docs/.

Earthly needs a Docker Runtime, so start Colima with colima start first. The +test-schema-validate and +test-schema-e2e targets need Doppler access. The wrapper fetches ClickHouse secrets and passes them to Earthly. If Colima is not running, Earthly cannot reach Docker.

The workflow covers generation, validation against ClickHouse, and E2E execution. For schema design, the skill directs users to call quality-tools:clickhouse-architect first; schema design itself is outside this skill’s role.

What it helps with

  • check YAML schema changes before committing
  • compare a schema contract with ClickHouse Cloud
  • generate Python types, DDL, and documentation
  • run the full schema E2E workflow

Example prompts for the agent

  • Run the full E2E check for my YAML schema.
  • Generate Python types, DDL, and docs from the schema.
  • Validate the schema against ClickHouse Cloud before I commit.

How to install schema-e2e-validation

1Skills CLI (Vercel)

npx skills add terrylica/cc-skills --skill schema-e2e-validation -a claude-code

2Manually: put the files in place

~/.claude/skills/schema-e2e-validation/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/schema-e2e-validation/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about schema-e2e-validation

How do I install schema-e2e-validation in Claude Code?

Run npx skills add terrylica/cc-skills --skill schema-e2e-validation -a claude-code. The skill file is available at ~/.claude/skills/schema-e2e-validation/SKILL.md.

What does schema-e2e-validation do?

It runs Earthly targets to generate Python types, DDL, and documentation, and to validate YAML schemas against ClickHouse Cloud.

What is required for schema E2E validation?

Earthly requires Docker Runtime, with Colima started beforehand. The +test-schema-validate and +test-schema-e2e targets also need Doppler access.

Source text

SKILL.md · MIT
---
name: schema-e2e-validation
description: Earthly E2E validation for YAML schema contracts. TRIGGERS - schema validation, YAML schema, schema contracts, regenerate types.
allowed-tools: Read, Bash, Grep
---

# Schema E2E Validation

> **Self-Evolving Skill**: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

## When to Use This Skill

Use this skill when:

- Validating schema changes before commit
- Verifying YAML schema matches live ClickHouse Cloud
- Regenerating Python types, DDL, or docs
- Running full schema workflow validation

## Prerequisites

### Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

```bash
colima start
```

**Check if running:**

```bash
docker ps  # Should not error
```

### Doppler Access (For validation targets)

Required for `+test-schema-validate` and `+test-schema-e2e`:

```bash
doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd
```

### Earthly Installation

```bash
brew install earthly
```

---

## Quick Commands

### Generation only (no secrets)

```bash
cd ~/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate
```

### Full E2E with validation (requires Doppler)

```bash
cd ~/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e
```

### All non-secret targets

```bash
cd ~/eon/gapless-network-data
earthly +all
```

---

## Artifacts

After running `+test-schema-generate` or `+test-schema-e2e`, check `./earthly-artifacts/`:

| Path                       | Contents                    |
| -------------------------- | --------------------------- |
| `types/blocks.py`          | Pydantic + TypedDict models |
| `types/__init__.py`        | Package init                |
| `ddl/ethereum_mainnet.sql` | ClickHouse DDL              |
| `docs/ethereum_mainnet.md` | Markdown documentation      |

For E2E, artifacts are under `e2e/types/`, `e2e/ddl/`, `e2e/docs/`.

---

## Earthfile Targets Reference

| Target                  | Secrets | Purpose                    |
| ----------------------- | ------- | -------------------------- |
| `+deps`                 | No      | Install uv + dependencies  |
| `+build`                | No      | Copy source files          |
| `+test-unit`            | No      | Run pytest                 |
| `+test-schema-generate` | No      | Generate types/DDL/docs    |
| `+test-schema-validate` | Yes     | Validate vs ClickHouse     |
| `+test-schema-e2e`      | Yes     | Full workflow + artifacts  |
| `+all`                  | No      | Run all non-secret targets |

---

## Troubleshooting

### "could not determine buildkit address - is Docker or Podman running?"

**Cause**: Docker/Colima not running

**Fix**:

```bash
colima start
# Wait for "done" message, then retry
earthly +test-schema-generate
```

### "unable to parse --secret-file argument"

**Cause**: Wrong flag name or malformed secrets file

**Fix**: The correct flag is `--secret-file-path` (NOT `--secret-file`). The wrapper script handles this, but if running manually:

```bash
# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target
```

Also ensure secrets file has no quotes around values:

```bash
# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud
```

### "OSError: Readme file does not exist: README.md"

**Cause**: hatchling build backend requires README.md in container

**Fix**: Ensure Earthfile copies README.md in deps target:

```earthfile
deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!
```

### "missing secret" during validation

**Cause**: Doppler not configured or secrets not passed

**Fix**:

```bash
# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate
```

### Cache Issues

Force rebuild without cache:

```bash
earthly --no-cache +test-schema-e2e
```

---

## Implementation Details

### Doppler Secret Injection

The wrapper script `scripts/earthly-with-doppler.sh`:

1. Downloads secrets from Doppler
2. Filters for `CLICKHOUSE_*` variables
3. Strips quotes (Doppler outputs `KEY="value"`, Earthly needs `KEY=value`)
4. Passes via `--secret-file-path` flag
5. Cleans up temp file on exit

### Secrets Required

| Secret                         | Purpose               |
| ------------------------------ | --------------------- |
| `CLICKHOUSE_HOST_READONLY`     | ClickHouse Cloud host |
| `CLICKHOUSE_USER_READONLY`     | Read-only user        |
| `CLICKHOUSE_PASSWORD_READONLY` | Read-only password    |

---

## Related Files

| File                                                                   | Purpose                  |
| ---------------------------------------------------------------------- | ------------------------ |
| `~/eon/gapless-network-data/Earthfile`                                 | Main build file          |
| `~/eon/gapless-network-data/scripts/earthly-with-doppler.sh`           | Secret injection wrapper |
| `~/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml`   | SSoT schema              |
| `~/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md` | ADR                      |

---

## Validation History

- **2025-12-03**: Created and validated with full E2E run against ClickHouse Cloud
- **Lessons Learned**:
  - `--secret-file-path` not `--secret-file` (Earthly v0.8.16)
  - Doppler `--format env` outputs quotes, must strip with `sed 's/"//g'`
  - README.md must be copied for hatchling build backend
  - Colima must be started before Earthly runs

---

## Design Authority

<!-- ADR: 2025-12-10-clickhouse-skill-delegation -->

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke **`quality-tools:clickhouse-architect`** first.

## Related Skills

| Skill                                      | Purpose                         |
| ------------------------------------------ | ------------------------------- |
| `quality-tools:clickhouse-architect`       | Schema design before validation |
| `devops-tools:clickhouse-cloud-management` | Cloud credentials for E2E tests |
| `devops-tools:clickhouse-pydantic-config`  | Client configuration            |


## Post-Execution Reflection

After this skill completes, check before closing:

1. **Did the command succeed?** — If not, fix the instruction or error table that caused the failure.
2. **Did parameters or output change?** — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. **Was a workaround needed?** — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

© terrylica · MIT · terrylica/cc-skills

Similar skills

All skills for Testing and QA →
data360-schema-get forcedotcom data360-schema-get — a skill for Claude Code and other AI editors that retrieves schema definitions for Data Lake Objects (DLO) and Data Model Objects (DMO) from Salesforce Data Cloud via REST API. CX Codex +1 5,100installs dms-schema-conversion aws This Claude Code skill manages schema migrations between different database engines with AWS DMS Schema Conversion. It handles project setup, compatibility checks, and schema or SQL conversion. CX Codex +1 5,500installs planetscale-schema-recommendations-agent-loop planetscale planetscale-schema-recommendations-agent-loop is a Claude Code skill that processes PlanetScale schema recommendations and turns them into plans for migrations, branches, and pull requests without touching production. CX Codex +1 2,000installs schema coreyhaines31 This Claude Code skill helps create and validate Schema.org markup in JSON-LD. It assesses page types, existing markup, and Google search feature support. CX Codex +1 91,700installs convex-schema-validator waynesutton convex-schema-validator is a Claude Code skill that creates convex/schema.ts with types, indexes, and validators. It runs when creating tables, adding fields, modeling relationships, or fixing validation errors. CX Codex +1 3,100installs zod-validation-utilities giuseppe-trisciuoglio zod-validation-utilities — a skill for building reusable Zod v4 schemas to validate API payloads, forms, and configuration input with full TypeScript type inference. CX Codex +1 4,100installs

From the same set79

The whole set →
issues-workflow issues-workflow puts plan, hierarchy and status into GitHub Issues: sub-issues under a parent, an issue linked to its branch and pull request, and a close on merge. CX Codex +1 124installs ml-failfast-validation ml-failfast-validation is a Claude Code skill that runs preflight checks for machine learning experiments, catching tensor, schema, and gradient errors before long training jobs. CX Codex +1 124installs pr-gfm-validator pr-gfm-validator turns relative markdown links in a pull request body into absolute blob URLs pinned to your feature branch, so reviewers stop landing on 404s. CX Codex +1 123installs pypi-doppler pypi-doppler — a skill for Claude Code and five other platforms that publishes Python packages to PyPI using a token stored in Doppler — local machine only. The bundled script detects CI environments and exits immediately if one is found. CX Codex +1 123installs plugin-validator plugin-validator — a skill that validates plugin structure before release and catches silent failures in hooks—error-handling gaps in shell commands and unhandled Python exceptions. Essential before publishing to the marketplace. CX Codex +1 122installs evolutionary-metric-ranking evolutionary-metric-ranking is a Claude Code skill that ranks and filters model and strategy configurations across multiple metrics using percentile cutoffs, intersection rules, and evolutionary search. CX Codex +1 121installs pre-ship-review The pre-ship-review skill runs structured quality checks before a pull request, release, or milestone. It catches integration failures where contracts, examples, and tests must align across files. CX Codex +1 120installs pueue-job-orchestration pueue-job-orchestration manages long-running tasks and batch processing using pueue queue orchestration, parallelism groups, and detailed CLI logs. CX Codex +1 120installs

Project sponsors

Companies that keep this analytics open