differential-fuzzer — Claude Code skill: install and usage

---
name: differential-fuzzer
description: Information about the differential fuzzer tool, how to run it and use it catch bugs in Turso. Always load this skill when running this tool
---

# Differential Fuzzer

Always load [Debugging skill for reference](../debugging/)

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

## Location

`testing/differential-oracle/fuzzer/`

## Running the Fuzzer

### Single Run

```bash
# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk
```

### Continuous Fuzzing (Loop Mode)

```bash
# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50
```

### Docker Runner (CI/Production)

```bash
# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer
```

Environment variables for docker-runner:
- `TIME_LIMIT_MINUTES` - Total runtime (default: 1440 = 24h)
- `PER_RUN_TIMEOUT_SECONDS` - Per-run timeout (default: 1200 = 20min)
- `NUM_STATEMENTS` - Statements per run (default: 1000)
- `LOG_TO_STDOUT` - Print fuzzer output (default: false)
- `GITHUB_TOKEN` - For auto-filing issues
- `SLACK_WEBHOOK_URL` - For notifications

## Output Files

All output goes to `simulator-output/` directory:

| File | Description |
|------|-------------|
| `test.sql` | All executed SQL statements. Failed statements prefixed with `-- FAILED:`, errors with `-- ERROR:` |
| `schema.json` | Database schema at end of run (or at failure) |
| `test.db` | Turso database file (only with `--keep-files`) |
| `test-sqlite.db` | SQLite database file (only with `--keep-files`) |

## Reproducing Errors

Always follow these steps

1. **Find the seed and profile** in the error output:
   ```
   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }
   ```

2. **Re-run with that seed and profile** (a seed only replays under the same profile):
   ```bash
   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files
   ```

3. **Read the minimized reproduction first.** On an oracle failure the fuzzer
   writes these files to `simulator-output/`:
   - `minimized.sql` - a shrunken state script plus the shrunken failing
     statement, produced automatically. Start here.
   - `turso-state.sql` / `sqlite-state.sql` - each engine's full state as a
     replayable script, when you need more than the minimized version kept.
   - `test.sql` - every executed statement (the failing one is marked
     `-- FAILED:`). The minimizer falls back to replaying this history when
     the failure depends on how the state was built, not just its contents.
   - `schema.json` - table structure at failure time.

4. **Probe the reproduction with `differential_probe`.** It runs a
   statement-per-line script on Turso and SQLite side by side, prints both
   outcomes for every statement, marks divergences, and compares the final
   table contents. Exit code 1 means something diverged.
   ```bash
   cargo run -q -p differential-fuzzer --bin differential_probe -- \
       simulator-output/minimized.sql
   ```
   Use it instead of piping SQL into the two shells: the tursodb shell cannot
   `ATTACH ':memory:' AS aux`, so fuzzer reproductions with an `aux` schema
   only run correctly through the probe. Reading from stdin also works:
   `echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe`.

5. **Bisect by editing the script.** Copy `minimized.sql`, simplify one thing
   at a time (replace an expression with a constant, drop a column, drop a
   state line), and re-run the probe after each edit. The divergence marker
   tells you immediately whether the edit kept the bug. This loop usually
   ends at a one-line kernel you can hand to `EXPLAIN` on both engines.

6. **Create a permanent `*.sqltest` regression** from the kernel before the
   fix is complete. Preserve only the schema, data, statements, and assertions
   required to expose the bug. The seed and minimized fuzzer artifact are
   debugging aids, not substitutes for this regression. Use a Rust test only
   when the behavior cannot be expressed in `.sqltest`, and always load the
   [Debugging skill for reference](../debugging/).

## Understanding Failures

### Oracle Failure Types

1. **Row set mismatch** - Turso returned different rows than SQLite
2. **Turso errored but SQLite succeeded** - Turso rejected valid SQL
3. **SQLite errored but Turso succeeded** - Turso accepted invalid SQL
4. **Schema mismatch** - Tables/columns differ after DDL

### Warning (non-fatal)

- **Unordered LIMIT mismatch** - LIMIT without ORDER BY may return different valid rows

## Key Source Files

| File | Purpose |
|------|---------|
| `main.rs` | CLI parsing, entry point |
| `runner.rs` | Main simulation loop, executes statements on both DBs |
| `oracle.rs` | Compares Turso vs SQLite results |
| `schema.rs` | Introspects schema from both databases |
| `memory/` | In-memory IO for deterministic simulation |

## Tracing

Set `RUST_LOG` for more detailed output:

```bash
RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345
```