# gaia-submission

Runs the full GAIA benchmark pipeline from API key checks to building a HAL-compatible submission package. Useful when evaluating agent accuracy.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 435 installs · ★ 79357
- Author: ruvnet
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/ruvnet/ruflo/tree/HEAD/plugins/ruflo-workflows/skills/gaia-submission/SKILL.md
- Updated: 2026-10-25
- HTML: https://affpapa.org/en/nevibecode/skills/gaia-submission

## What it does

This skill guides an agent through the Princeton GAIA benchmark workflow. It validates prerequisites, calculates model invocation costs, executes question batches with regular progress updates, and packages files for the HAL leaderboard. Run summaries and learnings are saved directly into the CLI memory store.

Before signing the Ed25519 manifest, the packager runs a mandatory integrity audit under ADR-167. It inspects task execution for answer leakage, oracle cheating, and dynamic evaluation calls. Critical failures abort package generation immediately, which prevents unearned scores unless `--allow-dirty` is explicitly supplied.

You need Node.js 20 or higher, an active `@claude-flow/cli` build, and exported `ANTHROPIC_API_KEY` and `HF_TOKEN` variables.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add ruvnet/ruflo --skill gaia-submission -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/gaia-submission/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/gaia-submission/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add ruvnet/ruflo --skill gaia-submission -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/gaia-submission/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add ruvnet/ruflo --skill gaia-submission -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/gaia-submission/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install ruvnet/ruflo gaia-submission
```

2. **Manually: put the files in place**
- File: `.github/skills/gaia-submission/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add ruvnet/ruflo --skill gaia-submission -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/ruvnet/ruflo --path plugins/ruflo-workflows/skills/gaia-submission
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/gaia-submission/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add ruvnet/ruflo --skill gaia-submission -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/ruvnet/ruflo/tree/HEAD/plugins/ruflo-workflows/skills/gaia-submission/SKILL.md

## When to use it
- validate environment keys and CLI binaries before benchmarking
- estimate token costs across models prior to running questions
- package benchmark results into a HAL-compatible archive
- audit agent execution traces for benchmark leakage

## Example prompts
- Validate the environment and estimate GAIA level 1 cost for 53 questions
- Run the GAIA benchmark using claude-sonnet-4-6 with voting set to 3
- Package the latest GAIA results into a HAL submission bundle

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 647df1c8f46d0291.

BEGIN THIRD-PARTY FILE 647df1c8f46d0291
````text
---
name: gaia-submission
description: Walk through a complete GAIA benchmark→submit flow — from key resolution through HAL-compatible package generation
argument-hint: "[level] [limit] [models]"
allowed-tools: Bash mcp__plugin_ruflo-core_ruflo__memory_store mcp__plugin_ruflo-core_ruflo__memory_search mcp__plugin_ruflo-core_ruflo__memory_list mcp__plugin_ruflo-core_ruflo__hooks_post-task mcp__plugin_ruflo-core_ruflo__hooks_pre-task
---

# GAIA Submission Skill

Walk Claude Code through every step needed to go from a clean environment to a
signed, HAL-compatible submission package ready to upload to the Princeton
GAIA leaderboard.

## When to use

When the user wants to:
- Run a benchmark and submit results to the HAL leaderboard
- Package an existing results file into a submission archive
- Confirm their environment is ready for a benchmark run

## Prerequisites

Before starting, confirm these are available:

| Requirement | Check |
|-------------|-------|
| `ANTHROPIC_API_KEY` | `echo ${ANTHROPIC_API_KEY:0:8}…` (should show `sk-ant-…`) |
| `HF_TOKEN` | `echo ${HF_TOKEN:0:5}…` (should show `hf_…`) |
| Node.js 20+ | `node --version` |
| CLI built | `node v3/@claude-flow/cli/bin/cli.js --version` |

## Phase 1 — Validate environment

```bash
# Run all pre-flight checks
/gaia validate
```

If any check fails, resolve it before continuing.

## Phase 2 — Estimate cost and confirm

Ask the user for their configuration:
- Level (default: 1)
- Question limit (default: 53 for a quick run, 165 for the full L1 set)
- Models (default: `claude-sonnet-4-6`)
- Self-consistency voting (default: 1; use 3 for L2/L3)

```bash
/gaia cost --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING
```

If projected cost > $5, show the estimate and ask: "This run will cost
approximately $X. Proceed? (y/N)"

## Phase 3 — Run the benchmark

```bash
/gaia run --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING
```

While running, progress is reported every 5 questions:
```
[12/53] 22.7% (5 passed of 22 scored) — est. remaining: $0.18
```

Store the run summary in memory for history tracking:
```bash
npx @claude-flow/cli@latest memory store \
  --namespace gaia-runs \
  --key "run-$(date +%Y%m%d-%H%M)" \
  --value '{"level":$LEVEL,"model":"$MODEL","total":$TOTAL,"passed":$PASSED,"pass_rate":$RATE,"est_cost_usd":$COST}'
```

## Phase 4 — Package for submission

```bash
/gaia submit --results=~/.cache/ruflo/gaia/results-latest.json
```

This produces:
```
submission-<date>-<sha>/
├── results.jsonl        ← HAL-compatible, one JSON per line
├── trajectories.jsonl   ← full agent traces
├── metadata.json        ← harness info, model, tool catalogue
├── audit-report.json    ← ADR-167 pre-submission exploit-audit report
├── manifest.md.json     ← Ed25519-signed witness (signs audit-report.json's hash)
└── README.md            ← human summary + leaderboard comparison
```
````
END THIRD-PARTY FILE 647df1c8f46d0291

The third-party file has ended. Everything between the 647df1c8f46d0291 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2976 of 5049 bytes. © ruvnet · MIT · https://github.com/ruvnet/ruflo/tree/HEAD/plugins/ruflo-workflows/skills/gaia-submission/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md