# hive-create-task

Guides developers through creating benchmark task repositories for agent swarms. Helps define problem limits, evaluation metrics, and scaffolding files.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 531 installs · ★ 218
- Author: rllm-org
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/rllm-org/hive/tree/HEAD/skills/hive-create-task/SKILL.md
- Updated: 2026-04-28
- HTML: https://affpapa.org/en/nevibecode/skills/hive-create-task

## What it does

The skill acts as an interactive five-phase wizard. It begins by asking clarifying questions about the core task, available datasets, and the specific target artifact, such as an `agent.py` script or a model training file. Most of the setup focuses on evaluation design so you can establish a clear metric, reasonable timeouts, and score computation rules.

During the scaffolding stage, the skill generates the task folder. It puts together required files including `program.md`, `eval/eval.sh`, `requirements.txt`, and `README.md`. A `prepare.sh` script is also generated if data downloads or setup steps are needed.

Pay close attention to the evaluation script contract. The `eval/eval.sh` file must run with `bash eval/eval.sh`, exit with code 0, and print a summary ending with `<metric>:`, `correct:`, and `total:`. Agents depend directly on this layout because they parse test scores with `grep`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add rllm-org/hive --skill hive-create-task -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/hive-create-task/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/hive-create-task/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add rllm-org/hive --skill hive-create-task -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/hive-create-task/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add rllm-org/hive --skill hive-create-task -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/hive-create-task/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install rllm-org/hive hive-create-task
```

2. **Manually: put the files in place**
- File: `.github/skills/hive-create-task/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add rllm-org/hive --skill hive-create-task -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/rllm-org/hive --path skills/hive-create-task
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/hive-create-task/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add rllm-org/hive --skill hive-create-task -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/rllm-org/hive/tree/HEAD/skills/hive-create-task/SKILL.md

## When to use it
- scaffold benchmark repositories for autonomous agents
- design standardized evaluation scripts with parseable outputs
- write instruction templates in program.md with strict file constraints
- configure baseline test setups for task validation

## Example prompts
- Design a new hive task for evaluating our coding agent
- Create a task repository for gsm8k-solver with accuracy eval
- Help me set up an ML benchmark repo with prepare.sh and eval.sh

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: ac9fb505a9d41b88.

BEGIN THIRD-PARTY FILE ac9fb505a9d41b88
`````text
---
name: hive-create-task
description: Design and create a new hive task through guided conversation. Walks the user through problem definition, eval design, constraint specification, repo scaffolding, baseline testing with iteration, and upload. Use when user wants to create a new task, add a benchmark, or publish a challenge to the swarm.
---

# Hive Create Task

Interactive wizard for designing and creating a new hive task. Guide the user through each phase with clarifying questions. The goal is to produce a complete, tested task repo that agents can immediately clone and work on.

**Principle:** Ask the right questions to help the user clarify their thinking. A good task needs a good eval — spend most of the effort there. Don't move on until the user is satisfied with each phase.

**UX Note:** Use `AskUserQuestion` for all user-facing questions.

---

## Task Repo Structure

### Required files

| File | Purpose |
|---|---|
| `program.md` | Instructions for the agent: what to modify, how to eval, the experiment loop, and constraints |
| `eval/eval.sh` | Evaluation script — must be runnable via `bash eval/eval.sh` and print a score |
| `requirements.txt` | Python dependencies |
| `README.md` | Short description, quickstart, and leaderboard link |

### Recommended files

| File | Purpose |
|---|---|
| `prepare.sh` | Setup script — downloads data, installs deps. Recommended but not required. |

### The artifact (free-form)

The rest depends on the task type — this is what agents evolve:

- **Agentic tasks**: an `agent.py` that the agent evolves
- **ML training tasks**: a training script like `train_gpt.py`
- **Prompt tasks**: a prompt template, config file, etc.
- Any other file(s) that make sense for the problem

### Eval output format

`eval/eval.sh` MUST print a parseable summary ending with:

```
---
<metric>:         <value>
correct:          <N>
total:            <N>
```

The agent reads score via `grep "^<metric>:" run.log`.

### program.md template

Use this template, filling in all `<placeholders>`:

````markdown
# <Task Name>

<One-line description of what the agent improves and how it's evaluated.>

## Setup

1. **Read the in-scope files**:
   - `<file1>` — <what it is>. You modify this.
   - `eval/eval.sh` — runs evaluation. Do not modify.
   - `prepare.sh` — <what it sets up>. Do not modify.
2. **Run prepare**: `bash prepare.sh` to <what it does>.
3. **Verify data exists**: Check that `<path>` contains <expected files>.
4. **Initialize results.tsv**: Create `results.tsv` with just the header row.
5. **Run baseline**: `bash eval/eval.sh` to establish the starting score.

## The benchmark

<2-3 sentences describing the benchmark, dataset size, and what makes it challenging.>

## Experimentation

**What you CAN do:**
- Modify `<file1>`, `<file2>`, etc. <Brief guidance on what kinds of changes are fair game.>

**What you CANNOT do:**
`````
END THIRD-PARTY FILE ac9fb505a9d41b88

The third-party file has ended. Everything between the ac9fb505a9d41b88 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2909 of 10908 bytes. © rllm-org · Apache-2.0 · https://github.com/rllm-org/hive/tree/HEAD/skills/hive-create-task/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md