# Nonobench

Runs language models against nonogram puzzles from 5x5 up to 20x20. Useful when you want a logic test rather than a recall test.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 10
- Author: mauricekleine
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/mauricekleine/nonobench
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/mcp/nonobench

## What it does

Nonobench is an open benchmark built around nonograms, also called Picross or Paint by Numbers. A model reads the numeric clues for every row and column, then fills the grid. Getting it right takes deduction and constraint satisfaction, so the score says something about how a model reasons.

The repository splits into `bench/`, which holds the runner, the results database and the exporter, and `visualizer/`, a TanStack Start dashboard on Cloudflare Workers that also carries the puzzle set. Runs are append-only: a model and puzzle pair that already succeeded is skipped, the database refuses to overwrite it, and failed attempts get retried on the next invocation. Everything lands in `bench/results.db` (SQLite).

Running it locally takes Bun and an OpenRouter key in `bench/.env`. To point it at your own server, set `NONOBENCH_LOCAL_BASE_URL` and `NONOBENCH_LOCAL_MODEL`; any OpenAI-compatible `/v1` endpoint works, vLLM, llama.cpp, LM Studio and Ollama included. The default bind address `127.0.0.1` only accepts connections from the machine itself, so the server has to listen on all interfaces. Keep that port off the internet. Most local servers ship without authentication, and anyone who reaches the port can use the model.

## Requirements
- Remote server: `https://www.nonobench.com/mcp`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http nonobench https://www.nonobench.com/mcp
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add nonobench --url https://www.nonobench.com/mcp
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "nonobench": {
            "url": "https://www.nonobench.com/mcp"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "nonobench": {
            "type": "http",
            "url": "https://www.nonobench.com/mcp"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http nonobench https://www.nonobench.com/mcp
```

## When to use it
- compare models on puzzles instead of trivia
- run one fixed puzzle set across several models
- benchmark a self-hosted model behind an OpenAI-compatible endpoint
- cap a session's spend on API calls
- re-grade stored answers with the current grader

## Example prompts
- Run the benchmark for `Qwen3-32B` at size 5x5 and print the plan first
- Execute `bun run bench --all-missing` and report how many puzzles are left
- Do a pilot on two 5x5 puzzles into `local-pilot.db`

## Similar
- [eval-model](https://affpapa.org/en/nevibecode/skills/eval-model.md) — skill · 981 installs
- [cost-benchmark](https://affpapa.org/en/nevibecode/skills/cost-benchmark.md) — skill · 679 installs
- [llm-eval-harness](https://affpapa.org/en/nevibecode/skills/llm-eval-harness.md) — skill · 372 installs
- [benchmark](https://affpapa.org/en/nevibecode/skills/takagoto-benchmark.md) — skill · ★ 24
- [benchmark-vllm-model](https://affpapa.org/en/nevibecode/skills/benchmark-vllm-model.md) — skill · 837 installs
- [local-llm-ops](https://affpapa.org/en/nevibecode/skills/local-llm-ops.md) — skill · 448 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md