Nonobench — Claude Code MCP server: a nonogram benchmark
Runs language models against nonogram puzzles from 5x5 up to 20x20. Useful when you want a logic test rather than a recall test.
claude mcp add --transport http nonobench https://www.nonobench.com/mcpWhat Nonobench does
Nonobench is an open benchmark built around nonograms, also called Picross or Paint by Numbers. A model reads the numeric clues for every row and column, then fills the grid. Getting it right takes deduction and constraint satisfaction, so the score says something about how a model reasons.
The repository splits into bench/, which holds the runner, the results database and the exporter, and visualizer/, a TanStack Start dashboard on Cloudflare Workers that also carries the puzzle set. Runs are append-only: a model and puzzle pair that already succeeded is skipped, the database refuses to overwrite it, and failed attempts get retried on the next invocation. Everything lands in bench/results.db (SQLite).
Running it locally takes Bun and an OpenRouter key in bench/.env. To point it at your own server, set NONOBENCH_LOCAL_BASE_URL and NONOBENCH_LOCAL_MODEL; any OpenAI-compatible /v1 endpoint works, vLLM, llama.cpp, LM Studio and Ollama included. The default bind address 127.0.0.1 only accepts connections from the machine itself, so the server has to listen on all interfaces. Keep that port off the internet. Most local servers ship without authentication, and anyone who reaches the port can use the model.
What it helps with
- compare models on puzzles instead of trivia
- run one fixed puzzle set across several models
- benchmark a self-hosted model behind an OpenAI-compatible endpoint
- cap a session's spend on API calls
- re-grade stored answers with the current grader
Example prompts for the agent
- Run the benchmark for
Qwen3-32Bat size 5x5 and print the plan first - Execute
bun run bench --all-missingand report how many puzzles are left - Do a pilot on two 5x5 puzzles into
local-pilot.db
How to install Nonobench
Platform CLI: remote server
claude mcp add --transport http nonobench https://www.nonobench.com/mcpPlatform CLI: remote server
codex mcp add nonobench --url https://www.nonobench.com/mcpConfig file
{
"mcpServers": {
"nonobench": {
"url": "https://www.nonobench.com/mcp"
}
}
}Config file
{
"servers": {
"nonobench": {
"type": "http",
"url": "https://www.nonobench.com/mcp"
}
}
}Platform CLI: remote server
gemini mcp add -t http nonobench https://www.nonobench.com/mcpStep by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about Nonobench
How do I install Nonobench in Claude Code?
One command: claude mcp add --transport http nonobench https://www.nonobench.com/mcp.
What does Nonobench do?
It puts models through a set of nonogram puzzles from 5x5 to 20x20 and stores the outcomes in SQLite. Grading is the same in structured and text mode: an answer has to satisfy every clue.
What do I need to benchmark my own model?
A server with an OpenAI-compatible /v1 endpoint, listening on all interfaces, plus NONOBENCH_LOCAL_BASE_URL and NONOBENCH_LOCAL_MODEL. The runner itself needs Bun and an OpenRouter key.
