Skip to main content
skill CX Codex

wiki-research-loop — Claude Code skill: research loop

wiki-research-loop is a Claude Code skill that researches a topic and adds pages to a wiki. It runs on requests such as research <topic> or instructions to grow a wiki automatically.

MIT ★ 2,966 updated 2026-09-29

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add rohitg00/pro-workflow
/plugin install pro-workflow@pro-workflow
Open on GitHub

What wiki-research-loop does

The skill works through a queue of research seeds, fetches material from the web, arXiv, or GitHub, then creates a wiki page or updates an existing one. It checks claims against saved pages and queues related follow-up topics. The loop stops at its budget or depth limit, or when results converge.

Run it with node $SKILL_ROOT/scripts/research-loop.js run <slug> and set limits or fetchers with flags. Each run saves a readable log and a JSON stats file. Web fetching uses the agent's available WebFetch tool; GitHub accepts an explicitly configured token, while arXiv needs no key. Page compilation uses an LLM through Anthropic/OpenAI passthrough. Private wikis reject non-local fetchers.

What it helps with

  • grow a wiki from queued research topics
  • update a page with material from fetched sources
  • stop a research run at its budget or depth limit
  • find repeated claims in new material

Example prompts for the agent

  • Research agent memory and add findings to the agent-memory wiki.
  • Continue research for the research-notes wiki using GitHub sources.
  • Add memory consolidation in agents to the agent-memory research queue.

How to install wiki-research-loop

1Plugin marketplace

/plugin marketplace add rohitg00/pro-workflow
/plugin install pro-workflow@pro-workflow

This installs the whole pro-workflow plugin with all its skills

2Skills CLI (Vercel)

npx skills add rohitg00/pro-workflow --skill wiki-research-loop -a claude-code

3Manually: put the files in place

~/.claude/skills/wiki-research-loop/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/wiki-research-loop/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about wiki-research-loop

How do I install wiki-research-loop in Claude Code?

Add the marketplace with /plugin marketplace add rohitg00/pro-workflow, then install the plugin with /plugin install pro-workflow@pro-workflow.

What does wiki-research-loop do?

It processes queued research topics, fetches sources, creates or updates wiki pages, and adds related topics to the queue. It halts at configured limits or when the results converge.

What does wiki-research-loop need to run?

The CLI requires Node. Web searches use the available WebFetch tool; GitHub can use an explicitly configured token, and arXiv needs no key. Page compilation calls an LLM through Anthropic/OpenAI passthrough.

Source text

SKILL.md · MIT
---
name: wiki-research-loop
description: Auto-grow a pro-workflow wiki by running a budget-capped BFS research loop over pluggable source fetchers (web, arXiv, GitHub). Each iteration pops a seed from the queue, fetches sources, drafts a wiki page, dedupes claims against existing pages, enqueues follow-up seeds. Halts on budget cap, depth cap, or convergence. Use when the user says "research <topic>", "grow the <slug> wiki", "auto-research", or wants a knowledge base that builds itself overnight.
user-invocable: true
---

# Wiki Research Loop

Driver that turns a wiki into an auto-grown knowledge base. Layers on top of `wiki-builder` and `wiki-query`.

## Loop semantics

```
seed-queue (pending) → next-seed
  → fetch sources via plugins (web | arxiv | github)
  → extract claims
  → dedupe vs index (FTS5; later vector via 3.3.2)
  → compile new page or amend existing
  → upsert page (auto-FTS-index)
  → enqueue follow-up seeds (max-depth gate)
  → mark seed done
  → if budget OR convergence OR kill-switch → halt
```

## Halt conditions (any one trips)

- `budget_usd` exceeded (loop tracks per-fetcher cost estimate)
- `max_pages_per_run` written
- `max_depth` reached on every active branch
- 3 consecutive pages add < 5 % new claims (convergence)
- File `~/.pro-workflow/STOP` exists (operator kill-switch)
- `wiki.config.md` `auto_research.enabled: false`
- Wiki `private: true` AND any non-local fetcher selected

## Commands

In a plugin session, call the `providers` MCP server's `run_provider_task` tool with `task: "research"` and the runner arguments below, for example `args: ["run", "agent-memory", "--fetchers", "github"]`. Its optional GitHub token comes from the plugin configuration dialog. The direct CLI uses `PRO_WORKFLOW_GITHUB_TOKEN` when explicitly supplied. Never retrieve tokens from existing machine credentials.

```
node $SKILL_ROOT/scripts/research-loop.js run <slug> [--max-pages N] [--max-depth N] [--budget-usd 0.50] [--fetchers web,arxiv,github]
node $SKILL_ROOT/scripts/research-loop.js seed <slug> "<query>" [--depth 0] [--parent-id N]
node $SKILL_ROOT/scripts/research-loop.js seeds <slug> [--status pending|active|done|failed]
node $SKILL_ROOT/scripts/research-loop.js cancel <slug>
node $SKILL_ROOT/scripts/research-loop.js status
```

CLI flags override `wiki.config.md` for one run only.

## Source fetchers

Pluggable. Each lives at `scripts/source-fetchers/<name>.js`. Interface:

```js
module.exports = {
  name: 'web',
  match: (q) => true,                       // is this fetcher useful?
  estimateCost: (q) => ({ usd: 0, tokens: 0 }),
  fetch: async (q, opts) => [               // returns RawDoc[]
    { url, title, content, fetched_at }
  ]
};
```

Built-in:
- **`web.js`** — Fetches via the user's available `WebFetch` tool through a stdin/stdout shim. Treats result as plain text/markdown.
- **`arxiv.js`** — `https://export.arxiv.org/api/query` (free, public, no key). Returns abstract + metadata.
- **`github.js`** — `https://api.github.com/search/repositories` + README pull (uses the explicitly configured GitHub token, otherwise unauthenticated rate limit).

Standalone CLI runs load custom fetchers from `~/.pro-workflow/fetchers/<name>.js`. The plugin MCP server uses only bundled fetchers so configured tokens are not handed to user-supplied modules.

## Budget enforcement

Pre-iteration: sum `estimateCost` across selected fetchers. If projected cumulative cost would exceed `budget_usd`, halt.

Post-iteration: track tokens used by the LLM compile step (Anthropic/OpenAI passthrough). Hard-kill on overrun.

Per-fetcher overrides via env: `WIKI_LOOP_BUDGET_USD`, `WIKI_LOOP_MAX_PAGES`, `WIKI_LOOP_MAX_DEPTH`.

## Seed queue

SQLite-backed via `wiki_seeds` table:

| field | meaning |
|-------|---------|
| `query` | natural-language seed |
| `status` | `pending` → `active` → `done`\|`failed` |
| `parent_id` | seed that produced this one |
| `depth` | BFS depth from root |

Loop pops by `(depth ASC, created_at ASC)` so it explores breadth-first.

## Convergence detection

After each compiled page, compute Jaccard overlap of claim-text tokens vs the prior 3 pages. If `< 5 %` novel content for 3 consecutive pages, halt and report `converged`.

## Kill switch

```
touch ~/.pro-workflow/STOP
```

Loop checks per-iteration and halts gracefully. Remove file to resume next run.

## Privacy guard

If `wiki.config.md` has `private: true`, the loop refuses any non-local fetcher and emits a warning. Only `raw/` ingestion via manual seeds is allowed.

## Reactive trigger (Phase 3.3.4)

`scripts/file-watcher.js` watches `wiki/<slug>/wiki/**/*.md`. On user-edited claim, enqueues a verification seed (`verify: <claim>`) at depth 0. Wired through pro-workflow's `file-watcher.js` hook.

## Cron tick (Phase 3.3.4)

`scripts/research-tick.js` is launchable from any cron-style runner. Picks the oldest opted-in wiki with pending seeds and runs a single iteration. Hook event: `pro-workflow:research-tick`.

## Output

Each run writes:

```
<wiki-root>/logs/research-<UTC-timestamp>.md   # human-readable run log
<wiki-root>/derived/run-<UTC-timestamp>.json   # structured stats
```

Run log lines:

```
[2026-05-08T10:42Z] seed-3 (depth=1) "memory consolidation in agents"
  fetcher=arxiv hits=3
  fetcher=web hits=2
  compiled wiki/concepts/memory-consolidation.md (claims=7, novel=4)
  enqueued 2 follow-up seeds
  cost so far: $0.04 / $0.50
```

## Integration with `wiki-query`

Every compiled page goes through `wiki-cli.js page` so FTS5 stays consistent. The dedupe step calls `searchWiki` with the candidate claim text to find near-duplicates.

## Status (Phase 3.3.1)

Ships: loop driver, seed queue, web/arxiv/github fetchers, budget caps, convergence detector, kill-switch, manual `run` command.

Defers:
- Vector dedupe (Phase 3.3.2 via sqlite-vec)
- LLM-judged claim novelty (current = Jaccard token overlap)
- Cron + reactive (Phase 3.3.4)

© Rohit Ghumare · MIT · rohitg00/pro-workflow

Similar skills

All skills for Agent setup: memory, context, orchestration →

From the same set83

The whole set →
sprint-status sprint-status tracks parallel Claude Code sessions and prevents confusion by enforcing session headers and standardized status lines. CX Codex +1 ★ 2,966repo stars skill-router The skill-router skill maps pro-workflow skills and commands to jobs, with guidance on when to use each and whether a person or the agent starts it. Use it when you’re unsure what fits. CX Codex +1 ★ 2,966repo stars permission-analyst permission-analyst analyzes permission prompts and auto mode denials, recommends permissions rules and autoMode entries. Use it when prompts slow you down or auto mode keeps falling back to confirmations. CX Codex ★ 2,966repo stars safe-mode safe-mode intercepts destructive terminal commands and confines agent edits to a specific directory using Claude Code hooks. CX Codex +1 ★ 2,966repo stars survey-generator survey-generator generates structured literature surveys on AI/ML topics. Output is a wiki page in markdown with bibliography, not a one-off HTML file. CX Codex +1 ★ 2,966repo stars context-engineering context-engineering guides Claude Code through the four context operations — Write, Select, Compress, and Isolate — to prevent context degradation. CX Codex +1 ★ 2,966repo stars learn learn is a Claude Code command that shows topic-based guidance and saves confirmed takeaways from a session. CX Codex +1 ★ 2,966repo stars pro-workflow pro-workflow — a skill that packages coding habits for AI agents: correction loops, pre-flight rules, worktree sessions and a wrap-up ritual. CX Codex +1 ★ 2,966repo stars

Project sponsors

Companies that keep this analytics open