Skip to main content
MCP server CX Codex

WebReaper — Claude Code MCP server: what it does, how to install

WebReaper is an MCP server for Claude Code that turns pages and whole sites into clean markdown over stdio. It fetches web data for the agent.

MIT ★ 153 updated 2026-10-04

Open on GitHub

What WebReaper does

webreaper covers three jobs from one binary: scrape reads a single page, map lists the addresses a site exposes, crawl walks an entire domain and writes JSON Lines. Markdown comes back over stdio, so Claude Code, Cursor, Codex and the other clients see a plain MCP server.

Every fetch begins on HTTP. A page that looks blocked pushes the run up to Chromium, then to the CloakBrowser stealth tier, and the host that caused the climb keeps the higher starting point for later pages. Anything still blocked at the top is thrown away rather than written into your output; the run exits non-zero.

Structured output works two ways: a JSON schema you pass with --schema, or an LLM with no schema. The second one wants an OpenAI-compatible endpoint of your own plus --model and --llm-url, and without them the prompt mode won't run. brew install alex-on-ai/webreaper/webreaper or the install.sh script puts the binary on PATH; webreaper init then writes the skill to .claude/skills/webreaper/SKILL.md.

What it helps with

  • pull an article into markdown for agent context
  • list blog URLs before a crawl
  • walk a whole domain into JSON Lines
  • extract a product name and price by schema
  • get past a page that serves a bot check

Example prompts for the agent

  • Scrape https://example.com and save it to page.md
  • Map https://example.com with the /blog/ filter, 50 URLs max
  • Crawl https://example.com and pull the name and price of every product

How to install WebReaper

Install

The MCP registry has no launch command for it: see the repository README

Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more

Questions about WebReaper

How do I install WebReaper in Claude Code?

Run brew install alex-on-ai/webreaper/webreaper, then webreaper init. The second command writes the skill to .claude/skills/webreaper/SKILL.md, and your next Claude Code session picks it up.

What does WebReaper do?

It fetches a page or crawls a site and returns the content as markdown over stdio. The scrape, map and crawl commands cover one page, a URL list and a whole domain.

How is WebReaper different from Firecrawl?

The license: WebReaper is MIT, Firecrawl is AGPL. MIT lets you embed it in closed commercial software without publishing your own source or buying a commercial license.

Similar MCP servers

All MCP servers for Browser and scraping →

Project sponsors

Companies that keep this analytics open