CRW Web Scraper — Claude Code MCP server: scrape, crawl, map
Turns pages into markdown or JSON. Scrape, crawl, map URLs, search the web, extract fields — one MCP server for your agent.
claude mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latestWhat CRW Web Scraper does
Five operations are exposed to the agent. Scrape handles a single URL and returns markdown, HTML, links, screenshots or schema JSON. Crawl follows a bounded set of pages on a site. Map lists URLs without fetching each one. Search queries the web and can scrape chosen results. Extract pulls structured fields from one URL or many.
The engine is written in Rust and ships as a small binary you can run locally, or you can point it at the managed API instead. Local use needs no account. With the cloud option your key lives in ~/.config/crw/config.toml rather than being duplicated into every tool's config. The installer detects Claude, Cursor, Codex, Gemini CLI, OpenCode and Windsurf when they are already configured; set CRW_NO_AGENTS=1 to skip that, or run crw setup to pick tools yourself.
Building from source requires Rust 1.85 or newer. The install command above runs the Docker image ghcr.io/fastcrw/crw:latest, so without Docker it will not start. The project is AGPL-3.0 — worth reading the terms if you plan to embed the engine in your own product.
What it helps with
- pull a page into markdown for later parsing
- list a section's URLs without downloading every page
- walk a site within a page limit and collect the results
- extract structured fields from several URLs at once
- find web material on a topic and scrape selected hits
Example prompts for the agent
- Scrape
https://example.comand give me the markdown - Map the URLs on this site, don't fetch the pages
- Search for Rust async runtime articles and scrape the top three results
How to install CRW Web Scraper
Platform CLI
claude mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latestPlatform CLI
codex mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latestConfig file
{
"mcpServers": {
"crw-web-scraper": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/fastcrw/crw:latest"
]
}
}
}Config file
{
"servers": {
"crw-web-scraper": {
"type": "stdio",
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/fastcrw/crw:latest"
]
}
}
}Platform CLI
gemini mcp add crw-web-scraper docker run -i --rm ghcr.io/fastcrw/crw:latestStep by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about CRW Web Scraper
How do I install CRW Web Scraper in Claude Code?
Run claude mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latest. Docker has to be installed and running.
What does CRW Web Scraper do?
It gives the agent page scraping to markdown or JSON, site crawling, URL mapping, web search and structured field extraction.
Do I need an API key for local mode?
No. Local runs are free and need no account. A key is only for the managed API, and it is stored in ~/.config/crw/config.toml.
