# CRW Web Scraper

Turns pages into markdown or JSON. Scrape, crawl, map URLs, search the web, extract fields — one MCP server for your agent.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 1173
- Author: fastcrw
- License: AGPL-3.0 · policy=link-only · description only; the file stays with the author
- Source: https://github.com/fastcrw/crw
- Updated: 2026-10-21
- HTML: https://affpapa.org/en/nevibecode/mcp/crw-web-scraper

## What it does

Five operations are exposed to the agent. Scrape handles a single URL and returns markdown, HTML, links, screenshots or schema JSON. Crawl follows a bounded set of pages on a site. Map lists URLs without fetching each one. Search queries the web and can scrape chosen results. Extract pulls structured fields from one URL or many.

The engine is written in Rust and ships as a small binary you can run locally, or you can point it at the managed API instead. Local use needs no account. With the cloud option your key lives in `~/.config/crw/config.toml` rather than being duplicated into every tool's config. The installer detects Claude, Cursor, Codex, Gemini CLI, OpenCode and Windsurf when they are already configured; set `CRW_NO_AGENTS=1` to skip that, or run `crw setup` to pick tools yourself.

Building from source requires Rust 1.85 or newer. The install command above runs the Docker image `ghcr.io/fastcrw/crw:latest`, so without Docker it will not start. The project is AGPL-3.0 — worth reading the terms if you plan to embed the engine in your own product.

## Requirements
- Needs: Docker

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latest
```

### Codex

**Platform CLI**
```bash
codex mcp add crw-web-scraper -- docker run -i --rm ghcr.io/fastcrw/crw:latest
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "crw-web-scraper": {
            "command": "docker",
            "args": [
                "run",
                "-i",
                "--rm",
                "ghcr.io/fastcrw/crw:latest"
            ]
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "crw-web-scraper": {
            "type": "stdio",
            "command": "docker",
            "args": [
                "run",
                "-i",
                "--rm",
                "ghcr.io/fastcrw/crw:latest"
            ]
        }
    }
}
```

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add crw-web-scraper docker run -i --rm ghcr.io/fastcrw/crw:latest
```

## When to use it
- pull a page into markdown for later parsing
- list a section's URLs without downloading every page
- walk a site within a page limit and collect the results
- extract structured fields from several URLs at once
- find web material on a topic and scrape selected hits

## Example prompts
- Scrape `https://example.com` and give me the markdown
- Map the URLs on this site, don't fetch the pages
- Search for Rust async runtime articles and scrape the top three results

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md