# Webcrawl Anything

Webcrawl Anything — an MCP server for Claude Code that searches the web, parses documents and crawls pages locally without an API key. Runs locally on your machine to integrate web tools into the agent environment.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 14 repo stars
- Author: brainfish-ai
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/brainfish-ai/webcrawl-anything/tree/HEAD/packages/mcp
- Updated: 2026-11-15
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/mcp/webcrawl-anything

## What it does

The server connects web search across verticals like shopping and flights, JavaScript-rendered page reading, and structured JSON data extraction to the agent environment. Tools parse PDF and Office files, YouTube videos, and audio into Markdown, map domain links, and crawl target pages.

The package runs locally on your machine without an API key by loading the core engine from `dist/core`. If that import fails, the process exits with an error. For cloud mode, configure a hosted endpoint or an API key.

The setup command registers the server and installs the agent skill automatically. Set the schema profile to compact in the client configuration to minimize context token usage.

## Requirements
- Needs: Node.js (commands run via npx)
- Remote server: `https://webcrawl.app/mcp`
- Environment variables: `WEBCRAWL_API_KEY`, `WEBCRAWL_MODE`, `WEBCRAWL_MCP_SCHEMA_PROFILE` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

1. **Platform CLI: remote server**
```bash
claude mcp add --transport http webcrawl https://webcrawl.app/mcp
```

2. **Platform CLI**
```bash
claude mcp add --env WEBCRAWL_API_KEY=<WEBCRAWL_API_KEY> --env WEBCRAWL_MODE=<WEBCRAWL_MODE> --env WEBCRAWL_MCP_SCHEMA_PROFILE=<WEBCRAWL_MCP_SCHEMA_PROFILE> --transport stdio mcp -- npx -y @webcrawl/mcp
```
Replace <…> with your own values

### Codex

1. **Platform CLI: remote server**
```bash
codex mcp add mcp --url https://webcrawl.app/mcp
```

2. **Platform CLI**
```bash
codex mcp add mcp --env WEBCRAWL_API_KEY=<WEBCRAWL_API_KEY> --env WEBCRAWL_MODE=<WEBCRAWL_MODE> --env WEBCRAWL_MCP_SCHEMA_PROFILE=<WEBCRAWL_MCP_SCHEMA_PROFILE> -- npx -y @webcrawl/mcp
```
Replace <…> with your own values

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "mcp": {
            "url": "https://webcrawl.app/mcp"
        }
    }
}
```
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@webcrawl/mcp"
            ],
            "env": {
                "WEBCRAWL_API_KEY": "<WEBCRAWL_API_KEY>",
                "WEBCRAWL_MODE": "<WEBCRAWL_MODE>",
                "WEBCRAWL_MCP_SCHEMA_PROFILE": "<WEBCRAWL_MCP_SCHEMA_PROFILE>"
            }
        }
    }
}
```
Replace <…> with your own values

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "mcp": {
            "type": "http",
            "url": "https://webcrawl.app/mcp"
        }
    }
}
```
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "mcp": {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "@webcrawl/mcp"
            ],
            "env": {
                "WEBCRAWL_API_KEY": "<WEBCRAWL_API_KEY>",
                "WEBCRAWL_MODE": "<WEBCRAWL_MODE>",
                "WEBCRAWL_MCP_SCHEMA_PROFILE": "<WEBCRAWL_MCP_SCHEMA_PROFILE>"
            }
        }
    }
}
```
Replace <…> with your own values

### Gemini CLI

1. **Platform CLI: remote server**
```bash
gemini mcp add -t http mcp https://webcrawl.app/mcp
```

2. **Platform CLI**
```bash
gemini mcp add -e WEBCRAWL_API_KEY=<WEBCRAWL_API_KEY> -e WEBCRAWL_MODE=<WEBCRAWL_MODE> -e WEBCRAWL_MCP_SCHEMA_PROFILE=<WEBCRAWL_MCP_SCHEMA_PROFILE> mcp npx -y @webcrawl/mcp
```
Replace <…> with your own values

## When to use it
- search the web and structured verticals
- parse PDF and Office documents into Markdown
- map domain links and extract sitemap entries
- extract structured JSON data from web pages
- rerank candidate documents by query relevance

## Example prompts
- Search the web for recent product reviews using webcrawl
- Read the documentation page and return clean Markdown via web_read
- Parse the PDF document into text using web_parse

## Similar
- [Olostep MCP Server](https://affpapa.org/en/nevibecode/mcp/olostep-mcp-server.md) — MCP server · ★ 25 repo stars
- [MarkUDown](https://affpapa.org/en/nevibecode/mcp/markudown.md) — MCP server · ★ 16 repo stars
- [firecrawl](https://affpapa.org/en/nevibecode/skills/firecrawl.md) — skill · 104500 installs
- [exa-mcp](https://affpapa.org/en/nevibecode/mcp/exa-mcp.md) — MCP server · 144 installs
- [google-search-agent-teneo](https://affpapa.org/en/nevibecode/skills/google-search-agent-teneo.md) — skill · 521 installs
- [agent-reach](https://affpapa.org/en/nevibecode/skills/terrylica-agent-reach.md) — skill · 195 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
