# PyreCrawl

A self-hosted MCP server that turns web pages into markdown for your agent, gets past Cloudflare and tracks page changes. No API keys involved.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 22
- Author: SanggonBoy
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/SanggonBoy/PyreCrawl
- Updated: 2026-09-21
- HTML: https://affpapa.org/en/nevibecode/mcp/pyrecrawl

## What it does

PyreCrawl exposes thirteen tools to the agent. `scrape` returns a single page as markdown, `crawl` walks a site with path filters and depth limits, and `map_site` lists internal links from a root. Research runs through `search` backed by DuckDuckGo and `search_papers` over arXiv and Crossref; `deep_research` bundles search results and scraped pages into an evidence pack with numbered citations. `document` pulls text out of PDF, DOCX and PPTX files without starting a browser.

The fetching logic picks the cheapest route that works. Plain HTTP goes first; a 403, a Cloudflare challenge or an empty body escalates to a stealth Chromium browser, and deep processing handles pages that need full JavaScript. For pages behind a login, `session` keeps cookies across steps. `monitor` snapshots a URL and returns a unified diff on later runs. Turn on the response cache with `PYRECRAWL_CACHE`; entries expire according to `PYRECRAWL_CACHE_TTL`.

UV comes first — it fetches Python itself, so you do not install an interpreter by hand. Browser engines need a one-time `pyrecrawl setup` that takes around two minutes. Skip it and stealth mode has nothing to run, so Cloudflare-protected sites stay out of reach. Set `PYRECRAWL_NO_TELEMETRY=1` to silence the anonymous startup ping.

## Requirements
- Needs: Python and uv (commands run via uvx)

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add pyrecrawl -- uvx pyrecrawl
```
uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>

### Codex

**Platform CLI**
```bash
codex mcp add pyrecrawl -- uvx pyrecrawl
```
uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "pyrecrawl": {
            "command": "uvx",
            "args": [
                "pyrecrawl"
            ]
        }
    }
}
```
uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "pyrecrawl": {
            "type": "stdio",
            "command": "uvx",
            "args": [
                "pyrecrawl"
            ]
        }
    }
}
```
uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add pyrecrawl uvx pyrecrawl
```
uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>

## When to use it
- turn one web page into markdown for the agent context
- get past Cloudflare on a protected site
- find academic papers on a topic and pull their PDFs
- watch a product page for price changes
- log in once and continue a multi-step browser flow

## Example prompts
- Scrape https://example.com and give me the key points
- Find papers on Rust memory safety and build a summary with citations
- Monitor this page and show me what changed since last time

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md