PyreCrawl — Claude Code MCP server: scrape, crawl and monitor sites
A self-hosted MCP server that turns web pages into markdown for your agent, gets past Cloudflare and tracks page changes. No API keys involved.
claude mcp add pyrecrawl -- uvx pyrecrawlWhat PyreCrawl does
PyreCrawl exposes thirteen tools to the agent. scrape returns a single page as markdown, crawl walks a site with path filters and depth limits, and map_site lists internal links from a root. Research runs through search backed by DuckDuckGo and search_papers over arXiv and Crossref; deep_research bundles search results and scraped pages into an evidence pack with numbered citations. document pulls text out of PDF, DOCX and PPTX files without starting a browser.
The fetching logic picks the cheapest route that works. Plain HTTP goes first; a 403, a Cloudflare challenge or an empty body escalates to a stealth Chromium browser, and deep processing handles pages that need full JavaScript. For pages behind a login, session keeps cookies across steps. monitor snapshots a URL and returns a unified diff on later runs. Turn on the response cache with PYRECRAWL_CACHE; entries expire according to PYRECRAWL_CACHE_TTL.
UV comes first — it fetches Python itself, so you do not install an interpreter by hand. Browser engines need a one-time pyrecrawl setup that takes around two minutes. Skip it and stealth mode has nothing to run, so Cloudflare-protected sites stay out of reach. Set PYRECRAWL_NO_TELEMETRY=1 to silence the anonymous startup ping.
What it helps with
- turn one web page into markdown for the agent context
- get past Cloudflare on a protected site
- find academic papers on a topic and pull their PDFs
- watch a product page for price changes
- log in once and continue a multi-step browser flow
Example prompts for the agent
- Scrape https://example.com and give me the key points
- Find papers on Rust memory safety and build a summary with citations
- Monitor this page and show me what changed since last time
How to install PyreCrawl
Platform CLI
claude mcp add pyrecrawl -- uvx pyrecrawluvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>
Platform CLI
codex mcp add pyrecrawl -- uvx pyrecrawluvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>
Config file
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": [
"pyrecrawl"
]
}
}
}uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>
Config file
{
"servers": {
"pyrecrawl": {
"type": "stdio",
"command": "uvx",
"args": [
"pyrecrawl"
]
}
}
}uvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>
Platform CLI
gemini mcp add pyrecrawl uvx pyrecrawluvx runs the command named after the package; if the README names another one, use uvx --from pyrecrawl <command>
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about PyreCrawl
How do I install PyreCrawl in Claude Code?
Run claude mcp add pyrecrawl -- uvx pyrecrawl. Then restart the agent or open a new session.
What does PyreCrawl do?
It converts pages to markdown, crawls sites through their links, extracts structured data with a CSS schema, and searches the web and arXiv. Everything runs locally with no API keys.
What does it need to work?
UV to launch it, plus a one-time pyrecrawl setup that installs Chromium and the stealth engines. Reading PDF, DOCX and PPTX files requires the optional [docs] extras.
