ShadowCrawl — Claude Code MCP server: scraping and search
Gives your agent web search and page extraction, including sites behind Cloudflare, DataDome and LinkedIn.
claude mcp add --transport http shadow-crawl https://shadowcrawl--devshero.run.toolsWhat ShadowCrawl does
ShadowCrawl is a remote MCP server exposing web_search, a unified web_fetch with single, batch and crawl modes, and extract_fields for structured extraction. There is also deep_research for multi-hop search plus scraping and synthesis, and memory_search for past research.
Anti-bot handling runs through Chromium CDP rendering, proxy rotation and block-aware retries. When a page throws a CAPTCHA or an OAuth login, the HITL tools take over: visual_scout and hitl_web_fetch with challenge and auth modes open a visible window, you solve it once, and the session is kept in saved cookies.
Browser automation lives in one stateful tool, scout_browser_automate. The agent passes an array of steps instead of coordinating dozens of separate calls, and the runtime covers navigation, locators, assertions, tabs, screenshots and PDF, route mocking, tracing, and cookie or storage operations against the profile at ~/.cortex-scout/agent_profile. Because the server is hosted, there is nothing to build locally — you only need the endpoint https://shadowcrawl--devshero.run.tools in your MCP config.
What it helps with
- pull data from a site behind Cloudflare or DataDome
- find pages for a query and grab their content in one call
- solve a CAPTCHA or OAuth once and keep the session
- extract structured fields from HTML without writing a parser
- run multi-hop research across several sources
Example prompts for the agent
- Search for articles about MCP servers and pull their titles and dates
- Fetch this Cloudflare-protected page and return the article text
- Open the page in the agent browser, solve the CAPTCHA and save the session
How to install ShadowCrawl
Platform CLI: remote server
claude mcp add --transport http shadow-crawl https://shadowcrawl--devshero.run.toolsPlatform CLI: remote server
codex mcp add shadowcrawl --url https://shadowcrawl--devshero.run.toolsConfig file
{
"mcpServers": {
"shadowcrawl": {
"url": "https://shadowcrawl--devshero.run.tools"
}
}
}Config file
{
"servers": {
"shadowcrawl": {
"type": "http",
"url": "https://shadowcrawl--devshero.run.tools"
}
}
}Platform CLI: remote server
gemini mcp add -t http shadowcrawl https://shadowcrawl--devshero.run.toolsStep by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about ShadowCrawl
How do I install ShadowCrawl in Claude Code?
Add the server with claude mcp add --transport http shadowcrawl https://shadowcrawl--devshero.run.tools.
What does ShadowCrawl do?
It gives the agent search, fetching and crawling, structured field extraction and browser automation. It also handles anti-bot protection and hands CAPTCHAs back to you.
How is its browser automation different from Playwright MCP?
The difference is packaging rather than capability: every action goes through one scout_browser_automate tool with a step array, so the agent spends fewer turns on a multi-step flow.
