# ShadowCrawl

Gives your agent web search and page extraction, including sites behind Cloudflare, DataDome and LinkedIn.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 70
- Author: cortex-works
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/cortex-works/cortex-scout
- Updated: 2026-09-20
- HTML: https://affpapa.org/en/nevibecode/mcp/shadowcrawl

## What it does

ShadowCrawl is a remote MCP server exposing `web_search`, a unified `web_fetch` with `single`, `batch` and `crawl` modes, and `extract_fields` for structured extraction. There is also `deep_research` for multi-hop search plus scraping and synthesis, and `memory_search` for past research.

Anti-bot handling runs through Chromium CDP rendering, proxy rotation and block-aware retries. When a page throws a CAPTCHA or an OAuth login, the HITL tools take over: `visual_scout` and `hitl_web_fetch` with `challenge` and `auth` modes open a visible window, you solve it once, and the session is kept in saved cookies.

Browser automation lives in one stateful tool, `scout_browser_automate`. The agent passes an array of steps instead of coordinating dozens of separate calls, and the runtime covers navigation, locators, assertions, tabs, screenshots and PDF, route mocking, tracing, and cookie or storage operations against the profile at `~/.cortex-scout/agent_profile`. Because the server is hosted, there is nothing to build locally — you only need the endpoint `https://shadowcrawl--devshero.run.tools` in your MCP config.

## Requirements
- Remote server: `https://shadowcrawl--devshero.run.tools`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http shadow-crawl https://shadowcrawl--devshero.run.tools
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add shadowcrawl --url https://shadowcrawl--devshero.run.tools
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "shadowcrawl": {
            "url": "https://shadowcrawl--devshero.run.tools"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "shadowcrawl": {
            "type": "http",
            "url": "https://shadowcrawl--devshero.run.tools"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http shadowcrawl https://shadowcrawl--devshero.run.tools
```

## When to use it
- pull data from a site behind Cloudflare or DataDome
- find pages for a query and grab their content in one call
- solve a CAPTCHA or OAuth once and keep the session
- extract structured fields from HTML without writing a parser
- run multi-hop research across several sources

## Example prompts
- Search for articles about MCP servers and pull their titles and dates
- Fetch this Cloudflare-protected page and return the article text
- Open the page in the agent browser, solve the CAPTCHA and save the session

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md