# crawlforge-mcp-server

An MCP server with 31 tools for scraping, crawling and deep research. Hands your agent clean Markdown and structured JSON from any site.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 6
- Author: mysleekdesigns
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/mysleekdesigns/crawlforge-mcp
- Updated: 2026-10-22
- HTML: https://affpapa.org/en/nevibecode/mcp/crawlforge-mcp-server

## What it does

Plug it into Claude, Codex, Cursor, GitHub Copilot, Windsurf, Cline, Zed or Gemini CLI and the assistant gains a toolbox: single-page fetches, site-wide crawls, search, Google SERP rank tracking, deep research with source verification, and an autonomous `agent` that takes a plain-language brief instead of a URL list. Output comes back as Markdown, schema-driven JSON, screenshots, links and metadata.

LLM extraction runs against a local Ollama model by default, so no OpenAI or Anthropic key is involved and nothing leaves your machine. Cloud providers are opt-in. Every request passes SSRF protection, backends sit behind a fail-closed allow-list, browser automation is limited to a vetted action list, and credits are gated per tool.

A CrawlForge API key is required. New accounts start with 1,000 credits and no card; the grant is one-time, not monthly, and credits do not expire. `crawlforge login` signs you in through the browser and stores the key in `~/.crawlforge/config.json`, while `crawlforge init` registers the server with your clients. Some tools lean on DataForSEO, so pass `DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` in the config for those.

## Requirements
- Needs: Node.js (commands run via npx)
- Environment variables: `CRAWLFORGE_API_KEY`, `DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add crawlforge-mcp-server -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-server
```
Replace <…> with your own values

### Codex

**Platform CLI**
```bash
codex mcp add crawlforge-mcp-server --env CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> --env DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> --env DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-server
```
Replace <…> with your own values

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "crawlforge-mcp-server": {
            "command": "npx",
            "args": [
                "-y",
                "crawlforge-mcp-server"
            ],
            "env": {
                "CRAWLFORGE_API_KEY": "<CRAWLFORGE_API_KEY>",
                "DATAFORSEO_LOGIN": "<DATAFORSEO_LOGIN>",
                "DATAFORSEO_PASSWORD": "<DATAFORSEO_PASSWORD>"
            }
        }
    }
}
```
Replace <…> with your own values

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "crawlforge-mcp-server": {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "crawlforge-mcp-server"
            ],
            "env": {
                "CRAWLFORGE_API_KEY": "<CRAWLFORGE_API_KEY>",
                "DATAFORSEO_LOGIN": "<DATAFORSEO_LOGIN>",
                "DATAFORSEO_PASSWORD": "<DATAFORSEO_PASSWORD>"
            }
        }
    }
}
```
Replace <…> with your own values

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> crawlforge-mcp-server npx -y crawlforge-mcp-server
```
Replace <…> with your own values

## When to use it
- pull an article body into Markdown without nav clutter
- walk a site's internal links and export the page tree
- check where a page ranks in Google results
- extract product fields into JSON against a schema
- crawl behind a login and collect the data

## Example prompts
- Crawl the homepage and list every outbound link you find
- Extract the title, price and description from this product page as JSON
- Research what people say about this library and give me an answer with sources

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md