crawlforge-mcp-server — Claude Code MCP server: scraping and crawling
An MCP server with 31 tools for scraping, crawling and deep research. Hands your agent clean Markdown and structured JSON from any site.
claude mcp add crawlforge-mcp-server -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-serverWhat crawlforge-mcp-server does
Plug it into Claude, Codex, Cursor, GitHub Copilot, Windsurf, Cline, Zed or Gemini CLI and the assistant gains a toolbox: single-page fetches, site-wide crawls, search, Google SERP rank tracking, deep research with source verification, and an autonomous agent that takes a plain-language brief instead of a URL list. Output comes back as Markdown, schema-driven JSON, screenshots, links and metadata.
LLM extraction runs against a local Ollama model by default, so no OpenAI or Anthropic key is involved and nothing leaves your machine. Cloud providers are opt-in. Every request passes SSRF protection, backends sit behind a fail-closed allow-list, browser automation is limited to a vetted action list, and credits are gated per tool.
A CrawlForge API key is required. New accounts start with 1,000 credits and no card; the grant is one-time, not monthly, and credits do not expire. crawlforge login signs you in through the browser and stores the key in ~/.crawlforge/config.json, while crawlforge init registers the server with your clients. Some tools lean on DataForSEO, so pass DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD in the config for those.
What it helps with
- pull an article body into Markdown without nav clutter
- walk a site's internal links and export the page tree
- check where a page ranks in Google results
- extract product fields into JSON against a schema
- crawl behind a login and collect the data
Example prompts for the agent
- Crawl the homepage and list every outbound link you find
- Extract the title, price and description from this product page as JSON
- Research what people say about this library and give me an answer with sources
How to install crawlforge-mcp-server
Platform CLI
claude mcp add crawlforge-mcp-server -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-serverReplace <…> with your own values
Platform CLI
codex mcp add crawlforge-mcp-server --env CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> --env DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> --env DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-serverReplace <…> with your own values
Config file
{
"mcpServers": {
"crawlforge-mcp-server": {
"command": "npx",
"args": [
"-y",
"crawlforge-mcp-server"
],
"env": {
"CRAWLFORGE_API_KEY": "<CRAWLFORGE_API_KEY>",
"DATAFORSEO_LOGIN": "<DATAFORSEO_LOGIN>",
"DATAFORSEO_PASSWORD": "<DATAFORSEO_PASSWORD>"
}
}
}
}Replace <…> with your own values
Config file
{
"servers": {
"crawlforge-mcp-server": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"crawlforge-mcp-server"
],
"env": {
"CRAWLFORGE_API_KEY": "<CRAWLFORGE_API_KEY>",
"DATAFORSEO_LOGIN": "<DATAFORSEO_LOGIN>",
"DATAFORSEO_PASSWORD": "<DATAFORSEO_PASSWORD>"
}
}
}
}Replace <…> with your own values
Platform CLI
gemini mcp add -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> crawlforge-mcp-server npx -y crawlforge-mcp-serverReplace <…> with your own values
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about crawlforge-mcp-server
How do I install crawlforge-mcp-server in Claude Code?
Run `claude mcp add crawlforge-mcp-server -e CRAWLFORGE_API_KEY=<CRAWLFORGE_API_KEY> -e DATAFORSEO_LOGIN=<DATAFORSEO_LOGIN> -e DATAFORSEO_PASSWORD=<DATAFORSEO_PASSWORD> -- npx -y crawlforge-mcp-server`, then restart Claude.
What does crawlforge-mcp-server do?
It exposes 31 tools for scraping, crawling, search and deep research to your agent. Results arrive as Markdown, JSON, screenshots and metadata.
Do I need an LLM API key for extraction?
No. extract_with_llm defaults to a local Ollama model, so no cloud provider key is needed.
