Skip to main content
MCP server CX Codex

Deepcrawl — Claude Code MCP server: what it does, how to install

Deepcrawl extracts clean markdown from web pages, crawls internal links, and mirrors full sites to disk without requiring external API keys.

MIT updated 2026-10-28

Open on GitHub

What Deepcrawl does

This server provides four tools to inspect web pages: single-page scraping, sitemap mapping, crawling up to 100 pages, and cloning pages to a local folder. During a clone, it captures stylesheets, images, scripts, and fonts, rewriting links to relative paths so the page opens offline.

Static markup is parsed immediately. For JavaScript-heavy single-page applications, you must install Playwright and Chromium on your machine, or dynamic content will not render.

What it helps with

  • extract clean markdown from a page without headers, footers, or ads
  • mirror a target website with all assets to a local folder
  • discover site URLs by reading sitemap.xml and homepage links
  • crawl internal links across documentation for RAG pipelines

Example prompts for the agent

  • Scrape https://example.com and give me the text in clean markdown
  • Clone https://competitor.com to a local folder with assets
  • Discover all URLs on https://docs.example.com using sitemap

How to install Deepcrawl

Install

The MCP registry has no launch command for it: see the repository README

Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more

Questions about Deepcrawl

How to install Deepcrawl in Claude Code?

Add it to your mcpServers configuration with command npx and arguments ["-y", "deepcrawl-mcp@latest"]. The server supervises background updates every 6 hours through npm.

What does Deepcrawl do?

It converts web pages into markdown, follows internal links across domains, and downloads complete HTML layouts with assets. It runs locally without API tokens or subscription fees.

Is Playwright required for Deepcrawl?

Static websites work out of the box. You only need Playwright if you scrape JavaScript-rendered SPAs with the jsRender: true flag enabled.

Similar MCP servers

All MCP servers for Browser and scraping →

Project sponsors

Companies that keep this analytics open