# Deepcrawl

Deepcrawl extracts clean markdown from web pages, crawls internal links, and mirrors full sites to disk without requiring external API keys.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI
- Author: Soflutionltd
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/Soflutionltd/deepcrawl
- Updated: 2026-10-28
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/mcp/deepcrawl

## What it does

This server provides four tools to inspect web pages: single-page scraping, sitemap mapping, crawling up to 100 pages, and cloning pages to a local folder. During a clone, it captures stylesheets, images, scripts, and fonts, rewriting links to relative paths so the page opens offline.

Static markup is parsed immediately. For JavaScript-heavy single-page applications, you must install Playwright and Chromium on your machine, or dynamic content will not render.

## Install

### Claude Code (native platform)

**Install**
- The MCP registry has no launch command for it: see the repository README

## When to use it
- extract clean markdown from a page without headers, footers, or ads
- mirror a target website with all assets to a local folder
- discover site URLs by reading `sitemap.xml` and homepage links
- crawl internal links across documentation for RAG pipelines

## Example prompts
- Scrape `https://example.com` and give me the text in clean markdown
- Clone `https://competitor.com` to a local folder with assets
- Discover all URLs on `https://docs.example.com` using sitemap

## Similar
- [tavily-crawl](https://affpapa.org/en/nevibecode/skills/tavily-crawl.md) — skill · 17900 installs
- [Firecrawl MCP Server](https://affpapa.org/en/nevibecode/mcp/firecrawl-mcp-server.md) — MCP server · ★ 7877 repo stars
- [WebReaper](https://affpapa.org/en/nevibecode/mcp/webreaper.md) — MCP server · ★ 153 repo stars
- [Nstdata Crawl](https://affpapa.org/en/nevibecode/mcp/nstdata-crawl.md) — MCP server · ★ 3 repo stars
- [Atlas Web Compiler & Scraper](https://affpapa.org/en/nevibecode/mcp/atlas-web-compiler-scraper.md) — MCP server · ★ 1 repo stars
- [Firecrawl](https://affpapa.org/en/nevibecode/mcp/firecrawl.md) — MCP server

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
