doc-scraper — Claude Code MCP server: features and setup
Crawl documentation sites into local Markdown that an agent can search and read offline. Useful when you want answers grounded in a chosen docs corpus rather than a live web search.
Open on GitHubWhat doc-scraper does
doc-scraper crawls documentation pages and stores them as Markdown. Through MCP, an agent can run BM25 searches against the local corpus or read a page directly, without network access. It can also revisit a site and show how the new crawl differs from an earlier one.
Run doc-scraper add to identify a documentation generator, check the content selector, and draft a configuration. The tool recognizes more than 30 generators. JavaScript-only pages are reported instead of being saved as empty content. Crawls respect robots.txt and sitemaps, apply per-host request limits, and can resume after interruption. A fresh crawl still needs access to the source site; offline access applies to content already stored locally.
You can install a release binary or use Go, then expose the MCP server to Claude Code over stdio. Its setup command takes the executable and configuration paths. Crawl output and state stay on your machine.
What it helps with
- save a documentation site as local Markdown pages
- search stored docs without making a web query
- read a chosen page through MCP
- compare a new crawl with an earlier one
- refresh a local docs corpus after site changes
Example prompts for the agent
- Add the Rust CLI Book docs as
rust_cli_bookand prepare its configuration. - Crawl the documentation site named
rust_cli_bookinto the local corpus. - Search
rust_cli_bookfor error-handling guidance and read the relevant page.
How to install doc-scraper
Install
The MCP registry has no launch command for it: see the repository README
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about doc-scraper
How do I install doc-scraper for Claude Code?
Download a release binary for your system or install it with go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest. Then connect the MCP server with claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yaml, replacing the paths with your executable and configuration locations.
What does doc-scraper do?
It turns documentation sites into local Markdown and gives agents MCP tools to search and read those saved pages. Searches run against the local corpus without network access.
Which documentation generators can doc-scraper detect?
It recognizes more than 30 generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook. It checks the content selector and can fall back to Readability for extraction.
