# doc-scraper

Crawl documentation sites into local Markdown that an agent can search and read offline. Useful when you want answers grounded in a chosen docs corpus rather than a live web search.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed · ★ 107
- Author: Sriram-PR
- License: Apache-2.0 · policy=link-only · description only; the file stays with the author
- Source: https://github.com/Sriram-PR/doc-scraper
- Updated: 2026-10-22
- HTML: https://affpapa.org/en/nevibecode/mcp/doc-scraper

## What it does

doc-scraper crawls documentation pages and stores them as Markdown. Through MCP, an agent can run BM25 searches against the local corpus or read a page directly, without network access. It can also revisit a site and show how the new crawl differs from an earlier one.

Run `doc-scraper add` to identify a documentation generator, check the content selector, and draft a configuration. The tool recognizes more than 30 generators. JavaScript-only pages are reported instead of being saved as empty content. Crawls respect `robots.txt` and sitemaps, apply per-host request limits, and can resume after interruption. A fresh crawl still needs access to the source site; offline access applies to content already stored locally.

You can install a release binary or use Go, then expose the MCP server to Claude Code over `stdio`. Its setup command takes the executable and configuration paths. Crawl output and state stay on your machine.

## Install

### Claude Code (native platform)

**Install**
- The MCP registry has no launch command for it: see the repository README

## When to use it
- save a documentation site as local Markdown pages
- search stored docs without making a web query
- read a chosen page through MCP
- compare a new crawl with an earlier one
- refresh a local docs corpus after site changes

## Example prompts
- Add the Rust CLI Book docs as `rust_cli_book` and prepare its configuration.
- Crawl the documentation site named `rust_cli_book` into the local corpus.
- Search `rust_cli_book` for error-handling guidance and read the relevant page.

## Similar
- [Reactive Resume](https://affpapa.org/en/nevibecode/mcp/reactive-resume.md) — MCP server · ★ 44874
- [Sui Knowledge Base (kapa.ai)](https://affpapa.org/en/nevibecode/mcp/sui-knowledge-base-kapaai.md) — MCP server · ★ 7784
- [Claude Code Ultimate Guide](https://affpapa.org/en/nevibecode/mcp/claude-code-ultimate-guide.md) — MCP server · ★ 6237
- [Zotero MCP](https://affpapa.org/en/nevibecode/mcp/zotero-mcp.md) — MCP server · ★ 5548
- [RevoGrid DataGrid MCP Pro](https://affpapa.org/en/nevibecode/mcp/revogrid-datagrid-mcp-pro.md) — MCP server · ★ 3450
- [Claude Code Explorer MCP](https://affpapa.org/en/nevibecode/mcp/codeaashu-claude-code-explorer-mcp.md) — MCP server · ★ 3409

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md