# docs-scraper

docs-scraper — a subagent that fetches a documentation page by URL and saves it as a markdown file in ai_docs/. Handy when your agent needs a local copy of the docs.

- subagent · `type=agent` · Claude Code, Cursor, Codex · ★ 1573 repo stars
- Author: disler
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/disler/claude-code-hooks-multi-agent-observability/tree/HEAD/.claude/agents/docs-scraper.md
- Updated: 2026-02-08
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/agents/docs-scraper

## What it does

Give it a documentation URL and it writes a markdown file into `ai_docs/`. The filename is built in kebab-case from the URL path or the page title. Navigation and other site chrome get stripped, code samples and tables stay as they were, and the text itself is left unedited.

Fetching goes through `mcp__firecrawl-mcp__firecrawl_scrape` first, asking for markdown output. Without a Firecrawl MCP server connected, the agent falls back to `WebFetch`. It also has `Write` and `Edit`, and the frontmatter pins the model to `haiku`.

After saving, it checks that the whole page made it into the file rather than a digest. Then it reports whether the run worked and where the file landed, with the source URL. For Cursor, the file goes to `.cursor/agents/docs-scraper.md`.

## Install

### Claude Code (native platform)

**Manually: put the files in place**
- File: `.claude/agents/docs-scraper.md` — For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

### Codex

**Manually: put the files in place**
- File: `.codex/agents/docs-scraper.toml` — Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

### Cursor

**Manually: put the files in place**
- File: `.cursor/agents/docs-scraper.md` — Cursor also reads .claude/agents/

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/disler/claude-code-hooks-multi-agent-observability/tree/HEAD/.claude/agents/docs-scraper.md

## When to use it
- save a library's documentation page into `ai_docs/` for offline reading
- get clean markdown without site menus and navigation
- hand another agent a local docs file instead of a link
- store docs pages as files with readable kebab-case names

## Example prompts
- Use the `docs-scraper` subagent to save the docs at this link into `ai_docs/`
- Fetch this docs page with `docs-scraper` and write it as a markdown file in `ai_docs/`
- Run `docs-scraper` on the API reference page and confirm the full text was saved

## Similar
- [feishu-doc-scraper](https://affpapa.org/en/nevibecode/skills/feishu-doc-scraper.md) — skill · 683 installs
- [firecrawl-knowledge-ingest](https://affpapa.org/en/nevibecode/skills/firecrawl-knowledge-ingest.md) — skill · 31600 installs
- [doc-scraper](https://affpapa.org/en/nevibecode/mcp/doc-scraper.md) — MCP server · ★ 110 repo stars
- [firecrawl-scraper](https://affpapa.org/en/nevibecode/skills/firecrawl-scraper.md) — skill · 407 installs
- [wps-doc-scraper](https://affpapa.org/en/nevibecode/skills/wps-doc-scraper.md) — skill · 398 installs
- [web-scraper](https://affpapa.org/en/nevibecode/skills/web-scraper.md) — skill · 232 installs

From the same set: [disler/claude-code-hooks-multi-agent-observability](https://affpapa.org/en/nevibecode/by/disler/claude-code-hooks-multi-agent-observability.md) — 20 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
