docs-scraper — Claude Code subagent: what it does, how to install
docs-scraper — a subagent that fetches a documentation page by URL and saves it as a markdown file in ai_docs/. Handy when your agent needs a local copy of the docs.
no license ★ 1,573 updated 2026-02-08
.claude/agents/docs-scraper.mdWhat docs-scraper does
Give it a documentation URL and it writes a markdown file into ai_docs/. The filename is built in kebab-case from the URL path or the page title. Navigation and other site chrome get stripped, code samples and tables stay as they were, and the text itself is left unedited.
Fetching goes through mcp__firecrawl-mcp__firecrawl_scrape first, asking for markdown output. Without a Firecrawl MCP server connected, the agent falls back to WebFetch. It also has Write and Edit, and the frontmatter pins the model to haiku.
After saving, it checks that the whole page made it into the file rather than a digest. Then it reports whether the run worked and where the file landed, with the source URL. For Cursor, the file goes to .cursor/agents/docs-scraper.md.
What it helps with
- save a library's documentation page into
ai_docs/for offline reading - get clean markdown without site menus and navigation
- hand another agent a local docs file instead of a link
- store docs pages as files with readable kebab-case names
Example prompts for the agent
- Use the
docs-scrapersubagent to save the docs at this link intoai_docs/ - Fetch this docs page with
docs-scraperand write it as a markdown file inai_docs/ - Run
docs-scraperon the API reference page and confirm the full text was saved
How to install docs-scraper
Manually: put the files in place
.claude/agents/docs-scraper.mdFor all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention
Manually: put the files in place
.codex/agents/docs-scraper.tomlCodex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/
Manually: put the files in place
.cursor/agents/docs-scraper.mdCursor also reads .claude/agents/
Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean
Questions about docs-scraper
How do I install docs-scraper in Claude Code?
Save the subagent file as .claude/agents/docs-scraper.md.
What does docs-scraper do?
It fetches a documentation page by URL, strips the site navigation and writes the text to a markdown file in ai_docs/. The documentation content itself is not rewritten.
Which tools does docs-scraper use?
mcp__firecrawl-mcp__firecrawl_scrape is the main one, with WebFetch as the fallback. It writes files through Write and Edit.