Skip to main content
subagent CX Codex

docs-scraper — Claude Code subagent: what it does, how to install

docs-scraper — a subagent that fetches a documentation page by URL and saves it as a markdown file in ai_docs/. Handy when your agent needs a local copy of the docs.

no license ★ 1,573 updated 2026-02-08

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/docs-scraper.md
Open on GitHub

What docs-scraper does

Give it a documentation URL and it writes a markdown file into ai_docs/. The filename is built in kebab-case from the URL path or the page title. Navigation and other site chrome get stripped, code samples and tables stay as they were, and the text itself is left unedited.

Fetching goes through mcp__firecrawl-mcp__firecrawl_scrape first, asking for markdown output. Without a Firecrawl MCP server connected, the agent falls back to WebFetch. It also has Write and Edit, and the frontmatter pins the model to haiku.

After saving, it checks that the whole page made it into the file rather than a digest. Then it reports whether the run worked and where the file landed, with the source URL. For Cursor, the file goes to .cursor/agents/docs-scraper.md.

What it helps with

  • save a library's documentation page into ai_docs/ for offline reading
  • get clean markdown without site menus and navigation
  • hand another agent a local docs file instead of a link
  • store docs pages as files with readable kebab-case names

Example prompts for the agent

  • Use the docs-scraper subagent to save the docs at this link into ai_docs/
  • Fetch this docs page with docs-scraper and write it as a markdown file in ai_docs/
  • Run docs-scraper on the API reference page and confirm the full text was saved

How to install docs-scraper

Manually: put the files in place

.claude/agents/docs-scraper.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about docs-scraper

How do I install docs-scraper in Claude Code?

Save the subagent file as .claude/agents/docs-scraper.md.

What does docs-scraper do?

It fetches a documentation page by URL, strips the site navigation and writes the text to a markdown file in ai_docs/. The documentation content itself is not rewritten.

Which tools does docs-scraper use?

mcp__firecrawl-mcp__firecrawl_scrape is the main one, with WebFetch as the fallback. It writes files through Write and Edit.

Similar subagents

All subagents for Browser and scraping →

From the same set20

The whole set →
fetch-docs-sonnet45 The fetch-docs-sonnet45 subagent retrieves documentation from a URL and writes it to a chosen file. Use it for documentation retrieval benchmarks. CX Codex ★ 1,573repo stars load_ai_docs load_ai_docs — a command that downloads documentation from AI platforms and saves it as markdown files for agents to reference locally. Use it when you want your sub-agents to have current docs without needing web access. CX Codex +1 ★ 1,573repo stars find_and_summarize find_and_summarize locates source files across target directories and creates concise two-sentence summaries in a YAML file. Useful for documenting modules without subagents. CX Codex +1 ★ 1,573repo stars create-worktree-subagent The create-worktree-subagent subagent creates isolated git worktrees for parallel development. Use it when the request explicitly delegates worktree creation to a subagent. CX Codex ★ 1,573repo stars quick-plan quick-plan turns requirements into structured implementation blueprints stored in the specs/ folder. Useful before starting complex refactoring or new feature work. CX Codex +1 ★ 1,573repo stars plan_new_feature plan_new_feature is a Claude Code command that turns a feature request into a development plan saved under specs/. It maps the work before coding starts. CX Codex +1 ★ 1,573repo stars load_ai_docs load_ai_docs — a command that downloads documentation from specified URLs and saves it as markdown files via model-specific subagents. Designed for documentation benchmarking and testing AI model capabilities. CX Codex +1 ★ 1,573repo stars t_metaprompt_workflow t_metaprompt_workflow generates structured Claude Code slash commands from a high-level task description while checking Anthropic docs. CX Codex +1 ★ 1,573repo stars

Project sponsors

Companies that keep this analytics open