web-scraper — Claude Code skill: what it does, how to install
web-scraper scrapes web pages and saves them locally as HTML or Markdown along with images using plain HTTP requests without a headless browser.
no license 232 installs ★ 51 updated 2026-02-12
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-codeWhat web-scraper does
The skill runs scripts/scrape.py to extract page content into HTML or Markdown files, downloading embedded graphics into a local images/ directory. It supports fetching single addresses or crawling full hierarchies with configurable link depth (--max-depth) and page caps (--max-pages). Respect for robots.txt and request delays default to active.
Executing the script requires Python 3.8 or newer alongside requests and beautifulsoup4. Because it relies on raw HTTP calls without running a browser engine, sites requiring client-side JavaScript rendering will not yield dynamic content.
What it helps with
- archive web pages as local Markdown with images
- crawl documentation sites recursively into folders
- download clean HTML snapshots without a headless browser
Example prompts for the agent
- Save
https://example.comas Markdown with all images - Crawl
https://docs.example.comrecursively up to depth 2 into~/Downloads/docs - Export
https://example.comas HTML without downloading images
How to install web-scraper
1Skills CLI (Vercel)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code2Manually: put the files in place
~/.claude/skills/web-scraper/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/web-scraper/
1Skills CLI (Vercel)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a codex2Manually: put the files in place
.agents/skills/web-scraper/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a cursor2Manually: put the files in place
.cursor/skills/web-scraper/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install agentbay-ai/agentbay-skills web-scraper2Manually: put the files in place
.github/skills/web-scraper/SKILL.md1Skills CLI (Vercel)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a gemini-cli2Platform CLI
gemini skills install https://github.com/agentbay-ai/agentbay-skills --path web-scraperManually: put the files in place
.devin/skills/web-scraper/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about web-scraper
How to install web-scraper in Claude Code?
Run npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code in your terminal. The definition installs to ~/.claude/skills/web-scraper/SKILL.md.
What does web-scraper do?
It downloads web pages via HTTP and stores them on disk as HTML or Markdown documents while capturing remote images locally.
Does web-scraper require a browser driver?
No browser is needed. The tool relies strictly on Python with requests and beautifulsoup4 for fetching and parsing.