Skip to main content
skill CX Codex

web-scraper — Claude Code skill: what it does, how to install

web-scraper scrapes web pages and saves them locally as HTML or Markdown along with images using plain HTTP requests without a headless browser.

no license 232 installs ★ 51 updated 2026-02-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code
Open on GitHub

What web-scraper does

The skill runs scripts/scrape.py to extract page content into HTML or Markdown files, downloading embedded graphics into a local images/ directory. It supports fetching single addresses or crawling full hierarchies with configurable link depth (--max-depth) and page caps (--max-pages). Respect for robots.txt and request delays default to active.

Executing the script requires Python 3.8 or newer alongside requests and beautifulsoup4. Because it relies on raw HTTP calls without running a browser engine, sites requiring client-side JavaScript rendering will not yield dynamic content.

What it helps with

  • archive web pages as local Markdown with images
  • crawl documentation sites recursively into folders
  • download clean HTML snapshots without a headless browser

Example prompts for the agent

  • Save https://example.com as Markdown with all images
  • Crawl https://docs.example.com recursively up to depth 2 into ~/Downloads/docs
  • Export https://example.com as HTML without downloading images

How to install web-scraper

1Skills CLI (Vercel)

npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code

2Manually: put the files in place

~/.claude/skills/web-scraper/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/web-scraper/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about web-scraper

How to install web-scraper in Claude Code?

Run npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code in your terminal. The definition installs to ~/.claude/skills/web-scraper/SKILL.md.

What does web-scraper do?

It downloads web pages via HTTP and stores them on disk as HTML or Markdown documents while capturing remote images locally.

Does web-scraper require a browser driver?

No browser is needed. The tool relies strictly on Python with requests and beautifulsoup4 for fetching and parsing.

Similar skills

All skills for Browser and scraping →

From the same set2

Project sponsors

Companies that keep this analytics open