# web-scraper

web-scraper scrapes web pages and saves them locally as HTML or Markdown along with images using plain HTTP requests without a headless browser.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 232 installs · ★ 51 repo stars
- Author: agentbay-ai
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/agentbay-ai/agentbay-skills/tree/HEAD/web-scraper/SKILL.md
- Updated: 2026-02-12
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/web-scraper

## What it does

The skill runs `scripts/scrape.py` to extract page content into HTML or Markdown files, downloading embedded graphics into a local `images/` directory. It supports fetching single addresses or crawling full hierarchies with configurable link depth (`--max-depth`) and page caps (`--max-pages`). Respect for `robots.txt` and request delays default to active.

Executing the script requires Python 3.8 or newer alongside `requests` and `beautifulsoup4`. Because it relies on raw HTTP calls without running a browser engine, sites requiring client-side JavaScript rendering will not yield dynamic content.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/web-scraper/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/web-scraper/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/web-scraper/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/web-scraper/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install agentbay-ai/agentbay-skills web-scraper
```

2. **Manually: put the files in place**
- File: `.github/skills/web-scraper/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/agentbay-ai/agentbay-skills --path web-scraper
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/web-scraper/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add agentbay-ai/agentbay-skills --skill web-scraper -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/agentbay-ai/agentbay-skills/tree/HEAD/web-scraper/SKILL.md

## When to use it
- archive web pages as local Markdown with images
- crawl documentation sites recursively into folders
- download clean HTML snapshots without a headless browser

## Example prompts
- Save `https://example.com` as Markdown with all images
- Crawl `https://docs.example.com` recursively up to depth 2 into `~/Downloads/docs`
- Export `https://example.com` as HTML without downloading images

## Similar
- [web-scraping](https://affpapa.org/en/nevibecode/skills/web-scraping.md) — skill · 6300 installs
- [Atlas Web Compiler & Scraper](https://affpapa.org/en/nevibecode/mcp/atlas-web-compiler-scraper.md) — MCP server · ★ 1 repo stars
- [feishu-doc-scraper](https://affpapa.org/en/nevibecode/skills/feishu-doc-scraper.md) — skill · 683 installs
- [wps-doc-scraper](https://affpapa.org/en/nevibecode/skills/wps-doc-scraper.md) — skill · 398 installs
- [page-monitoring](https://affpapa.org/en/nevibecode/skills/page-monitoring.md) — skill · 620 installs
- [docs-scraper](https://affpapa.org/en/nevibecode/agents/docs-scraper.md) — subagent · ★ 1573 repo stars

From the same set: [agentbay-ai/agentbay-skills](https://affpapa.org/en/nevibecode/by/agentbay-ai/agentbay-skills.md) — 2 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
