# web-scraping

The web-scraping skill for Claude Code extracts data from websites into Markdown, CSV, or JSON. It triggers when asked to collect or monitor web content.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf (Devin Desktop), Claude.ai · ★ 285 repo stars
- Author: makushingu
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/makushingu/harness-maxxing/tree/HEAD/skills/web-scraping/SKILL.md
- Updated: 2026-11-17
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/makushingu-web-scraping

## What it does

The skill facilitates data collection from static and JavaScript-rendered pages using three distinct fetchers. Standard pages utilize `Fetcher`, while `DynamicFetcher` processes JavaScript content and `StealthyFetcher` handles sites with browser detection. The environment must have `scrapling` and `scrapling-python` packages installed on the server.

Scripts utilize `css` or `xpath` selectors to target specific elements. For development, inspect the raw HTML by running `scrapling extract get URL page.html` to confirm selectors before finalizing scripts. Results should be stored within the `data/` directory. When automating, the `maxxing-cron` skill executes scripts periodically to compare changes against previous results.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add makushingu/harness-maxxing --skill web-scraping -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/web-scraping/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/web-scraping/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add makushingu/harness-maxxing --skill web-scraping -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/web-scraping/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add makushingu/harness-maxxing --skill web-scraping -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/web-scraping/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install makushingu/harness-maxxing skills/web-scraping
```

2. **Manually: put the files in place**
- File: `.github/skills/web-scraping/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add makushingu/harness-maxxing --skill web-scraping -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/makushingu/harness-maxxing --path skills/web-scraping
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/web-scraping/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/makushingu/harness-maxxing/tree/HEAD/skills/web-scraping/SKILL.md

## When to use it
- Extract product pricing from an online catalog
- Convert raw article text into formatted Markdown
- Monitor webpage updates for price or content changes

## Example prompts
- Scrape data from example.com and save it to `data/products.json`
- Extract content using `StealthyFetcher` for the site example.com
- Set up a monitor to check for changes on example.com and report them

## Similar
- [universal-scraping-architect](https://affpapa.org/en/nevibecode/plugins/universal-scraping-architect.md) — plugin · ★ 30251 repo stars
- [scrapy-web-scraping](https://affpapa.org/en/nevibecode/skills/scrapy-web-scraping.md) — skill · 2100 installs
- [brightdata-web-mcp](https://affpapa.org/en/nevibecode/skills/brightdata-web-mcp.md) — skill · ★ 42059 repo stars
- [scraping-twitter-profiles](https://affpapa.org/en/nevibecode/skills/scraping-twitter-profiles.md) — skill · 930 installs
- [web-crawler](https://affpapa.org/en/nevibecode/skills/web-crawler.md) — skill · 4000 installs
- [Atlas Web Compiler & Scraper](https://affpapa.org/en/nevibecode/mcp/atlas-web-compiler-scraper.md) — MCP server · ★ 1 repo stars

## Other versions with the same name
- [web-scraping · jamditis](https://affpapa.org/en/nevibecode/skills/web-scraping.md) — 6300 installs
- [web-scraping · mindrally](https://affpapa.org/en/nevibecode/skills/mindrally-web-scraping.md) — 4500 installs

From the same set: [makushingu/harness-maxxing](https://affpapa.org/en/nevibecode/by/makushingu/harness-maxxing.md) — 21 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
