# scrape

Pulls web page content as Markdown using the Bright Data Web Unlocker API. It navigates anti-bot measures and CAPTCHAs during extraction.

- skill · `type=skill` · Claude Code · 1002 installs
- Author: davila7
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/web-data/scrape
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/scrape

## What it does

The skill triggers `scripts/scrape.sh`, which uses `curl` to send extraction requests to the Web Unlocker endpoint. It processes pages that require JavaScript rendering, bypasses challenges, and outputs formatted text with headings and paragraphs.

Requests will fail immediately without `BRIGHTDATA_API_KEY` and `BRIGHTDATA_UNLOCKER_ZONE` exported in your environment. Both parameters originate from the Bright Data dashboard at `brightdata.com/cp`. Outside of these variables and `curl`, no additional utilities are needed.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill web-data/scrape
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill scrape -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/scrape/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/scrape/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/scrape/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/scrape/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates scrape
```

2. **Manually: put the files in place**
- File: `.github/skills/scrape/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/web-data/scrape
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/scrape/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/web-data/scrape

## When to use it
- extract web article text as formatted Markdown
- bypass automated bot verification and CAPTCHA obstacles
- retrieve content from pages that depend on client JavaScript
- pull product details from a store URL

## Example prompts
- Extract the article at `https://example.com/article` into Markdown
- Run scrape on `https://shop.example.com/product/123` to fetch product details

## Similar
- [agent-browser](https://affpapa.org/en/nevibecode/skills/vercel-labs-agent-browser.md) — skill · 1200000 installs
- [webapp-testing](https://affpapa.org/en/nevibecode/skills/webapp-testing.md) — skill · 186758 installs
- [computer-use](https://affpapa.org/en/nevibecode/skills/computer-use.md) — skill · 239600 installs
- [browser-use](https://affpapa.org/en/nevibecode/skills/browser-use.md) — skill · 98200 installs
- [playwright-cli](https://affpapa.org/en/nevibecode/skills/playwright-cli.md) — skill · 195700 installs
- [last30days](https://affpapa.org/en/nevibecode/skills/last30days.md) — skill · 53000 installs

## Other versions with the same name
- [scrape · brightdata](https://affpapa.org/en/nevibecode/skills/brightdata-scrape.md) — 10800 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 495 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md