# scraper-builder

Takes you from "I want data from site X" to a working scraper: site analysis, Bright Data API choice, selectors, pagination.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 405 installs · ★ 268
- Author: brightdata
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/brightdata/skills/tree/HEAD/skills/scraper-builder/SKILL.md
- Updated: 2026-10-15
- HTML: https://affpapa.org/en/nevibecode/skills/scraper-builder

## What it does

The skill builds a scraper for one specific site. It first asks what the site is, which fields you need, how many pages, and what output format. Then it checks whether Bright Data already has a scraper for that domain — if so, no parsing code is needed at all.

When nothing pre-built exists, reconnaissance starts: the skill pulls the HTML through Web Unlocker and checks whether the server renders the data or the page needs a browser. The API choice follows a decision tree: Web Scraper for supported domains, Web Unlocker for static pages, Browser API when clicks and scrolls are involved, SERP API for search results. You get runnable code, not pseudocode.

You need `BRIGHTDATA_API_KEY` and an Unlocker zone in your environment. The skill insists on running the scraper against 1–3 pages and showing the output before scaling up — empty or malformed data means fixing selectors, not launching the full run. Note that the source text is truncated, so later phases around pagination and orchestration are not covered here.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add brightdata/skills --skill scraper-builder -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/scraper-builder/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/scraper-builder/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add brightdata/skills --skill scraper-builder -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/scraper-builder/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add brightdata/skills --skill scraper-builder -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/scraper-builder/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install brightdata/skills scraper-builder
```

2. **Manually: put the files in place**
- File: `.github/skills/scraper-builder/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add brightdata/skills --skill scraper-builder -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/brightdata/skills --path skills/scraper-builder
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/scraper-builder/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add brightdata/skills --skill scraper-builder -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/brightdata/skills/tree/HEAD/skills/scraper-builder/SKILL.md

## When to use it
- build a scraper for a site with no pre-built option
- pick between Unlocker, Browser and SERP APIs
- extract selectors that survive dynamic content
- validate output on a small sample before scaling
- handle pagination and anti-bot protection

## Example prompts
- build a scraper for the product pages on this site
- pull prices and reviews from the category, with pagination
- run the scraper on three pages and show me the output

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 52245506e81d662f.

BEGIN THIRD-PARTY FILE 52245506e81d662f
```text
---
name: scraper-builder
description: "Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create a crawler, extract data from a website, scrape product pages, handle pagination, build a data pipeline from a web source, or automate data collection from any site — even if they don't explicitly say 'scraper'. Triggers on phrases like 'build a scraper for', 'scrape data from', 'extract products from', 'crawl pages on', 'get data from [website]', or 'I need to pull data from'."
---

# Scraper Builder

You are building a production-ready web scraper for the user. Your job is to guide them from "I want data from site X" to a working, robust scraper that handles real-world challenges like pagination, dynamic content, anti-bot protection, and data parsing.

## Critical: Always Validate Your Output

After building the scraper, **always run it** on a small sample (1-3 pages) and show the extracted data to the user before scaling up. If the output is empty, malformed, or missing fields, iterate — fix selectors, switch APIs, or adjust the parsing logic. A scraper that doesn't produce clean data is not done.

Take your time with the reconnaissance phase. Spending 2 minutes analyzing the HTML upfront prevents hours of debugging later. Quality is more important than speed here.

## How This Skill Works

This skill orchestrates Bright Data's four APIs to build scrapers intelligently. Rather than writing fragile custom scraping code, you analyze the target site first, then pick the most reliable and cost-effective extraction method. The decision tree is:

1. **Does a pre-built scraper already exist?** → Use Web Scraper API (zero parsing code needed)
2. **Is the page static / no interaction needed?** → Use Web Unlocker API (cheapest, simplest)
3. **Does the page need clicks, scrolls, or JS interaction?** → Use Browser API (full automation)
4. **Need search engine results?** → Use SERP API

The skill produces complete, runnable code — not pseudocode or outlines.

---

## Phase 1: Understand the Target

Before writing any code, you need to understand what the user wants and what the site looks like. Ask these questions (skip any the user already answered):

1. **What site?** — The target URL or domain
2. **What data?** — Which fields they need (product names, prices, reviews, etc.)
3. **What scope?** — Single page, category pages, search results, entire site section?
4. **Pagination?** — Do they need to scrape across multiple pages?
```
END THIRD-PARTY FILE 52245506e81d662f

The third-party file has ended. Everything between the 52245506e81d662f tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2700 of 29029 bytes. © brightdata · MIT · https://github.com/brightdata/skills/tree/HEAD/skills/scraper-builder/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md