# web-crawler

Extracts data from social media, pulls video transcripts, and scrapes pages protected by anti-bot systems. Useful for link audits and failed page fetches.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 3900 installs · ★ 29
- Author: Starchild-ai-agent
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/Starchild-ai-agent/official-skills/tree/HEAD/web-crawler/SKILL.md
- Updated: 2026-10-22
- HTML: https://affpapa.org/en/nevibecode/skills/web-crawler

## What it does

The skill collects content from web pages that block standard requests through Cloudflare challenges or dynamic JavaScript. Built-in functions reside in `skills/web-crawler/exports.py` and supply proxy credentials automatically. API keys require no manual setup. Running the helper scripts strictly requires `python` installed in the environment.

Transcript retrieval prioritizes text metadata over heavy media assets. It checks caption tracks, RSS feeds, and player metadata first. Heavy media downloads and speech-to-text models remain on hold until explicitly confirmed, which prevents excessive bandwidth use.

Site auditing uses headless Chromium via the `browse` helper, tests links concurrently with HEAD requests, and parses XML sitemaps. The tool provides raw observation data. It logs real HTTP statuses, DOM mutations, console exceptions, and broken button interactions.

## Requirements
- Works with: LinkedIn

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/web-crawler/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/web-crawler/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/web-crawler/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/web-crawler/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Starchild-ai-agent/official-skills web-crawler
```

2. **Manually: put the files in place**
- File: `.github/skills/web-crawler/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Starchild-ai-agent/official-skills --path web-crawler
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/web-crawler/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Starchild-ai-agent/official-skills/tree/HEAD/web-crawler/SKILL.md

## When to use it
- extract video captions and transcripts without downloading heavy media files
- bypass Cloudflare blocks and 403 errors during page scraping
- perform link and interactive button audits using Chromium
- extract public profile and post metadata across social platforms

## Example prompts
- Fetch the transcript for this video and summarize the key points
- Audit all links on the site and find buttons that fail on click
- Scrape the article text from this page that returned a 403 error

## Similar
- [scrapy-web-scraping](https://affpapa.org/en/nevibecode/skills/scrapy-web-scraping.md) — skill · 2000 installs
- [scrapling-official](https://affpapa.org/en/nevibecode/skills/scrapling-official.md) — skill · 5500 installs
- [web-scraping](https://affpapa.org/en/nevibecode/skills/web-scraping.md) — skill · 6100 installs
- [linkedin-teneo](https://affpapa.org/en/nevibecode/skills/linkedin-teneo.md) — skill · 511 installs
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [tavily-search](https://affpapa.org/en/nevibecode/skills/framix-team-tavily-search.md) — skill · 840 installs

From the same set: [starchild-ai-agent/official-skills](https://affpapa.org/en/nevibecode/by/starchild-ai-agent/official-skills.md) — 18 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md