web-crawler — Claude Code skill: what it does, how to install
Extracts data from social media, pulls video transcripts, and scrapes pages protected by anti-bot systems. Useful for link audits and failed page fetches.
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a claude-codeWhat web-crawler does
The skill collects content from web pages that block standard requests through Cloudflare challenges or dynamic JavaScript. Built-in functions reside in skills/web-crawler/exports.py and supply proxy credentials automatically. API keys require no manual setup. Running the helper scripts strictly requires python installed in the environment.
Transcript retrieval prioritizes text metadata over heavy media assets. It checks caption tracks, RSS feeds, and player metadata first. Heavy media downloads and speech-to-text models remain on hold until explicitly confirmed, which prevents excessive bandwidth use.
Site auditing uses headless Chromium via the browse helper, tests links concurrently with HEAD requests, and parses XML sitemaps. The tool provides raw observation data. It logs real HTTP statuses, DOM mutations, console exceptions, and broken button interactions.
What it helps with
- extract video captions and transcripts without downloading heavy media files
- bypass Cloudflare blocks and 403 errors during page scraping
- perform link and interactive button audits using Chromium
- extract public profile and post metadata across social platforms
Example prompts for the agent
- Fetch the transcript for this video and summarize the key points
- Audit all links on the site and find buttons that fail on click
- Scrape the article text from this page that returned a 403 error
How to install web-crawler
1Skills CLI (Vercel)
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a claude-code2Manually: put the files in place
~/.claude/skills/web-crawler/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/web-crawler/
1Skills CLI (Vercel)
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a codex2Manually: put the files in place
.agents/skills/web-crawler/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a cursor2Manually: put the files in place
.cursor/skills/web-crawler/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install Starchild-ai-agent/official-skills web-crawler2Manually: put the files in place
.github/skills/web-crawler/SKILL.md1Skills CLI (Vercel)
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a gemini-cli2Platform CLI
gemini skills install https://github.com/Starchild-ai-agent/official-skills --path web-crawlerManually: put the files in place
.devin/skills/web-crawler/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about web-crawler
How to install web-crawler in Claude Code?
Run npx skills add Starchild-ai-agent/official-skills --skill web-crawler -a claude-code. Make sure python is available on your path.
What does the web-crawler skill do?
It extracts data from protected web pages, pulls captions from media links, and audits broken links or broken UI buttons using Chromium.
Are external API keys required for scraping?
No, functions inside exports.py inject proxy credentials automatically. You do not need to set or search for scraping API keys in your environment.
