Skip to main content
affpapa.org
RU EN
skill CX Codex

web-scraping · mindrally — Claude Code skill: Python scraping rules

A skill that tells your agent which Python library to reach for on a given site, and how to keep the crawl from breaking or getting blocked.

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓ 
npx skills add Mindrally/skills --skill web-scraping -a claude-code
Open on GitHub

What web-scraping does

This skill turns the agent into a scraping assistant for Python projects. It maps the task to the tool: requests and BeautifulSoup for static pages, lxml when parsing speed matters, Selenium or Playwright for pages rendered by JavaScript, Scrapy for structured crawls. Large extractions point to firecrawl and jina, complex automation to agentQL and multion.

It also covers behaviour. Rate limiting and delays, respecting robots.txt, sensible user agents, retry logic, graceful error handling. The failure cases get their own section: network timeouts, blocked requests, session cookies, pagination.

Then there is the data side — cleaning and validating what you pulled, encoding problems, deduplication, storing results without waste. Nothing extra to install; the skill works with the libraries you already bring into the project. If your codebase already has a retry strategy of its own, the agent will nudge it toward the rules described here, so it is worth comparing the two before you let it rewrite anything.

What it helps with

  • pick a library for a JavaScript-rendered page
  • add delays so the crawl stops hammering the server
  • walk paginated results without dropping rows
  • fix broken encoding in scraped text
  • deduplicate records before loading them into a database

Example prompts for the agent

  • Scrape the product names and prices from this catalog, pages follow ?page=
  • This site renders its cards with JavaScript — what should I use?
  • Add retry on timeout and a delay between pages to my scraper

How to install web-scraping

1Skills CLI (Vercel)

npx skills add Mindrally/skills --skill web-scraping -a claude-code

2Manually: put the files in place

~/.claude/skills/web-scraping/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/web-scraping/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about web-scraping

How do I install web-scraping in Claude?

Run npx skills add Mindrally/skills --skill web-scraping -a claude-code. The skill file then sits at ~/.claude/skills/web-scraping/SKILL.md.

What does web-scraping do?

It gives the agent a set of Python scraping rules: which library fits static pages, dynamic pages and large crawls, how to handle errors, and what to do with the extracted data.

Which libraries does it recommend?

requests, BeautifulSoup and lxml for static pages; Selenium, Playwright and pyppeteer for JavaScript; Scrapy, firecrawl and jina for large crawls; agentQL and multion for complex automation.

Source text

SKILL.md · Apache-2.0 
---
name: web-scraping
description: Expert in web scraping and data extraction with Python tools
metadata:
  maintainer: Mindrally
  source: https://github.com/Mindrally/skills
---

# Web Scraping

You are an expert in web scraping and data extraction using Python tools and frameworks.

## Core Tools

### Static Sites
- Use requests for HTTP requests
- Use BeautifulSoup for HTML parsing
- Use lxml for fast XML/HTML processing

### Dynamic Content
- Use Selenium for JavaScript-rendered pages
- Use Playwright for modern web automation
- Use Puppeteer (via pyppeteer) for headless browsing

### Large-Scale Extraction
- Use Scrapy for structured crawling
- Use jina for AI-powered extraction
- Use firecrawl for large-scale scraping

### Complex Workflows
- Use agentQL for structured queries
- Use multion for complex automation

## Best Practices

- Implement rate limiting and delays
- Respect robots.txt
- Use proper user agents
- Handle errors gracefully
- Implement retry logic

## Error Handling

- Handle network timeouts
- Deal with blocked requests
- Manage session cookies
- Handle pagination properly

## Ethical Considerations

- Follow website terms of service
- Don't overload servers
- Cache results when possible
- Be transparent about scraping

## Data Processing

- Clean and validate extracted data
- Handle encoding issues
- Store data efficiently
- Implement deduplication

© Mindrally · Apache-2.0 · Mindrally/skills

Similar skills

All skills for Browser and scraping →

web-scraping by other authors comes first.

web-scraping jamditis Defines patterns for ethical scraping, access-failure handling, and cascading fallbacks across multiple extraction libraries. CX Codex +1 6,100installs universal-scraping-architect Mehansh Barthwal Designs data extraction pipelines using Firecrawl or local Python scripts, validating output JSON before saving. It helps handle scraping with automated quality checks. 35installs browser-automation Alireza Rezvani A Claude Code skill that writes Playwright scripts for scraping, form filling, screenshots and saved sessions, plus the stealth settings that keep headless browsers from being blocked. 636installs extract actionbook Extracts structured data from websites, outputs JSON or CSV, and writes a standalone Playwright script. CX Codex +1 2,300installs playwright-stealth-verify liarjsdev Checks whether an automated browser exposes a coherent fingerprint. Use it to measure headless settings or stealth-plugin changes in a test harness. CX Codex +1 102,500installs adspower-antidetect-browser reason-machines A skill that gives your agent ready calls for creating, launching and stopping AdsPower browser profiles over the local API. CX Codex +1 821installs e2e-testing-automation aj-geddes A skill that helps you write end-to-end tests: browser flows, user journeys and frontend-backend integration checks. CX Codex +1 689installs

From the same set168

fastapi-python Tells the agent how to write FastAPI code: async I/O, Pydantic v2 models, early returns and router layout. Useful when you build a Python API. CX Codex +1 13,900installs computer-vision-opencv A skill that keeps your agent inside the conventions of OpenCV and PyTorch work: filters, feature matching, video pipelines, tracking. Useful when the repo already deals with frames. CX Codex +1 3,800installs framer-motion Tells your agent which properties to animate in Motion and which ones wreck layout. Useful when the project already runs React and TypeScript. CX Codex +1 3,000installs chrome-extension-development Walks your agent through Manifest V3 extension work: manifest, service worker, content scripts, popup UI, and Chrome Web Store publishing. CX Codex +1 2,800installs mysql-best-practices A skill that gives the agent rules for MySQL schema design, indexing and query optimization. Useful when it writes migrations or debugs a slow query. CX Codex +1 2,700installs odoo-development A skill that turns your AI agent into an Odoo developer: Python ORM models, XML views, access rules, module layout. CX Codex +1 2,300installs expo-react-native-typescript A skill that hands your agent a rulebook for Expo, React Native and TypeScript work: file layout, navigation, state, testing. CX Codex +1 2,200installs redis-best-practices Redis guidelines and patterns for Claude Code. Helps configure data structures, caching strategies, and atomic operations. CX Codex +1 2,200installs

Project sponsors

Companies that keep this analytics open
Gold sponsor
Nutra.Club

Impressions, clicks and code copies — open statistics. We keep the numbers public for the same reason NeBlask itself is public. Become a sponsor