# block-ai-scrapers

block-ai-scrapers — a skill that prevents AI agents from scraping your site while preserving SEO indexing. It implements multi-layer request filtering to block unwanted bots.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 565 installs · ★ 1797 repo stars
- Author: illyism
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/Illyism/seo-skills/tree/HEAD/seo/block-ai-scrapers/SKILL.md
- Updated: 2026-11-16
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/block-ai-scrapers

## What it does

This tool builds a defense system that stops AI agents, headless browsers, and scripts like `curl` or `Playwright` from harvesting your content. Instead of serving standard `403` or `429` responses, it returns a gate page with zero dataset rows and directs the user to your official product link.

The architecture includes IP and ASN verification to reliably confirm `Googlebot` while blocking scrapers from cloud data centers. Setup requires integration within your server-side logic, such as middleware in frameworks like Next.js or edge platforms like Cloudflare Workers.

## Requirements
- Works with: Playwright

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Illyism/seo-skills --skill block-ai-scrapers -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/block-ai-scrapers/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/block-ai-scrapers/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Illyism/seo-skills --skill block-ai-scrapers -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/block-ai-scrapers/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Illyism/seo-skills --skill block-ai-scrapers -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/block-ai-scrapers/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Illyism/seo-skills seo/block-ai-scrapers/SKILL.md
```

2. **Manually: put the files in place**
- File: `.github/skills/block-ai-scrapers/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Illyism/seo-skills --skill block-ai-scrapers -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Illyism/seo-skills --path seo/block-ai-scrapers
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/block-ai-scrapers/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Illyism/seo-skills --skill block-ai-scrapers -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Illyism/seo-skills/tree/HEAD/seo/block-ai-scrapers/SKILL.md

## When to use it
- prevent AI agents from scraping website content
- protect API endpoints and database credits
- configure robots.txt to exclude unwanted bots
- maintain search engine indexing for Googlebot

## Example prompts
- Setup the anti-scraping filter using block-ai-scrapers
- Configure my middleware to redirect blocked AI agents
- Update robots.txt rules to block Perplexity and ClaudeBot

## Similar
- [nuxt-seo](https://affpapa.org/en/nevibecode/skills/nuxt-seo.md) — skill · 3300 installs
- [site-audit](https://affpapa.org/en/nevibecode/skills/site-audit.md) — skill · ★ 778 repo stars
- [anti-detect-browser](https://affpapa.org/en/nevibecode/skills/anti-detect-browser.md) — skill · 168800 installs
- [web-scraping](https://affpapa.org/en/nevibecode/skills/web-scraping.md) — skill · 6300 installs
- [safe-browser](https://affpapa.org/en/nevibecode/skills/safe-browser.md) — skill · 1800 installs
- [firecrawl-migration-deep-dive](https://affpapa.org/en/nevibecode/skills/firecrawl-migration-deep-dive.md) — skill · 1 install

From the same set: [illyism/seo-skills](https://affpapa.org/en/nevibecode/by/illyism/seo-skills.md) — 29 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
