agent-ready-ai-rules — Claude Code skill: AI bot rules in robots.txt
agent-ready-ai-rules is a skill that adds RFC 9309 rules for AI search and training crawlers to robots.txt, controlling page visibility in AI answers and use for model training.
Apache-2.0 ★ 141 updated 2026-10-29
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a claude-codeWhat agent-ready-ai-rules does
The skill configures User-agent entries for different crawler types. Search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot determine visibility in AI answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot) control dataset inclusion—blocking them does not affect search results but prevents use for model training.
Cloudflare users can manage these rules through the AI Crawl Control dashboard without editing the file directly. Validation runs via the isitagentready.com API: POST to https://isitagentready.com/api/scan and check that checks.botAccessControl.robotsTxtAiRules.status returns "pass".
Installs with a single command depending on your IDE. After setup, the skill file appears in your platform's skills folder—open it and follow the steps.
What it helps with
- prevent AI crawlers from indexing your site for search
- block training bots from using content for model training
- allow some bot types while blocking others
- validate robots.txt against RFC 9309
- manage bot rules through a UI instead of file editing
Example prompts for the agent
- Set up robots.txt to block ClaudeBot from training while allowing Claude-SearchBot to index pages
- Add AI search crawler rules to robots.txt per RFC 9309
- Check if GPTBot is correctly blocked in my robots.txt
How to install agent-ready-ai-rules
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a claude-code2Manually: put the files in place
~/.claude/skills/ai-rules/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/ai-rules/
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a codex2Manually: put the files in place
.agents/skills/ai-rules/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a cursor2Manually: put the files in place
.cursor/skills/ai-rules/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install fabricioctelles/skills skills/agent-ready-cloudflare/ai-rules2Manually: put the files in place
.github/skills/ai-rules/SKILL.md1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a gemini-cli2Platform CLI
gemini skills install https://github.com/fabricioctelles/skills --path skills/agent-ready-cloudflare/ai-rulesManually: put the files in place
.devin/skills/ai-rules/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about agent-ready-ai-rules
How do I install agent-ready-ai-rules in Claude Code?
Run npx skills add fabricioctelles/skills --skill ai-rules -a claude-code, then open ~/.claude/skills/ai-rules/SKILL.md and follow the instructions.
What does this skill do in robots.txt?
It adds User-agent blocks for search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), training crawlers (GPTBot, ClaudeBot, and others), and user-triggered fetchers (ChatGPT-User, Claude-User). You choose which to allow or deny.
Why separate search and training crawlers?
Blocking training crawlers does not prevent search visibility—they are separate crawlers. Search crawlers affect AI result inclusion, while training crawlers govern whether your content enters model datasets.
Source text
SKILL.md · Apache-2.0
---
name: agent-ready-ai-rules
description: >
Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt
---
# Implement AI Bot Rules in robots.txt
Add explicit `User-agent` entries for AI crawlers in your robots.txt per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
## Requirements
- Add `User-agent` blocks for AI search crawlers: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. These fetch the pages AI assistants use in their answers, so blocking them affects whether you appear in AI answers
- Add `User-agent` blocks for AI training crawlers: `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Bytespider`, `Amazonbot`. Blocking these does not affect whether your pages appear in AI search results
- User-triggered fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) retrieve pages when a user asks; some vendors do not apply robots.txt to them
- Set `Allow` and/or `Disallow` rules matching your content policy
- A `User-agent: *` group with rules also passes, but applies one policy to every crawler. Explicit entries let you treat search and training crawlers differently
## Cloudflare
[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
lets you manage AI bot rules from the dashboard without editing robots.txt manually.
## Validate
```
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
```
Check that `checks.botAccessControl.robotsTxtAiRules.status` is `"pass"`.
© fabricioctelles · Apache-2.0 · fabricioctelles/skills