Skip to main content
skill CX Codex

agent-ready-ai-rules — Claude Code skill: AI bot rules in robots.txt

agent-ready-ai-rules is a skill that adds RFC 9309 rules for AI search and training crawlers to robots.txt, controlling page visibility in AI answers and use for model training.

Apache-2.0 ★ 141 updated 2026-10-29

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a claude-code
Open on GitHub

What agent-ready-ai-rules does

The skill configures User-agent entries for different crawler types. Search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot determine visibility in AI answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot) control dataset inclusion—blocking them does not affect search results but prevents use for model training.

Cloudflare users can manage these rules through the AI Crawl Control dashboard without editing the file directly. Validation runs via the isitagentready.com API: POST to https://isitagentready.com/api/scan and check that checks.botAccessControl.robotsTxtAiRules.status returns "pass".

Installs with a single command depending on your IDE. After setup, the skill file appears in your platform's skills folder—open it and follow the steps.

What it helps with

  • prevent AI crawlers from indexing your site for search
  • block training bots from using content for model training
  • allow some bot types while blocking others
  • validate robots.txt against RFC 9309
  • manage bot rules through a UI instead of file editing

Example prompts for the agent

  • Set up robots.txt to block ClaudeBot from training while allowing Claude-SearchBot to index pages
  • Add AI search crawler rules to robots.txt per RFC 9309
  • Check if GPTBot is correctly blocked in my robots.txt

How to install agent-ready-ai-rules

1Skills CLI (Vercel)

npx skills add fabricioctelles/skills --skill agent-ready-ai-rules -a claude-code

2Manually: put the files in place

~/.claude/skills/ai-rules/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/ai-rules/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about agent-ready-ai-rules

How do I install agent-ready-ai-rules in Claude Code?

Run npx skills add fabricioctelles/skills --skill ai-rules -a claude-code, then open ~/.claude/skills/ai-rules/SKILL.md and follow the instructions.

What does this skill do in robots.txt?

It adds User-agent blocks for search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), training crawlers (GPTBot, ClaudeBot, and others), and user-triggered fetchers (ChatGPT-User, Claude-User). You choose which to allow or deny.

Why separate search and training crawlers?

Blocking training crawlers does not prevent search visibility—they are separate crawlers. Search crawlers affect AI result inclusion, while training crawlers govern whether your content enters model datasets.

Source text

SKILL.md · Apache-2.0
---
name: agent-ready-ai-rules
description: >
  Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt
---
# Implement AI Bot Rules in robots.txt

Add explicit `User-agent` entries for AI crawlers in your robots.txt per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## Requirements

- Add `User-agent` blocks for AI search crawlers: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. These fetch the pages AI assistants use in their answers, so blocking them affects whether you appear in AI answers
- Add `User-agent` blocks for AI training crawlers: `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Bytespider`, `Amazonbot`. Blocking these does not affect whether your pages appear in AI search results
- User-triggered fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) retrieve pages when a user asks; some vendors do not apply robots.txt to them
- Set `Allow` and/or `Disallow` rules matching your content policy
- A `User-agent: *` group with rules also passes, but applies one policy to every crawler. Explicit entries let you treat search and training crawlers differently

## Cloudflare

[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
lets you manage AI bot rules from the dashboard without editing robots.txt manually.

## Validate

```
POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}
```

Check that `checks.botAccessControl.robotsTxtAiRules.status` is `"pass"`.

© fabricioctelles · Apache-2.0 · fabricioctelles/skills

Similar skills

All skills for Security →

From the same set46

The whole set →
agent-ready-oauth-protected-resource agent-ready-oauth-protected-resource publishes OAuth Protected Resource Metadata so agents can discover how to authenticate. It fits Cloudflare setups that expose a protected resource. CX Codex +1 ★ 141repo stars agent-ready-link-headers agent-ready-link-headers is a Claude Code skill that adds Link response headers to a site's homepage so AI agents can discover its API and documentation automatically. CX Codex +1 ★ 141repo stars agent-ready-markdown-negotiation agent-ready-markdown-negotiation implements Accept: text/markdown content negotiation so AI agents receive clean Markdown representations instead of raw HTML. CX Codex +1 ★ 141repo stars agent-ready-webmcp agent-ready-webmcp wires a site's actions into the WebMCP API so browser-based AI agents can find and call them. CX Codex +1 ★ 141repo stars agent-ready-mpp agent-ready-mpp teaches AI agents to configure Machine Payment Protocol (MPP) discovery in OpenAPI schemas by adding x-payment-info tags to payable endpoints. CX Codex +1 ★ 141repo stars agent-ready-dns-aid The agent-ready-dns-aid skill guides DNS-AID records that let agents locate endpoints through DNS. Use it when publishing SVCB or HTTPS records under _agents. CX Codex +1 ★ 141repo stars agent-ready-personal-agent-protocol agent-ready-personal-agent-protocol publishes Personal Agent Protocol (Poppy) draft 0.1 discovery metadata at /.well-known/poppy.json and validates OAuth issuer binding. CX Codex +1 ★ 141repo stars agent-ready-auth-md agent-ready-auth-md publishes an auth.md file and OAuth metadata so agents can discover how to register with a service. Kicks in when a scanner flags the authMd check as failing. CX Codex +1 ★ 141repo stars

Project sponsors

Companies that keep this analytics open