Skip to main content
skill CX Codex

agent-ready-robots-txt — Claude Code skill: robots.txt by RFC 9309

The agent-ready-robots-txt skill publishes a standards-compliant robots.txt at your site root per RFC 9309. Use it when agents need to control bot access and reference your sitemap.

Apache-2.0 ★ 141 updated 2026-10-29

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a claude-code
Open on GitHub

What agent-ready-robots-txt does

The skill sets up robots.txt with User-agent directives, Allow and Disallow rules, and a link to sitemap.xml when present. The file must be served as text/plain with HTTP 200 from your domain root.

For Cloudflare, AI Crawl Control provides dashboard management of robots.txt and lets you add rules specific to AI bots. This skill integrates with the other parts of agent-ready-cloudflare.

After installation, verify the setup: send a POST to https://isitagentready.com/api/scan with {"url": "https://YOUR-SITE.com"} and confirm that checks.discoverability.robotsTxt.status returns "pass".

What it helps with

  • serve robots.txt from site root with correct MIME type
  • set up rules for search engines and AI crawlers
  • reference your sitemap in robots.txt
  • validate robots.txt compliance

Example prompts for the agent

  • Install agent-ready-robots-txt so the agent can set up robots.txt for your domain
  • Ask the agent to add GPTBot and Googlebot rules to robots.txt
  • Verify the skill correctly adds your sitemap URL to robots.txt

How to install agent-ready-robots-txt

1Skills CLI (Vercel)

npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a claude-code

2Manually: put the files in place

~/.claude/skills/robots-txt/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/robots-txt/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about agent-ready-robots-txt

How do I install agent-ready-robots-txt in Claude Code?

Run npx skills add fabricioctelles/skills --skill robots-txt -a claude-code, then open ~/.claude/skills/robots-txt/SKILL.md. For Codex use npx skills add fabricioctelles/skills --skill robots-txt -a codex and .agents/skills/robots-txt/SKILL.md.

What does agent-ready-robots-txt do?

It automates creation and deployment of robots.txt per RFC 9309, with User-agent directives, Allow/Disallow rules, and sitemap reference. This is the standard way to control bot access to your site.

How do I check if robots.txt is working?

POST to https://isitagentready.com/api/scan with {"url": "https://YOUR-SITE.com"} and look for checks.discoverability.robotsTxt.status to return "pass".

Source text

SKILL.md · Apache-2.0
---
name: agent-ready-robots-txt
description: >
  Sub-skill de agent-ready-cloudflare: Implement robots.txt
---
# Implement robots.txt

Publish a valid robots.txt at your site root per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## Requirements

- Serve `/robots.txt` as `text/plain` with HTTP 200
- Include `User-agent` directives with `Allow`/`Disallow` rules
- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`

## Cloudflare

[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
can manage your robots.txt from the dashboard, including AI-specific bot rules.

## Validate

```
POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}
```

Check that `checks.discoverability.robotsTxt.status` is `"pass"`.

© fabricioctelles · Apache-2.0 · fabricioctelles/skills

Similar skills

All skills for SEO and GEO →

From the same set46

The whole set →
agent-ready-sitemap The agent-ready-sitemap skill guides an agent through adding and checking an XML sitemap for public site pages. Use it when search crawlers need a clear page index. CX Codex +1 ★ 141repo stars agent-ready-llms-txt The agent-ready-llms-txt skill publishes a /llms.txt file on your site so agents and AI models can quickly understand its structure and content. Use it to make your site agent-ready. CX Codex +1 ★ 141repo stars agent-ready-x402 agent-ready-x402 configures x402 payment protocol support for backend routes. Useful when your API needs to accept automated HTTP payments from AI agents. CX Codex +1 ★ 141repo stars agent-ready-acp The agent-ready-acp skill guides the publication of an ACP discovery document so AI agents can find an Agentic Commerce Protocol implementation. CX Codex +1 ★ 141repo stars agent-ready-api-catalog agent-ready-api-catalog implements an RFC 9727 API catalog so autonomous agents can discover your endpoints and OpenAPI specifications. CX Codex +1 ★ 141repo stars agent-ready-content-signals The agent-ready-content-signals skill adds Content Signals to robots.txt, letting you set preferences for AI training, search, and AI input. CX Codex +1 ★ 141repo stars agent-ready-mcp-server-card agent-ready-mcp-server-card — a skill that publishes an MCP Server Card, a JSON file for agent discovery. Use it when running your own MCP server and want tools like isitagentready.com to find and validate it. CX Codex +1 ★ 141repo stars agent-ready-a2a-agent-card agent-ready-a2a-agent-card generates an A2A Agent Card file for agent-to-agent discovery. Useful when setting up agent endpoints on Cloudflare Workers. CX Codex +1 ★ 141repo stars

Project sponsors

Companies that keep this analytics open