agent-ready-robots-txt — Claude Code skill: robots.txt by RFC 9309
The agent-ready-robots-txt skill publishes a standards-compliant robots.txt at your site root per RFC 9309. Use it when agents need to control bot access and reference your sitemap.
Apache-2.0 ★ 141 updated 2026-10-29
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a claude-codeWhat agent-ready-robots-txt does
The skill sets up robots.txt with User-agent directives, Allow and Disallow rules, and a link to sitemap.xml when present. The file must be served as text/plain with HTTP 200 from your domain root.
For Cloudflare, AI Crawl Control provides dashboard management of robots.txt and lets you add rules specific to AI bots. This skill integrates with the other parts of agent-ready-cloudflare.
After installation, verify the setup: send a POST to https://isitagentready.com/api/scan with {"url": "https://YOUR-SITE.com"} and confirm that checks.discoverability.robotsTxt.status returns "pass".
What it helps with
- serve robots.txt from site root with correct MIME type
- set up rules for search engines and AI crawlers
- reference your sitemap in robots.txt
- validate robots.txt compliance
Example prompts for the agent
- Install agent-ready-robots-txt so the agent can set up robots.txt for your domain
- Ask the agent to add GPTBot and Googlebot rules to robots.txt
- Verify the skill correctly adds your sitemap URL to robots.txt
How to install agent-ready-robots-txt
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a claude-code2Manually: put the files in place
~/.claude/skills/robots-txt/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/robots-txt/
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a codex2Manually: put the files in place
.agents/skills/robots-txt/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a cursor2Manually: put the files in place
.cursor/skills/robots-txt/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install fabricioctelles/skills skills/agent-ready-cloudflare/robots-txt2Manually: put the files in place
.github/skills/robots-txt/SKILL.md1Skills CLI (Vercel)
npx skills add fabricioctelles/skills --skill agent-ready-robots-txt -a gemini-cli2Platform CLI
gemini skills install https://github.com/fabricioctelles/skills --path skills/agent-ready-cloudflare/robots-txtManually: put the files in place
.devin/skills/robots-txt/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about agent-ready-robots-txt
How do I install agent-ready-robots-txt in Claude Code?
Run npx skills add fabricioctelles/skills --skill robots-txt -a claude-code, then open ~/.claude/skills/robots-txt/SKILL.md. For Codex use npx skills add fabricioctelles/skills --skill robots-txt -a codex and .agents/skills/robots-txt/SKILL.md.
What does agent-ready-robots-txt do?
It automates creation and deployment of robots.txt per RFC 9309, with User-agent directives, Allow/Disallow rules, and sitemap reference. This is the standard way to control bot access to your site.
How do I check if robots.txt is working?
POST to https://isitagentready.com/api/scan with {"url": "https://YOUR-SITE.com"} and look for checks.discoverability.robotsTxt.status to return "pass".
Source text
SKILL.md · Apache-2.0
---
name: agent-ready-robots-txt
description: >
Sub-skill de agent-ready-cloudflare: Implement robots.txt
---
# Implement robots.txt
Publish a valid robots.txt at your site root per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
## Requirements
- Serve `/robots.txt` as `text/plain` with HTTP 200
- Include `User-agent` directives with `Allow`/`Disallow` rules
- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`
## Cloudflare
[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
can manage your robots.txt from the dashboard, including AI-specific bot rules.
## Validate
```
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
```
Check that `checks.discoverability.robotsTxt.status` is `"pass"`.
© fabricioctelles · Apache-2.0 · fabricioctelles/skills