AI Bot Access Checker — Claude Code MCP server
The AI Bot Access Checker MCP server for Claude Code checks whether search and AI crawlers can reach a page. It compares their responses with a regular browser request and checks robots.txt rules.
MIT updated 2026-11-12
claude mcp add --transport http ai-bot-access-checker https://tools.seopaslaugos.io/mcpWhat AI Bot Access Checker does
The server tests a URL with 13 search and AI crawlers. It reads robots.txt, sends requests using each crawler’s User-Agent, then compares the responses with one from a regular browser. This separates robots.txt rules from blocks imposed by a server or firewall.
Results distinguish allowed access, robots.txt blocks, server blocks, and inconclusive checks. Requests come from Cloudflare’s network, not the crawlers’ own IP ranges; a firewall that verifies crawler IPs can make a result inconclusive. Only public URLs using http or https on ports 80 and 443 are checked. The remote MCP endpoint is https://tools.seopaslaugos.io/mcp; an MCP client is required, but no API key is needed.
What it helps with
- check whether
robots.txtlets GPTBot and ClaudeBot access a page - find server or firewall blocks affecting crawlers
- compare crawler responses with a regular browser response
- identify why an access check returned an inconclusive result
Example prompts for the agent
- Check whether GPTBot can reach
https://example.com. - Compare ClaudeBot and Googlebot access to
https://example.com/pricing. - Find out what is blocking PerplexityBot on
https://example.com.
How to install AI Bot Access Checker
Platform CLI: remote server
claude mcp add --transport http ai-bot-access-checker https://tools.seopaslaugos.io/mcpPlatform CLI: remote server
codex mcp add ai-bot-access-checker --url https://tools.seopaslaugos.io/mcpConfig file
{
"mcpServers": {
"ai-bot-access-checker": {
"url": "https://tools.seopaslaugos.io/mcp"
}
}
}Config file
{
"servers": {
"ai-bot-access-checker": {
"type": "http",
"url": "https://tools.seopaslaugos.io/mcp"
}
}
}Platform CLI: remote server
gemini mcp add -t http ai-bot-access-checker https://tools.seopaslaugos.io/mcpStep by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about AI Bot Access Checker
How do I install AI Bot Access Checker in Claude Code?
Run claude mcp add --transport http ai-bot-access-checker https://tools.seopaslaugos.io/mcp to connect the remote MCP server.
What does AI Bot Access Checker do?
It tests how robots.txt and the server handle requests from search and AI crawlers, then compares their responses with a regular browser request.
What does AI Bot Access Checker need to run?
It needs an MCP client that can reach https://tools.seopaslaugos.io/mcp. No API key is required, and checks are limited to public http or https URLs on ports 80 and 443.