# AI Bot Access Checker

The AI Bot Access Checker MCP server for Claude Code checks whether search and AI crawlers can reach a page. It compares their responses with a regular browser request and checks robots.txt rules.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI
- Author: seopaslaugos-io
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/seopaslaugos-io/ai-bot-access-checker
- Updated: 2026-11-12
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/mcp/ai-bot-access-checker

## What it does

The server tests a URL with 13 search and AI crawlers. It reads `robots.txt`, sends requests using each crawler’s User-Agent, then compares the responses with one from a regular browser. This separates robots.txt rules from blocks imposed by a server or firewall.

Results distinguish allowed access, robots.txt blocks, server blocks, and inconclusive checks. Requests come from Cloudflare’s network, not the crawlers’ own IP ranges; a firewall that verifies crawler IPs can make a result inconclusive. Only public URLs using `http` or `https` on ports 80 and 443 are checked. The remote MCP endpoint is `https://tools.seopaslaugos.io/mcp`; an MCP client is required, but no API key is needed.

## Requirements
- Remote server: `https://tools.seopaslaugos.io/mcp`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http ai-bot-access-checker https://tools.seopaslaugos.io/mcp
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add ai-bot-access-checker --url https://tools.seopaslaugos.io/mcp
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "ai-bot-access-checker": {
            "url": "https://tools.seopaslaugos.io/mcp"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "ai-bot-access-checker": {
            "type": "http",
            "url": "https://tools.seopaslaugos.io/mcp"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http ai-bot-access-checker https://tools.seopaslaugos.io/mcp
```

## When to use it
- check whether `robots.txt` lets GPTBot and ClaudeBot access a page
- find server or firewall blocks affecting crawlers
- compare crawler responses with a regular browser response
- identify why an access check returned an inconclusive result

## Example prompts
- Check whether GPTBot can reach `https://example.com`.
- Compare ClaudeBot and Googlebot access to `https://example.com/pricing`.
- Find out what is blocking PerplexityBot on `https://example.com`.

## Similar
- [okx-cex-bot](https://affpapa.org/en/nevibecode/skills/okx-cex-bot.md) — skill · 10300 installs
- [Airdrop Checker — Wallet Eligibility & Claim Status](https://affpapa.org/en/nevibecode/mcp/airdrop-checker-wallet-eligibility-claim-status.md) — MCP server · 5548 installs
- [SSL Certificate Checker — Validity, Expiry & Grade](https://affpapa.org/en/nevibecode/mcp/ssl-certificate-checker-validity-expiry-grade.md) — MCP server · 5042 installs
- [Password Strength Checker — Entropy & Crack Time](https://affpapa.org/en/nevibecode/mcp/password-strength-checker-entropy-crack-time.md) — MCP server · 4119 installs
- [discord-bot](https://affpapa.org/en/nevibecode/skills/discord-bot.md) — skill · 5500 installs
- [telegram-bot-builder](https://affpapa.org/en/nevibecode/skills/sickn33-telegram-bot-builder.md) — skill · 3800 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
