# Crawlie

Crawlie lets an agent run technical SEO and GEO audits, inspect findings, and compare reports. Use it when site checks need to run through MCP.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 115
- Author: spronta
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/spronta/crawlie
- Updated: 2026-07-18
- HTML: https://affpapa.org/en/nevibecode/mcp/crawlie

## What it does

Crawlie crawls a whole site or selected URLs and checks issues such as broken links, redirects, missing metadata, and AI-search readiness. Its MCP tools let an agent run audits, inspect affected URLs, review top fixes, and compare saved reports. Separate Health, GEO, and Accessibility scores keep one category from masking another.

You can connect the local `crawlie-mcp` process or use the hosted MCP endpoint at `https://crawlie.app/mcp`. The Claude Code setup command is `claude mcp add --transport http crawlie https://crawlie.app/mcp`; the hosted service requires a Crawlie API key. The CLI also supports JSON output, saved report history, and CI failure thresholds.

Crawlie works with Claude Code, Codex, Cursor, GitHub Copilot, Cline, Zed, and other MCP clients. Hosted crawls count against the selected plan, so the API access and available usage need checking before relying on the cloud connection.

## Requirements
- Remote server: `https://crawlie.app/mcp`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http crawlie https://crawlie.app/mcp
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add crawlie --url https://crawlie.app/mcp
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "crawlie": {
            "url": "https://crawlie.app/mcp"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "crawlie": {
            "type": "http",
            "url": "https://crawlie.app/mcp"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http crawlie https://crawlie.app/mcp
```

## When to use it
- scan a site for broken links and redirects
- spot missing metadata and GEO gaps
- produce a technical SEO report
- compare findings across two reports
- sort fixes by severity

## Example prompts
- Run a full technical SEO and GEO audit for the site, then list the highest-priority fixes.
- Check `https://example.com/pricing` for metadata, link, and AI-search readiness issues.
- Compare the latest crawl with the previous report and explain what changed.

## Similar
- [New-Term Radar](https://affpapa.org/en/nevibecode/mcp/new-term-radar.md) — MCP server · ★ 1766
- [Google Maps MCP Server](https://affpapa.org/en/nevibecode/mcp/google-maps-mcp-server.md) — MCP server · ★ 479
- [squirrelscan](https://affpapa.org/en/nevibecode/mcp/squirrelscan.md) — MCP server · ★ 296
- [Tin](https://affpapa.org/en/nevibecode/mcp/tin.md) — MCP server · ★ 263
- [lighthouse-mcp](https://affpapa.org/en/nevibecode/mcp/lighthouse-mcp.md) — MCP server · ★ 212
- [SerpApi](https://affpapa.org/en/nevibecode/mcp/serpapi.md) — MCP server · ★ 182

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md