# Crawlberg

Converts web pages and entire sites into clean Markdown or structured JSON using a local Docker container.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 190
- Author: xberg-io
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/xberg-io/crawlberg
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/mcp/crawlberg

## What it does

The server crawls websites and parses single pages, turning raw HTML into clean Markdown and structured metadata. Its Rust core pulls out headings, links, images, Open Graph tags, and JSON-LD blocks. When dealing with heavy client-side JavaScript, it falls back to a headless browser to retrieve rendered content.

It supports depth-first and breadth-first crawling while honoring `robots.txt` directives and discovering sitemaps. Requests are automatically throttled per domain to prevent overloading target servers. By default, SSRF mitigation blocks calls to loopback and private IP ranges.

Running this tool requires a working Docker daemon on your machine. The image `ghcr.io/xberg-io/crawlberg:1.12.0` runs through standard input and output, leaving no lingering containers behind. If you plan to crawl internal networks, you will need to disable the SSRF block in the configuration.

## Requirements
- Needs: Docker

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0
```

### Codex

**Platform CLI**
```bash
codex mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "crawlberg": {
            "command": "docker",
            "args": [
                "run",
                "-i",
                "--rm",
                "ghcr.io/xberg-io/crawlberg:1.12.0"
            ]
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "crawlberg": {
            "type": "stdio",
            "command": "docker",
            "args": [
                "run",
                "-i",
                "--rm",
                "ghcr.io/xberg-io/crawlberg:1.12.0"
            ]
        }
    }
}
```

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add crawlberg docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0
```

## When to use it
- extract web page content into clean Markdown for LLM prompts
- crawl an entire documentation site with configurable depth limits
- pull structured JSON-LD and Open Graph metadata without custom parsers
- scrape JavaScript-heavy single-page applications via a headless browser

## Example prompts
- Scrape the provided page URL and return clean Markdown with its metadata.
- Crawl this documentation site up to depth 2 and save all articles.
- Extract Open Graph tags and JSON-LD structures from the targeted blog post.

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md