Crawlberg — Claude Code MCP server: what it does, how to install
Converts web pages and entire sites into clean Markdown or structured JSON using a local Docker container.
claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0What Crawlberg does
The server crawls websites and parses single pages, turning raw HTML into clean Markdown and structured metadata. Its Rust core pulls out headings, links, images, Open Graph tags, and JSON-LD blocks. When dealing with heavy client-side JavaScript, it falls back to a headless browser to retrieve rendered content.
It supports depth-first and breadth-first crawling while honoring robots.txt directives and discovering sitemaps. Requests are automatically throttled per domain to prevent overloading target servers. By default, SSRF mitigation blocks calls to loopback and private IP ranges.
Running this tool requires a working Docker daemon on your machine. The image ghcr.io/xberg-io/crawlberg:1.12.0 runs through standard input and output, leaving no lingering containers behind. If you plan to crawl internal networks, you will need to disable the SSRF block in the configuration.
What it helps with
- extract web page content into clean Markdown for LLM prompts
- crawl an entire documentation site with configurable depth limits
- pull structured JSON-LD and Open Graph metadata without custom parsers
- scrape JavaScript-heavy single-page applications via a headless browser
Example prompts for the agent
- Scrape the provided page URL and return clean Markdown with its metadata.
- Crawl this documentation site up to depth 2 and save all articles.
- Extract Open Graph tags and JSON-LD structures from the targeted blog post.
How to install Crawlberg
Platform CLI
claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0Platform CLI
codex mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0Config file
{
"mcpServers": {
"crawlberg": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.0"
]
}
}
}Config file
{
"servers": {
"crawlberg": {
"type": "stdio",
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.0"
]
}
}
}Platform CLI
gemini mcp add crawlberg docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about Crawlberg
How to install Crawlberg in Claude Code?
Run this command in your terminal: claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.0. A running Docker engine is required.
What does Crawlberg do?
It crawls websites or scrapes individual URLs, converting raw HTML into clean Markdown and structured metadata. The server can render dynamic JavaScript pages through a headless browser and filters private network addresses.
Can Crawlberg scrape private networks or localhost?
By default, SSRF protection blocks all requests to loopback addresses, private subnets, and cloud metadata endpoints. Crawling internal hosts requires explicitly overriding this behavior through environment variables or CrawlConfig.
