Skip to main content
plugin CX Codex

crawlberg — Claude Code plugin: what it does, how to install

crawlberg fetches web pages, converts HTML to Markdown, and crawls domains via a local CLI and MCP server. It brings clean web content directly into agent context.

MIT ★ 1,596 updated 2026-10-29

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add hashgraph-online/awesome-codex-plugins
/plugin install crawlberg@awesome-codex-plugins
Open on GitHub

What crawlberg does

The plugin launches a local MCP server through scripts/mcp-launch.sh and provides tools to scrape, crawl, and inspect web pages. On first use, the launcher checks for an existing crawlberg executable in your PATH, tries package runners like npx, or pulls a prebuilt binary automatically.

Inside are dedicated skills for converting individual URLs to Markdown, recursing through links within a domain, and mapping sitemaps. It also includes an interaction command to click, scroll, and type before capturing page markup. This requires a browser.

If a site requires client-side JavaScript execution, Chrome or Chromium must be available locally. Without a local browser or an external CDP endpoint, headless fallback will fail on dynamic sites.

What it helps with

  • convert a single web page to clean Markdown
  • crawl an entire domain while respecting depth and page limits
  • extract URLs from a sitemap without downloading page content
  • interact with dynamic web pages using click and scroll actions

Example prompts for the agent

  • Scrape https://example.com/article and output the content as Markdown
  • Crawl https://example.com staying on domain and save the pages
  • Map all URLs from https://example.com sitemap

How to install crawlberg

Plugin marketplace

/plugin marketplace add hashgraph-online/awesome-codex-plugins
/plugin install crawlberg@awesome-codex-plugins

Step by step, with commands for each platform: Claude Code plugins: marketplaces, installing, and building your own

Questions about crawlberg

How do I install crawlberg in Claude Code?

Run both marketplace commands in your agent session: /plugin marketplace add hashgraph-online/awesome-codex-plugins, then /plugin install crawlberg@awesome-codex-plugins.

What does crawlberg do?

It connects an MCP server and CLI commands to fetch pages, crawl websites, and convert HTML into Markdown.

Do I need Chrome installed to use crawlberg?

Static pages download fine without a browser. Chrome or Chromium is only needed when pages require JavaScript rendering or interactive browser actions.

Source text

crawlberg · MIT
# crawlberg

Crawl, scrape, and convert websites to Markdown using the local `crawlberg` CLI in your agent.

<!-- ~keep TODO: add screenshot -->

## Install

### From the marketplace (recommended)

Pending review for official Claude marketplace.

Self-host:

```text
/plugin marketplace add xberg-io/plugins
/plugin install crawlberg@xberg
```

### Binary requirement

The MCP server runs through an auto-installing launcher (`scripts/mcp-launch.sh`): on first use it reuses any working `crawlberg` binary already on `PATH`, then tries `npx`/`uvx`, then Homebrew, then a prebuilt download. No manual install is required for MCP.

To install the CLI yourself (recommended for direct CLI use, and reused by the launcher):

```bash
brew install xberg-io/tap/crawlberg
# or run it without a persistent install (the CLI proxy package self-installs the binary):
npx @xberg-io/crawlberg-cli --help
uvx --from crawlberg-cli crawlberg --help
# or build from source (the mcp/api subcommands are non-default features):
cargo install crawlberg-cli --features all
```

The published npm (`@xberg-io/crawlberg`) and PyPI (`crawlberg`) packages are language *library* bindings, not the CLI — install via Homebrew or the from-source build above for the `crawlberg` binary.

Headless fallback requires Chrome/Chromium on your system. The CLI launches it on demand; skip the binary if you only plan to use `--browser-mode never`.

## Skills shipped

| Skill | Trigger |
|-------|---------|
| **crawlberg** | Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out. |
| **crawling-a-site** | Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection. |
| **scraping-html-to-markdown** | Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape <url>`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages. |
| **mapping-urls** | Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl. |
| **automating-the-browser** | Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact <url> --actions` with the real action schema, result shape, limits, and external-CDP options. |
| **serving-the-api** | Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it. |
| **headless-fallback** | Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto\|always\|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost. |

## MCP / CLI

The plugin wires up the `crawlberg` MCP server via `scripts/mcp-launch.sh`, which resolves or installs a version-matched binary, then runs `crawlberg mcp` over stdio. Override binary resolution with `CRAWLBERG_LAUNCHER=auto|download`.

The MCP server exposes nine tools:

- `scrape` — fetch and convert a single URL to Markdown or JSON.
- `crawl` — follow links across a domain, bounded by depth and page count.
- `map` — enumerate URLs from sitemaps and link extraction.
- `batch_scrape` — scrape multiple URLs concurrently.
- `batch_crawl` — crawl multiple seed URLs concurrently.
- `download` — download a document from a URL and report its file metadata.
- `interact` — drive a headless browser with click, type, scroll actions.
- `generate_citations` — rewrite markdown links as numbered citations with a reference list.
- `get_version` — report the crawlberg library version.

The CLI offers the same operations as subcommands — `scrape`, `crawl`, `map`, `interact`, `batch-scrape`, `batch-crawl`, `download`, `citations`, `version` — plus `serve` (a Firecrawl-v1-compatible REST API server) and `mcp` (the stdio MCP server). See the `crawlberg` skill for the full per-subcommand flag surface, and the `serving-the-api` skill for when to run the server instead of the CLI or MCP.

## Configuration

Pass flags or use inline JSON via `--config`:

```bash
crawlberg scrape https://example.com \
  --format markdown \
  --browser-mode auto \
  --timeout 30000
```

For complex configs, use JSON:

```bash
crawlberg crawl https://example.com \
  --config '{"max_depth":3,"max_pages":200,"max_concurrent":8,"respect_robots_txt":true}'
```

See the `crawlberg` and `crawling-a-site` skills for the full flag surface.

## Examples

Fetch a single page and print Markdown:

```text
crawlberg scrape https://example.com/article --format markdown
```

Crawl a site at depth 3 with rate limiting:

```text
crawlberg crawl https://example.com --depth 3 --max-pages 200 --concurrent 8 --stay-on-domain --format markdown
```

Enumerate URLs from a sitemap:

```text
crawlberg map https://example.com --limit 500
```

## Versioning

The plugin version tracks the marketplace `VERSION` file. See [CHANGELOG.md](../../CHANGELOG.md) for release notes.

## License

MIT.

## See also

- **Marketplace**: [xberg-io/plugins](https://github.com/xberg-io/plugins)
- **Upstream**: [xberg-io/crawlberg](https://github.com/xberg-io/crawlberg)
- **Sibling plugins**: [xberg](../xberg/README.md)

© Kreuzberg, Inc. · MIT · hashgraph-online/awesome-codex-plugins

Similar plugins

All plugins for Browser and scraping →

From the same set391

The whole set →
langfuse Plugin runs a local Langfuse MCP server with 48 tools: fetch traces, debug exceptions, analyze sessions, manage prompts and datasets. CX Codex ★ 1,596repo stars lacuna-music lacuna-music generates instrumental tracks and vocal songs from text prompts using the Lacuna MCP server. Useful when you need custom audio or mock soundtracks inside your agent. CX Codex ★ 1,596repo stars kuudo-skills kuudo-skills — a skill pack for Amazon ad audits, listings, pricing, stock, and profit checks. Use it when an agent needs a guided workflow over commerce data. CX Codex ★ 1,596repo stars krexa krexa turns a trading idea, post or video into fixed rules, backtests them, runs paper trades, and starts a live agent only after you sign. CX Codex ★ 1,596repo stars kachilu-browser The kachilu-browser plugin for Claude Code and Codex automates browser tasks, detects CAPTCHA and Cloudflare challenges, and routes completion through a local flow; from WSL2 it drives a Windows browser. CX Codex ★ 1,596repo stars jimeng-skills jimeng-skills runs a production storyboard pipeline for short dramas using six dedicated role subagents. It coordinates the process from raw script intake to verified scene prompts and exports. CX Codex ★ 1,596repo stars jev-social jev-social is a Claude Code plugin that researches Instagram, TikTok, and LinkedIn in Chrome and builds reports with source citations. The agent selects searches, profile or post views, and comment reads. CX Codex ★ 1,596repo stars jetformat Points the agent at the local jetformat CLI instead of hand-rolling PDF bytes or unzipping Office files. Useful when a task ends in a converted, merged or rendered document. CX Codex ★ 1,596repo stars

Project sponsors

Companies that keep this analytics open