# Octocrawl

Octocrawl is an MCP server for Claude Code that turns web pages into Markdown and structured data. Connect it to an agent to read pages, crawl sites, or extract details from listings.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 7 repo stars
- Author: 77777R7
- License: AGPL-3.0 · policy=link-only · description only; the file stays with the author
- Source: https://github.com/77777R7/Octocrawl
- Updated: 2026-11-16
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/mcp/octocrawl

## What it does

Octocrawl fetches URLs and returns page content as Markdown, along with links, metadata, and HTML. It can discover site URLs, crawl sections, and process URL batches; each page result includes a source record and a processing status.

Tables and repeated listings can be returned as JSON or CSV, and a JSON Schema can define fields to extract. Dynamic pages need browser mode. Pages available only through a browser require Chromium, installed with `npx playwright install chromium`. Local use requires Node.js 22.13+ or 24+. The hosted server accepts initial connections without an API key, with that access limited to 20 pages per day over HTTP.

## Requirements
- Needs: Node.js (commands run via npx)
- Remote server: `https://mcp.octocrawl.dev/mcp`
- Environment variables: `W2L_API_URL`, `W2L_API_TOKEN`, `W2L_MCP_TOOLSETS`, `W2L_MCP_READ_ONLY` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

1. **Platform CLI: remote server**
```bash
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
```

2. **Platform CLI**
```bash
claude mcp add --env W2L_API_URL=<W2L_API_URL> --env W2L_API_TOKEN=<W2L_API_TOKEN> --env W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> --env W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> --transport stdio octocrawl -- npx -y @octocrawl/mcp
```
Replace <…> with your own values

### Codex

1. **Platform CLI: remote server**
```bash
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp
```

2. **Platform CLI**
```bash
codex mcp add octocrawl --env W2L_API_URL=<W2L_API_URL> --env W2L_API_TOKEN=<W2L_API_TOKEN> --env W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> --env W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> -- npx -y @octocrawl/mcp
```
Replace <…> with your own values

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "octocrawl": {
            "url": "https://mcp.octocrawl.dev/mcp"
        }
    }
}
```
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "octocrawl": {
            "command": "npx",
            "args": [
                "-y",
                "@octocrawl/mcp"
            ],
            "env": {
                "W2L_API_URL": "<W2L_API_URL>",
                "W2L_API_TOKEN": "<W2L_API_TOKEN>",
                "W2L_MCP_TOOLSETS": "<W2L_MCP_TOOLSETS>",
                "W2L_MCP_READ_ONLY": "<W2L_MCP_READ_ONLY>"
            }
        }
    }
}
```
Replace <…> with your own values

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "octocrawl": {
            "type": "http",
            "url": "https://mcp.octocrawl.dev/mcp"
        }
    }
}
```
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "octocrawl": {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "@octocrawl/mcp"
            ],
            "env": {
                "W2L_API_URL": "<W2L_API_URL>",
                "W2L_API_TOKEN": "<W2L_API_TOKEN>",
                "W2L_MCP_TOOLSETS": "<W2L_MCP_TOOLSETS>",
                "W2L_MCP_READ_ONLY": "<W2L_MCP_READ_ONLY>"
            }
        }
    }
}
```
Replace <…> with your own values

### Gemini CLI

1. **Platform CLI: remote server**
```bash
gemini mcp add -t http octocrawl https://mcp.octocrawl.dev/mcp
```

2. **Platform CLI**
```bash
gemini mcp add -e W2L_API_URL=<W2L_API_URL> -e W2L_API_TOKEN=<W2L_API_TOKEN> -e W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> -e W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> octocrawl npx -y @octocrawl/mcp
```
Replace <…> with your own values

## When to use it
- retrieve a web page as Markdown from its URL
- find site pages through sitemaps and links
- extract table and listing data as JSON or CSV
- crawl a site section with depth and page limits
- keep a result for every URL in a batch

## Example prompts
- Read this page and return its content as Markdown: `https://example.com`.
- Find documentation pages about authentication on this site and collect their contents.
- Extract product names and prices from this page as JSON.

## Similar
- [firecrawl-company-directories](https://affpapa.org/en/nevibecode/skills/firecrawl-company-directories.md) — skill · 31600 installs
- [scrape](https://affpapa.org/en/nevibecode/skills/brightdata-scrape.md) — skill · 11100 installs
- [Olostep MCP Server](https://affpapa.org/en/nevibecode/mcp/olostep-mcp-server.md) — MCP server · ★ 25 repo stars
- [reporting-pipelines](https://affpapa.org/en/nevibecode/skills/reporting-pipelines.md) — skill · 371 installs
- [firecrawl-lead-research](https://affpapa.org/en/nevibecode/skills/firecrawl-lead-research.md) — skill · 31700 installs
- [just-scrape](https://affpapa.org/en/nevibecode/skills/just-scrape.md) — skill · 245000 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
