Octocrawl — Claude Code MCP server: web data and setup
Octocrawl is an MCP server for Claude Code that turns web pages into Markdown and structured data. Connect it to an agent to read pages, crawl sites, or extract details from listings.
AGPL-3.0 ★ 7 updated 2026-11-16
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcpWhat Octocrawl does
Octocrawl fetches URLs and returns page content as Markdown, along with links, metadata, and HTML. It can discover site URLs, crawl sections, and process URL batches; each page result includes a source record and a processing status.
Tables and repeated listings can be returned as JSON or CSV, and a JSON Schema can define fields to extract. Dynamic pages need browser mode. Pages available only through a browser require Chromium, installed with npx playwright install chromium. Local use requires Node.js 22.13+ or 24+. The hosted server accepts initial connections without an API key, with that access limited to 20 pages per day over HTTP.
What it helps with
- retrieve a web page as Markdown from its URL
- find site pages through sitemaps and links
- extract table and listing data as JSON or CSV
- crawl a site section with depth and page limits
- keep a result for every URL in a batch
Example prompts for the agent
- Read this page and return its content as Markdown:
https://example.com. - Find documentation pages about authentication on this site and collect their contents.
- Extract product names and prices from this page as JSON.
How to install Octocrawl
1Platform CLI: remote server
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp2Platform CLI
claude mcp add --env W2L_API_URL=<W2L_API_URL> --env W2L_API_TOKEN=<W2L_API_TOKEN> --env W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> --env W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> --transport stdio octocrawl -- npx -y @octocrawl/mcpReplace <…> with your own values
1Platform CLI: remote server
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp2Platform CLI
codex mcp add octocrawl --env W2L_API_URL=<W2L_API_URL> --env W2L_API_TOKEN=<W2L_API_TOKEN> --env W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> --env W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> -- npx -y @octocrawl/mcpReplace <…> with your own values
Config file
{
"mcpServers": {
"octocrawl": {
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}{
"mcpServers": {
"octocrawl": {
"command": "npx",
"args": [
"-y",
"@octocrawl/mcp"
],
"env": {
"W2L_API_URL": "<W2L_API_URL>",
"W2L_API_TOKEN": "<W2L_API_TOKEN>",
"W2L_MCP_TOOLSETS": "<W2L_MCP_TOOLSETS>",
"W2L_MCP_READ_ONLY": "<W2L_MCP_READ_ONLY>"
}
}
}
}Replace <…> with your own values
Config file
{
"servers": {
"octocrawl": {
"type": "http",
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}{
"servers": {
"octocrawl": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@octocrawl/mcp"
],
"env": {
"W2L_API_URL": "<W2L_API_URL>",
"W2L_API_TOKEN": "<W2L_API_TOKEN>",
"W2L_MCP_TOOLSETS": "<W2L_MCP_TOOLSETS>",
"W2L_MCP_READ_ONLY": "<W2L_MCP_READ_ONLY>"
}
}
}
}Replace <…> with your own values
1Platform CLI: remote server
gemini mcp add -t http octocrawl https://mcp.octocrawl.dev/mcp2Platform CLI
gemini mcp add -e W2L_API_URL=<W2L_API_URL> -e W2L_API_TOKEN=<W2L_API_TOKEN> -e W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> -e W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> octocrawl npx -y @octocrawl/mcpReplace <…> with your own values
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about Octocrawl
How do I install Octocrawl in Claude Code?
To connect the hosted server, run claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp. To use the local MCP server, run `claude mcp add --env W2L_API_URL=<W2L_API_URL> --env W2L_API_TOKEN=<W2L_API_TOKEN> --env W2L_MCP_TOOLSETS=<W2L_MCP_TOOLSETS> --env W2L_MCP_READ_ONLY=<W2L_MCP_READ_ONLY> --transport stdio octocrawl -- npx -y @octocrawl/mcp`.
What does Octocrawl do?
It reads web pages and returns Markdown, structured data, and a record of the source. It can also discover site URLs, crawl sections, and handle URL batches.
What does Octocrawl need to run?
The hosted server can be connected initially without an API key. Local use requires Node.js 22.13+ or 24+; browser-only pages require Chromium installed with npx playwright install chromium.