# Nstdata Crawl

Nstdata Crawl — an MCP server for Claude Code and Codex that scrapes single pages and crawls sites into Markdown. Connects scraping tools directly to your coding assistant.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 3 repo stars
- Author: nstdata-ai
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/nstdata-ai/nstdata-mcp-server
- Updated: 2026-11-16
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/mcp/nstdata-crawl

## What it does

The server supplies six scraping tools to extract web content, traverse paginated sites, and retrieve referenced text or binary resources. It operates either as an OAuth-authenticated remote endpoint at `https://mcp.nstdata.io/crawl` or locally via standard input-output with `npx`.

Local execution requires Node.js 22 (version 22.18 or newer) or 24, along with an `NSTDATA_API_TOKEN` retrieved from the service dashboard. The binary parses environment variables directly rather than loading `.env` files. Submitting crawl tasks consumes remote platform credits, and the tools do not poll or retry failed attempts on their own.

Cancelling a wait command locally does not terminate an active background crawl, so task IDs must be preserved to resume tracking. An optional HTTP/HTTPS proxy module provides two supplementary tools.

## Requirements
- Remote server: `https://mcp.nstdata.io/crawl`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http crawl https://mcp.nstdata.io/crawl
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add nstdata-crawl --url https://mcp.nstdata.io/crawl
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "nstdata-crawl": {
            "url": "https://mcp.nstdata.io/crawl"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "nstdata-crawl": {
            "type": "http",
            "url": "https://mcp.nstdata.io/crawl"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http nstdata-crawl https://mcp.nstdata.io/crawl
```

## When to use it
- extract single page text into Markdown
- crawl paginated sections of a target website
- fetch remote references in text or binary form
- route requests through managed HTTP or HTTPS proxies

## Example prompts
- Scrape the target documentation page and return the text in Markdown
- Run a bounded crawl across the specified website pages
- Retrieve the finished crawl output using this task identifier

## Similar
- [tavily-crawl](https://affpapa.org/en/nevibecode/skills/tavily-crawl.md) — skill · 17900 installs
- [firecrawl-crawl](https://affpapa.org/en/nevibecode/skills/firecrawl-crawl.md) — skill · 81400 installs
- [Firecrawl MCP Server](https://affpapa.org/en/nevibecode/mcp/firecrawl-mcp-server.md) — MCP server · ★ 7877 repo stars
- [WebReaper](https://affpapa.org/en/nevibecode/mcp/webreaper.md) — MCP server · ★ 153 repo stars
- [Atlas Web Compiler & Scraper](https://affpapa.org/en/nevibecode/mcp/atlas-web-compiler-scraper.md) — MCP server · ★ 1 repo stars
- [Deepcrawl](https://affpapa.org/en/nevibecode/mcp/deepcrawl.md) — MCP server

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
