# WebScraping.AI

This MCP server fetches web content using Chromium rendering, proxy rotation, and prompt injection sandboxing. It helps coding agents inspect dynamic sites and pull structured data.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 45
- Author: webscraping-ai
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/webscraping-ai/webscraping-ai-mcp-server
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/mcp/webscrapingai

## What it does

The package links your coding assistant to WebScraping.AI via stdio. It runs headless Chromium behind proxies. That allows extracting content from JavaScript-heavy sites while emulating mobile or desktop screens. The integration also parses search engine results from Google and turns supported social networks into clean JSON payloads.

Built-in tools include `webscraping_ai_question` for page queries and `webscraping_ai_fields` for schema extraction. Concurrency defaults to 5 parallel requests via `WEBSCRAPING_AI_CONCURRENCY_LIMIT`. For security, setting `WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING` to true encloses scraped text in safety borders. This helps stop indirect prompt injections.

Using this local server requires Node.js and a valid `WEBSCRAPING_AI_API_KEY`. Without this API key, all operations fail. You can also connect to the hosted endpoint at `https://mcp.webscraping.ai/mcp` with OAuth if you skip local setup.

## Requirements
- Needs: Node.js (commands run via npx)
- Environment variables: `WEBSCRAPING_AI_API_KEY`, `WEBSCRAPING_AI_DEFAULT_PROXY_TYPE`, `WEBSCRAPING_AI_DEFAULT_JS_RENDERING`, `WEBSCRAPING_AI_CONCURRENCY_LIMIT` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add --env WEBSCRAPING_AI_API_KEY=<WEBSCRAPING_AI_API_KEY> --env WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=<WEBSCRAPING_AI_DEFAULT_PROXY_TYPE> --env WEBSCRAPING_AI_DEFAULT_JS_RENDERING=<WEBSCRAPING_AI_DEFAULT_JS_RENDERING> --env WEBSCRAPING_AI_CONCURRENCY_LIMIT=<WEBSCRAPING_AI_CONCURRENCY_LIMIT> --transport stdio webscrapingai -- npx -y webscraping-ai-mcp
```
Replace <…> with your own values

### Codex

**Platform CLI**
```bash
codex mcp add webscrapingai --env WEBSCRAPING_AI_API_KEY=<WEBSCRAPING_AI_API_KEY> --env WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=<WEBSCRAPING_AI_DEFAULT_PROXY_TYPE> --env WEBSCRAPING_AI_DEFAULT_JS_RENDERING=<WEBSCRAPING_AI_DEFAULT_JS_RENDERING> --env WEBSCRAPING_AI_CONCURRENCY_LIMIT=<WEBSCRAPING_AI_CONCURRENCY_LIMIT> -- npx -y webscraping-ai-mcp
```
Replace <…> with your own values

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "webscrapingai": {
            "command": "npx",
            "args": [
                "-y",
                "webscraping-ai-mcp"
            ],
            "env": {
                "WEBSCRAPING_AI_API_KEY": "<WEBSCRAPING_AI_API_KEY>",
                "WEBSCRAPING_AI_DEFAULT_PROXY_TYPE": "<WEBSCRAPING_AI_DEFAULT_PROXY_TYPE>",
                "WEBSCRAPING_AI_DEFAULT_JS_RENDERING": "<WEBSCRAPING_AI_DEFAULT_JS_RENDERING>",
                "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "<WEBSCRAPING_AI_CONCURRENCY_LIMIT>"
            }
        }
    }
}
```
Replace <…> with your own values

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "webscrapingai": {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "webscraping-ai-mcp"
            ],
            "env": {
                "WEBSCRAPING_AI_API_KEY": "<WEBSCRAPING_AI_API_KEY>",
                "WEBSCRAPING_AI_DEFAULT_PROXY_TYPE": "<WEBSCRAPING_AI_DEFAULT_PROXY_TYPE>",
                "WEBSCRAPING_AI_DEFAULT_JS_RENDERING": "<WEBSCRAPING_AI_DEFAULT_JS_RENDERING>",
                "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "<WEBSCRAPING_AI_CONCURRENCY_LIMIT>"
            }
        }
    }
}
```
Replace <…> with your own values

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add -e WEBSCRAPING_AI_API_KEY=<WEBSCRAPING_AI_API_KEY> -e WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=<WEBSCRAPING_AI_DEFAULT_PROXY_TYPE> -e WEBSCRAPING_AI_DEFAULT_JS_RENDERING=<WEBSCRAPING_AI_DEFAULT_JS_RENDERING> -e WEBSCRAPING_AI_CONCURRENCY_LIMIT=<WEBSCRAPING_AI_CONCURRENCY_LIMIT> webscrapingai npx -y webscraping-ai-mcp
```
Replace <…> with your own values

## When to use it
- Render client-side JavaScript before extracting text or markup
- Retrieve structured product attributes and pricing into JSON
- Scrape Google search result pages for a target location
- Shield LLM prompts from malicious injections inside scraped pages

## Example prompts
- Extract product details from `https://example.com/product` into structured JSON
- Load `https://example.com` with Chromium and summarize the page content
- Search Google for recent news and return the first page as JSON

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89433
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32221
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23263
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 12927

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md