# ScrapingBot

Gives an agent tools to fetch web pages and query public data from Google, TikTok, Instagram, and Amazon. An API key is required.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI
- Author: mike-scrapingbot
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/mike-scrapingbot/scrapingbot
- Updated: 2026-10-03
- HTML: https://affpapa.org/en/nevibecode/mcp/scrapingbot

## What it does

ScrapingBot exposes tools for retrieving web pages as HTML or Markdown, including pages that need JavaScript rendering. An agent can request selected fields as JSON or run browser actions such as clicking, typing, scrolling, and waiting. Separate tools cover public data from Google, TikTok, Instagram, and Amazon.

The hosted service connects over Streamable HTTP, so there is no local server to install. Add it to Claude Code with `claude mcp add --transport http scrapingbot https://scrapingbot.io/api/mcp`, then provide a ScrapingBot API key in the `x-api-key` header. Requests cannot be authorized without that key. Tool calls use credits according to the API behind each tool.

## Requirements
- Remote server: `https://scrapingbot.io/api/mcp`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http scrapingbot https://scrapingbot.io/api/mcp
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add scrapingbot --url https://scrapingbot.io/api/mcp
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "scrapingbot": {
            "url": "https://scrapingbot.io/api/mcp"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "scrapingbot": {
            "type": "http",
            "url": "https://scrapingbot.io/api/mcp"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http scrapingbot https://scrapingbot.io/api/mcp
```

## When to use it
- fetch a web page as HTML or Markdown
- extract selected page fields into JSON
- search Google for public results
- retrieve public TikTok or Instagram profile data
- run browser actions such as clicking and scrolling

## Example prompts
- Fetch `https://example.com` and return the page as Markdown.
- Extract the product name and price from this page as JSON.
- Search Google for reviews of this place and summarize the results.

## Similar
- [REA](https://affpapa.org/en/nevibecode/mcp/rea.md) — MCP server · ★ 397855
- [GitHub](https://affpapa.org/en/nevibecode/mcp/github.md) — MCP server · ★ 34011
- [Netdata](https://affpapa.org/en/nevibecode/mcp/netdata.md) — MCP server · ★ 81191
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119324
- [Codebase Memory](https://affpapa.org/en/nevibecode/mcp/codebase-memory.md) — MCP server · ★ 47670
- [mcp-server](https://affpapa.org/en/nevibecode/mcp/mcp-server.md) — MCP server · ★ 43929

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md