# webfetch

Self-hosted web search pipeline for AI agents featuring local scraping, answer compression, and SQLite semantic caching.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 60
- Author: firish
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/firish/webfetch
- Updated: 2026-07-23
- HTML: https://affpapa.org/en/nevibecode/mcp/webfetch

## What it does

The server handles web queries locally instead of offloading retrieval to hosted search endpoints. It blends results from DuckDuckGo, Brave, Serper, and Tavily using reciprocal rank fusion. DuckDuckGo operates out of the box without external credentials. Retrieved pages are parsed with trafilatura and readability, while Playwright manages JavaScript execution and blocks.

Results pass through a BM25 and neural reranking pipeline before shrinking to sentences answering the query. Everything saves to a two-tier SQLite cache. Reworded prompts resolve against previous lookups at zero token cost. Installing the complete semantic stack requires pip 24 or newer inside your environment.

## Requirements
- Needs: Python and uv (commands run via uvx)
- Environment variables: `BRAVE_API_KEY`, `SERPER_API_KEY`, `TAVILY_API_KEY`, `WEBFETCH_PROVIDER` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add --env BRAVE_API_KEY=<BRAVE_API_KEY> --env SERPER_API_KEY=<SERPER_API_KEY> --env TAVILY_API_KEY=<TAVILY_API_KEY> --env WEBFETCH_PROVIDER=<WEBFETCH_PROVIDER> --transport stdio webfetch -- uvx webfetch-llm
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from webfetch-llm <command>

### Codex

**Platform CLI**
```bash
codex mcp add webfetch --env BRAVE_API_KEY=<BRAVE_API_KEY> --env SERPER_API_KEY=<SERPER_API_KEY> --env TAVILY_API_KEY=<TAVILY_API_KEY> --env WEBFETCH_PROVIDER=<WEBFETCH_PROVIDER> -- uvx webfetch-llm
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from webfetch-llm <command>

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "webfetch": {
            "command": "uvx",
            "args": [
                "webfetch-llm"
            ],
            "env": {
                "BRAVE_API_KEY": "<BRAVE_API_KEY>",
                "SERPER_API_KEY": "<SERPER_API_KEY>",
                "TAVILY_API_KEY": "<TAVILY_API_KEY>",
                "WEBFETCH_PROVIDER": "<WEBFETCH_PROVIDER>"
            }
        }
    }
}
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from webfetch-llm <command>

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "webfetch": {
            "type": "stdio",
            "command": "uvx",
            "args": [
                "webfetch-llm"
            ],
            "env": {
                "BRAVE_API_KEY": "<BRAVE_API_KEY>",
                "SERPER_API_KEY": "<SERPER_API_KEY>",
                "TAVILY_API_KEY": "<TAVILY_API_KEY>",
                "WEBFETCH_PROVIDER": "<WEBFETCH_PROVIDER>"
            }
        }
    }
}
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from webfetch-llm <command>

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add -e BRAVE_API_KEY=<BRAVE_API_KEY> -e SERPER_API_KEY=<SERPER_API_KEY> -e TAVILY_API_KEY=<TAVILY_API_KEY> -e WEBFETCH_PROVIDER=<WEBFETCH_PROVIDER> webfetch uvx webfetch-llm
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from webfetch-llm <command>

## When to use it
- Run agent web searches using DuckDuckGo without paying search API subscriptions
- Compress raw HTML pages down to query-relevant sentences before feeding context
- Retrieve cached search findings for paraphrased queries via SQLite
- Render JavaScript-heavy pages and bypass access blocks using Playwright

## Example prompts
- Search the web for recent library releases using webfetch
- Pull the full page content from the documentation link with fetch_url
- Display the search token usage summary using savings_report

## Similar
- [REA](https://affpapa.org/en/nevibecode/mcp/rea.md) — MCP server · ★ 386989
- [GitHub](https://affpapa.org/en/nevibecode/mcp/github.md) — MCP server · ★ 33918
- [Netdata](https://affpapa.org/en/nevibecode/mcp/netdata.md) — MCP server · ★ 81115
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119160
- [Codebase Memory](https://affpapa.org/en/nevibecode/mcp/codebase-memory.md) — MCP server · ★ 47392
- [mcp-server](https://affpapa.org/en/nevibecode/mcp/mcp-server.md) — MCP server · ★ 43872

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md