searxNcrawl — Claude Code MCP server: search and crawl
Gives your agent web search through SearXNG and You.com, and turns pages into clean Markdown. Useful when the agent needs live pages instead of guesses.
claude mcp add --env SEARXNG_URL=<SEARXNG_URL> --env SEARXNG_USERNAME=<SEARXNG_USERNAME> --env SEARXNG_PASSWORD=<SEARXNG_PASSWORD> --env YDC_API_KEY=<YDC_API_KEY> --env SEARCH_RESULT_FIELDS=<SEARCH_RESULT_FIELDS> --transport stdio searxncrawl -- uvx searxncrawlWhat searxNcrawl does
searxNcrawl is an MCP server plus a command-line toolkit for web search and page crawling. Queries go to the providers you configure in parallel, then the hits are merged and labelled with their source; language and time filters are available. Crawling runs on Crawl4AI and takes one URL or many, returning Markdown or JSON with repeated blocks dropped and links optionally removed.
Whole sites are supported too, with depth and page caps — that fits documentation sets. A crawl-capture step records a login session once, after which the crawler can read pages behind it. The server speaks STDIO for local clients or exposes an HTTP endpoint with host and CORS guards. The same search, crawl and crawl-capture commands work from a terminal, and the package can be imported in Python.
You need uv. Search wants at least one provider: a SearXNG instance with JSON output turned on, or a You.com key in YDC_API_KEY. Without SearXNG the search tool has nothing to query, while crawling still runs. The first crawl pulls Playwright's Chromium, roughly 550 MB, so expect a wait on a slow connection. Running uvx searxncrawl setup finds Claude, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop on your machine, asks for the SearXNG URL and an optional You.com key, prints every config change and applies it only after you agree; each file is backed up first and setup --remove reverts the edits.
What it helps with
- pull a library's current docs and get them as text
- mirror a docs section into a single Markdown file
- read a page that sits behind a login without exporting cookies by hand
- compare SearXNG and You.com results for the same query
- hand the agent article text with nav and ads stripped
Example prompts for the agent
- Crawl the docs at this URL into Markdown, depth 2, no more than 30 pages.
- Capture the login session for this site, then pull the text of the settings page.
How to install searxNcrawl
Platform CLI
claude mcp add --env SEARXNG_URL=<SEARXNG_URL> --env SEARXNG_USERNAME=<SEARXNG_USERNAME> --env SEARXNG_PASSWORD=<SEARXNG_PASSWORD> --env YDC_API_KEY=<YDC_API_KEY> --env SEARCH_RESULT_FIELDS=<SEARCH_RESULT_FIELDS> --transport stdio searxncrawl -- uvx searxncrawlReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from searxncrawl <command>
Platform CLI
codex mcp add searxncrawl --env SEARXNG_URL=<SEARXNG_URL> --env SEARXNG_USERNAME=<SEARXNG_USERNAME> --env SEARXNG_PASSWORD=<SEARXNG_PASSWORD> --env YDC_API_KEY=<YDC_API_KEY> --env SEARCH_RESULT_FIELDS=<SEARCH_RESULT_FIELDS> -- uvx searxncrawlReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from searxncrawl <command>
Config file
{
"mcpServers": {
"searxncrawl": {
"command": "uvx",
"args": [
"searxncrawl"
],
"env": {
"SEARXNG_URL": "<SEARXNG_URL>",
"SEARXNG_USERNAME": "<SEARXNG_USERNAME>",
"SEARXNG_PASSWORD": "<SEARXNG_PASSWORD>",
"YDC_API_KEY": "<YDC_API_KEY>",
"SEARCH_RESULT_FIELDS": "<SEARCH_RESULT_FIELDS>"
}
}
}
}Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from searxncrawl <command>
Config file
{
"servers": {
"searxncrawl": {
"type": "stdio",
"command": "uvx",
"args": [
"searxncrawl"
],
"env": {
"SEARXNG_URL": "<SEARXNG_URL>",
"SEARXNG_USERNAME": "<SEARXNG_USERNAME>",
"SEARXNG_PASSWORD": "<SEARXNG_PASSWORD>",
"YDC_API_KEY": "<YDC_API_KEY>",
"SEARCH_RESULT_FIELDS": "<SEARCH_RESULT_FIELDS>"
}
}
}
}Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from searxncrawl <command>
Platform CLI
gemini mcp add -e SEARXNG_URL=<SEARXNG_URL> -e SEARXNG_USERNAME=<SEARXNG_USERNAME> -e SEARXNG_PASSWORD=<SEARXNG_PASSWORD> -e YDC_API_KEY=<YDC_API_KEY> -e SEARCH_RESULT_FIELDS=<SEARCH_RESULT_FIELDS> searxncrawl uvx searxncrawlReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from searxncrawl <command>
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about searxNcrawl
How do I install searxNcrawl in Claude Code?
Run claude mcp add searxncrawl -e SEARXNG_PASSWORD=<SEARXNG_PASSWORD> -e YDC_API_KEY=<YDC_API_KEY> -- uvx searxncrawl. A longer form passes more variables: `claude mcp add --env SEARXNG_URL=<SEARXNG_URL> --env SEARXNG_USERNAME=<SEARXNG_USERNAME> --env SEARXNG_PASSWORD=<SEARXNG_PASSWORD> --env YDC_API_KEY=<YDC_API_KEY> --env SEARCH_RESULT_FIELDS=<SEARCH_RESULT_FIELDS> --transport stdio searxncrawl -- uvx searxncrawl`.
What does searxNcrawl do?
It searches the web via SearXNG and You.com and crawls single pages or whole sites, handing the agent clean Markdown rather than raw HTML. It also works as a CLI and a Python library.
Do I need an API key for search?
With your own SearXNG instance no key is needed, but JSON output must be enabled there. You.com requires YDC_API_KEY. Crawling works without any search provider.
