# Atlas Web Compiler & Scraper

An MCP server that turns web pages into clean markdown, maps site structure and compiles pages in batches without an LLM in the loop.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 1
- Author: Atlas-Compiler
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/Atlas-Compiler/Atlas/tree/HEAD/mcp
- Updated: 2026-10-16
- HTML: https://affpapa.org/en/nevibecode/mcp/atlas-web-compiler-scraper

## What it does

Atlas Web Compiler & Scraper wires your coding agent into the Atlas pipeline, which parses pages into structured markdown. The compiler engine does the parsing, so nothing gets summarised away and no invented text shows up in the output. Long pages are handled too: the server builds a table of contents, splits content into sections, watches the token budget and marks anything it had to cut.

Two transports are available. Run the stdio bridge locally with `npx -y @atlascompiler/mcp`, or point remote clients at the HTTP endpoint on `POST /mcp`. The setup wizard `npx -y @atlascompiler/mcp init` validates your key and merges the server block into editor configs without writing the token into project files. Credentials live in `~/.atlas/credentials.json` with `0600` permissions; in CI you pass `ATLAS_API_KEY` instead.

The CLI refuses `--api-key` and `-k` flags and exits with code 1, so the key never lands in `ps aux` output. Outbound requests go through `atlas-egress`, and `atlas://` resources stay scoped to a single workspace. You need an `atlas_...` key from the Atlas dashboard before any of this works.

## Requirements
- Needs: Node.js (commands run via npx)
- Remote server: `https://api.atlas-compiler.com/mcp`
- Environment variables: `ATLAS_API_KEY` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

1. **Platform CLI: remote server**
```bash
claude mcp add --transport http atlas https://api.atlas-compiler.com/mcp
```

2. **Platform CLI**
```bash
claude mcp add --env ATLAS_API_KEY=<ATLAS_API_KEY> --transport stdio mcp -- npx -y @atlascompiler/mcp
```
Replace <…> with your own values

### Codex

1. **Platform CLI: remote server**
```bash
codex mcp add mcp --url https://api.atlas-compiler.com/mcp
```

2. **Platform CLI**
```bash
codex mcp add mcp --env ATLAS_API_KEY=<ATLAS_API_KEY> -- npx -y @atlascompiler/mcp
```
Replace <…> with your own values

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "mcp": {
            "url": "https://api.atlas-compiler.com/mcp"
        }
    }
}
```
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@atlascompiler/mcp"
            ],
            "env": {
                "ATLAS_API_KEY": "<ATLAS_API_KEY>"
            }
        }
    }
}
```
Replace <…> with your own values

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "mcp": {
            "type": "http",
            "url": "https://api.atlas-compiler.com/mcp"
        }
    }
}
```
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "mcp": {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "@atlascompiler/mcp"
            ],
            "env": {
                "ATLAS_API_KEY": "<ATLAS_API_KEY>"
            }
        }
    }
}
```
Replace <…> with your own values

### Gemini CLI

1. **Platform CLI: remote server**
```bash
gemini mcp add -t http mcp https://api.atlas-compiler.com/mcp
```

2. **Platform CLI**
```bash
gemini mcp add -e ATLAS_API_KEY=<ATLAS_API_KEY> mcp npx -y @atlascompiler/mcp
```
Replace <…> with your own values

## When to use it
- pull article text into markdown without a model rewriting it
- map a site's sections before crawling
- compile a batch of pages in one run
- trim a long page to a token budget with a visible cut marker
- configure an MCP server in your editor without committing the key

## Example prompts
- Compile https://atlas-compiler.com into markdown and show me the table of contents
- Map this site and crawl the documentation section
- Gather these pages into one file, trimming to the token budget

## Similar
- [CRW Web Scraper](https://affpapa.org/en/nevibecode/mcp/crw-web-scraper.md) — MCP server · ★ 1181
- [web-crawler](https://affpapa.org/en/nevibecode/skills/web-crawler.md) — skill · 3900 installs
- [tavily-crawl](https://affpapa.org/en/nevibecode/skills/tavily-crawl.md) — skill · 17400 installs
- [doc-scraper](https://affpapa.org/en/nevibecode/mcp/doc-scraper.md) — MCP server · ★ 107
- [Browser Scraper](https://affpapa.org/en/nevibecode/mcp/browser-scraper.md) — MCP server · ★ 11
- [Firecrawl MCP Server](https://affpapa.org/en/nevibecode/mcp/firecrawl-mcp-server.md) — MCP server · ★ 7695

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md