# Web Content & URL Tools

Give coding agents tools to extract web page content, inspect site metadata, and check SEO, TLS, domains, and email authentication through a remote MCP connection.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI
- Author: avaloncompany
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/avaloncompany/web-content-mcp-server
- Updated: 2026-10-04
- HTML: https://affpapa.org/en/nevibecode/mcp/web-content-url-tools

## What it does

Web Content & URL Tools connects an AI coding agent to web inspection utilities. It converts page content into Markdown, retrieves metadata and document structure, identifies site technologies, follows redirects, and examines TLS certificates and security headers. Other tools inspect robots.txt, sitemaps, feeds, SEO issues, and broken links.

The server also checks domain and email authentication records, including SPF, DMARC, and DKIM. Identity diagnostics cover SAML metadata and OpenID Connect configuration. Developer utilities handle cron expressions, regex tests, color conversions, Unicode, and time zones. Hyperliquid market data is available as well. The server exposes 26 tools in total.

Connect to the hosted endpoint over HTTP instead of running the server locally. For Claude Code, use `claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp`. Calls are billed through an Apify account: successful requests cost $0.001 or $0.003, depending on the tool. Pages that render their content only in a browser may yield little readable text, and private or internal URLs are rejected.

## Requirements
- Remote server: `https://avalonai--web-content-url-tools-mcp.apify.actor/mcp`

## Install

### Claude Code (native platform)

**Platform CLI: remote server**
```bash
claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp
```

### Codex

**Platform CLI: remote server**
```bash
codex mcp add web-content-url-tools --url https://avalonai--web-content-url-tools-mcp.apify.actor/mcp
```

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "web-content-url-tools": {
            "url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
        }
    }
}
```

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "web-content-url-tools": {
            "type": "http",
            "url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
        }
    }
}
```

### Gemini CLI

**Platform CLI: remote server**
```bash
gemini mcp add -t http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp
```

## When to use it
- Convert a web page into Markdown for agent analysis
- Extract page metadata, headings, links, and images
- Inspect TLS certificates and security headers
- Audit on-page SEO and identify broken links
- Validate email authentication and SSO configuration

## Example prompts
- Turn this web page into Markdown and outline its main sections.
- Inspect the site's TLS certificate, redirect chain, and security headers.
- Audit this site for SEO issues and broken links.

## Similar
- [Scrapling MCP Server](https://affpapa.org/en/nevibecode/mcp/scrapling-mcp-server.md) — MCP server · ★ 89899
- [browser-use](https://affpapa.org/en/nevibecode/mcp/browser-use.md) — MCP server · ★ 119231
- [AIHawk](https://affpapa.org/en/nevibecode/mcp/aihawk.md) — MCP server · ★ 32260
- [Skyvern](https://affpapa.org/en/nevibecode/mcp/skyvern.md) — MCP server · ★ 23283
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13181
- [apify-mcp-server](https://affpapa.org/en/nevibecode/mcp/apify-mcp-server.md) — MCP server · ★ 13345

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md