# web-content-fetcher

Takes a URL and returns the page's main content as clean Markdown, keeping headings, links, images, lists and code blocks. Handy when you want an article or docs page read inside the chat.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 3000 installs · ★ 611
- Author: shirenchuang
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/shirenchuang/web-content-fetcher/tree/HEAD/SKILL.md
- Updated: 2026-03-28
- HTML: https://affpapa.org/en/nevibecode/skills/web-content-fetcher

## What it does

The skill takes a URL and hands back Markdown. Its main path is the `fetch.py` script built on Scrapling: a fast HTTP request by default, and if too little content comes back, the script falls back to stealth mode with a headless browser on its own. A domain routing table tells you where to reach for `--stealth` right away — `mp.weixin.qq.com`, `zhuanlan.zhihu.com`, `juejin.cn`. Everything else usually works with the plain call.

The fallback is Jina Reader via `web_fetch("https://r.jina.ai/<url>")`. It is quick, but returns 403 on WeChat, so Chinese platforms stay with Scrapling. The free tier allows 200 requests per day.

Dependencies come first: `pip install scrapling html2text`. On a system-managed Python in macOS or Linux, add `--break-system-packages` or use a venv. The script checks what is missing and tells you. One failed attempt on a URL is where it stops — no retries, so context tokens do not get burned.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/web-content-fetcher/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/web-content-fetcher/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/web-content-fetcher/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/web-content-fetcher/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install shirenchuang/web-content-fetcher web-content-fetcher
```

2. **Manually: put the files in place**
- File: `.github/skills/web-content-fetcher/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/shirenchuang/web-content-fetcher
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/web-content-fetcher/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/shirenchuang/web-content-fetcher/tree/HEAD/SKILL.md

## When to use it
- turn a link into readable Markdown text
- read a WeChat or Zhihu post that blocks plain requests
- keep headings, links and code blocks intact while extracting a page
- cap the output length so it does not flood the context
- get page metadata as JSON alongside the text

## Example prompts
- Read this article and summarise the key points: `https://sspai.com/post/73145`
- Extract the text from `https://mp.weixin.qq.com/s/xxx` and save it as Markdown
- Fetch this docs page, but keep it under 15000 characters

## Similar
- [agent-browser](https://affpapa.org/en/nevibecode/skills/vercel-labs-agent-browser.md) — skill · 1200000 installs
- [webapp-testing](https://affpapa.org/en/nevibecode/skills/webapp-testing.md) — skill · 186758 installs
- [computer-use](https://affpapa.org/en/nevibecode/skills/computer-use.md) — skill · 239400 installs
- [browser-use](https://affpapa.org/en/nevibecode/skills/browser-use.md) — skill · 98200 installs
- [playwright-cli](https://affpapa.org/en/nevibecode/skills/playwright-cli.md) — skill · 195700 installs
- [last30days](https://affpapa.org/en/nevibecode/skills/last30days.md) — skill · 53000 installs

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md