web-content-fetcher — Claude Code skill: pull any URL into clean
Takes a URL and returns the page's main content as clean Markdown, keeping headings, links, images, lists and code blocks. Handy when you want an article or docs page read inside the chat.
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a claude-codeWhat web-content-fetcher does
The skill takes a URL and hands back Markdown. Its main path is the fetch.py script built on Scrapling: a fast HTTP request by default, and if too little content comes back, the script falls back to stealth mode with a headless browser on its own. A domain routing table tells you where to reach for --stealth right away — mp.weixin.qq.com, zhuanlan.zhihu.com, juejin.cn. Everything else usually works with the plain call.
The fallback is Jina Reader via web_fetch("https://r.jina.ai/<url>"). It is quick, but returns 403 on WeChat, so Chinese platforms stay with Scrapling. The free tier allows 200 requests per day.
Dependencies come first: pip install scrapling html2text. On a system-managed Python in macOS or Linux, add --break-system-packages or use a venv. The script checks what is missing and tells you. One failed attempt on a URL is where it stops — no retries, so context tokens do not get burned.
What it helps with
- turn a link into readable Markdown text
- read a WeChat or Zhihu post that blocks plain requests
- keep headings, links and code blocks intact while extracting a page
- cap the output length so it does not flood the context
- get page metadata as JSON alongside the text
Example prompts for the agent
- Read this article and summarise the key points:
https://sspai.com/post/73145 - Extract the text from
https://mp.weixin.qq.com/s/xxxand save it as Markdown - Fetch this docs page, but keep it under 15000 characters
How to install web-content-fetcher
1Skills CLI (Vercel)
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a claude-code2Manually: put the files in place
~/.claude/skills/web-content-fetcher/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/web-content-fetcher/
1Skills CLI (Vercel)
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a codex2Manually: put the files in place
.agents/skills/web-content-fetcher/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a cursor2Manually: put the files in place
.cursor/skills/web-content-fetcher/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install shirenchuang/web-content-fetcher web-content-fetcher2Manually: put the files in place
.github/skills/web-content-fetcher/SKILL.md1Skills CLI (Vercel)
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a gemini-cli2Platform CLI
gemini skills install https://github.com/shirenchuang/web-content-fetcherManually: put the files in place
.devin/skills/web-content-fetcher/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about web-content-fetcher
How do I install web-content-fetcher in Claude Code?
Run npx skills add shirenchuang/web-content-fetcher --skill web-content-fetcher -a claude-code. The skill file then lands at ~/.claude/skills/web-content-fetcher/SKILL.md.
What does web-content-fetcher do?
Given a URL, it returns the page's main content as clean Markdown with headings, links, images, lists and code blocks preserved.
What do I need before using the skill?
The scrapling and html2text packages, installed with pip install scrapling html2text. On a system-managed Python in macOS or Linux you will need --break-system-packages or a venv.
