blocked-page-recovery — Claude Code skill: page recovery and install
blocked-page-recovery is a Claude Code skill that searches for copies of pages blocked by 403, 429, paywalls, or bot checks. It recovers research material while keeping source dates clear.
MIT 167 installs ★ 265,216 updated 2026-10-25
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-codeWhat blocked-page-recovery does
The skill checks routes in sequence: Wayback Machine, archive.today, Jina Reader when JINA_API_KEY is set, then API or RSS options on the same host. A real browser is the final fallback. The bundled scripts/recover_page.py script runs the checks and reports where the recovered copy came from.
It validates page content rather than trusting an HTTP success code, since a rate-limit screen can look like a valid response. Keep the date attached to an archived copy; it does not establish what the live page says now. For current prices or availability, an old snapshot is context, not an answer. Do not route cookies or authorization headers through proxy relays.
What it helps with
- Recover an archived copy after a 403 or 429 response
- Find a paywalled article in an available archive
- Check a site's API or feed when its HTML is blocked
- Reject redirect stubs that masquerade as fetched pages
Example prompts for the agent
- Recover this page from an archive if the site returns 403.
- Try the available routes and include the snapshot date in the result.
- After repeated HTML blocks, look for an API or RSS feed on the same host.
How to install blocked-page-recovery
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-code2Manually: put the files in place
~/.claude/skills/blocked-page-recovery/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/blocked-page-recovery/
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a codex2Manually: put the files in place
.agents/skills/blocked-page-recovery/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a cursor2Manually: put the files in place
.cursor/skills/blocked-page-recovery/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install NousResearch/hermes-agent skills/web/blocked-page-recovery2Manually: put the files in place
.github/skills/blocked-page-recovery/SKILL.md1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a gemini-cli2Platform CLI
gemini skills install https://github.com/NousResearch/hermes-agent --path skills/web/blocked-page-recoveryManually: put the files in place
.devin/skills/blocked-page-recovery/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about blocked-page-recovery
How do I install blocked-page-recovery in Claude Code?
Run npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-code. The skill file is at ~/.claude/skills/blocked-page-recovery/SKILL.md.
What does blocked-page-recovery do?
It searches archives, Jina Reader, and API or feed routes for a page that could not be fetched directly. It also checks that a response contains the page rather than a misleading interstitial.
Does Jina Reader require an API key?
Yes. Set JINA_API_KEY for that route; otherwise the skill skips Jina Reader.
Source text
SKILL.md · MIT
---
name: blocked-page-recovery
description: "Use when a fetch fails: 403/429, paywall, WAF, bot wall."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]
related_skills: [grounded-citations]
---
# Blocked-Page Recovery
When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall,
or a bot-detection interstitial — don't give up and don't loop on the same
URL. Third-party services often hold a **copy** of the page. Work down this
ladder, cheapest first.
## The ladder
```
1. Wayback Machine — archive.org "available" API (snapshot + timestamp)
2. archive.today — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader — only if JINA_API_KEY is set (live server-side render)
4. API-first pivot — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser — browser tool as the last, most expensive resort
```
Run it in one shot with the bundled script:
```bash
python3 scripts/recover_page.py "https://example.com/blocked-article" --json
```
The script tries each route in order, validates every body (see "Fake
successes" below), and prints the first genuine hit with its provenance.
## Provenance discipline (non-negotiable)
Every recovered copy carries a provenance you MUST preserve when citing:
| Route | Provenance | How to cite |
|-------|-----------|-------------|
| Wayback / archive.today | `snapshot` | Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale. |
| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |
| Live fetch / browser | `live` | Cite normally. |
If the user needs *current* data (prices, availability, breaking news), a
snapshot is context, not an answer — say so explicitly and note its age.
## Manual routes
### 1. Wayback Machine (best provenance, try first)
```bash
# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url
```
For enumerating many snapshots (or recovering deleted pages), the CDX index:
```bash
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"
```
CDX intermittently returns 503 under load — if it does, fall back to the
`available` API; don't retry-hammer it.
Works for: any publicly crawled URL. Fails for: robots-blocked sites,
never-crawled URLs, JS-only SPAs (snapshots don't render).
### 2. archive.today (paywalls, deleted content)
User-submitted archives — often has paywalled news articles Wayback lacks.
Rate-limits aggressively (429) and rotates domains, so iterate:
```bash
for d in archive.ph archive.md archive.li archive.is; do
curl -sL --max-time 20 "https://$d/newest/{URL}" -o ~/.hermes/cache/scratch/page.html \
-w "%{http_code}" && break
done
```
**Validate the body, not the status code** — a 429 still ships several KB of
rate-limit HTML that looks like a success to a size check alone.
### 3. Jina Reader (requires JINA_API_KEY)
`r.jina.ai` re-renders the live page in a real browser server-side and
returns markdown. Anonymous access is dead (401 → Turnstile); a key is
required:
```bash
curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"
```
Handles JS SPAs that archives can't. Skip this route entirely when the env
var is unset.
### 4. API-first pivot
WAFs protect the HTML surface far more aggressively than the data endpoints
behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and
look for:
- `/api/...`, `/graphql`, or `.json` variants of the page URL
- An RSS/Atom feed (`/feed`, `/rss`, `<link rel="alternate">` in any copy
you did recover)
- A sitemap (`/sitemap.xml`) revealing canonical URLs that may not be gated
## Fake successes — routes that LIE
These return HTTP 200 with a plausible body that is NOT the page. The script
rejects them automatically; reject them manually too:
- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`
returns 200 + tens of KB, but it's a Google Search interstitial with a JS
redirect, not a cache. Never use it.
- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte
`<title>Redirecting</title>` meta-refresh stub pointing back at the
original (blocked) URL. Treating that as success creates a fetch loop.
- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for
the target's actual content (title words, expected strings), not just size.
Detection heuristics the script applies: body under a per-route byte floor;
meta-refresh/JS-redirect stubs whose target is the original host; interstitial
titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").
## Proxy relays: don't
Generic "web proxy" relays are man-in-the-middle by construction. Never send
cookies or Authorization headers through one, and don't use them for anything
the user will rely on — provenance is unverifiable. Prefer archives, which at
least timestamp their copies.
© Hermes Agent · MIT · NousResearch/hermes-agent