Skip to main content
skill CX Codex

blocked-page-recovery — Claude Code skill: page recovery and install

blocked-page-recovery is a Claude Code skill that searches for copies of pages blocked by 403, 429, paywalls, or bot checks. It recovers research material while keeping source dates clear.

MIT 167 installs ★ 265,216 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-code
Open on GitHub

What blocked-page-recovery does

The skill checks routes in sequence: Wayback Machine, archive.today, Jina Reader when JINA_API_KEY is set, then API or RSS options on the same host. A real browser is the final fallback. The bundled scripts/recover_page.py script runs the checks and reports where the recovered copy came from.

It validates page content rather than trusting an HTTP success code, since a rate-limit screen can look like a valid response. Keep the date attached to an archived copy; it does not establish what the live page says now. For current prices or availability, an old snapshot is context, not an answer. Do not route cookies or authorization headers through proxy relays.

What it helps with

  • Recover an archived copy after a 403 or 429 response
  • Find a paywalled article in an available archive
  • Check a site's API or feed when its HTML is blocked
  • Reject redirect stubs that masquerade as fetched pages

Example prompts for the agent

  • Recover this page from an archive if the site returns 403.
  • Try the available routes and include the snapshot date in the result.
  • After repeated HTML blocks, look for an API or RSS feed on the same host.

How to install blocked-page-recovery

1Skills CLI (Vercel)

npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-code

2Manually: put the files in place

~/.claude/skills/blocked-page-recovery/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/blocked-page-recovery/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about blocked-page-recovery

How do I install blocked-page-recovery in Claude Code?

Run npx skills add NousResearch/hermes-agent --skill blocked-page-recovery -a claude-code. The skill file is at ~/.claude/skills/blocked-page-recovery/SKILL.md.

What does blocked-page-recovery do?

It searches archives, Jina Reader, and API or feed routes for a page that could not be fetched directly. It also checks that a response contains the page rather than a misleading interstitial.

Does Jina Reader require an API key?

Yes. Set JINA_API_KEY for that route; otherwise the skill skips Jina Reader.

Source text

SKILL.md · MIT
---
name: blocked-page-recovery
description: "Use when a fetch fails: 403/429, paywall, WAF, bot wall."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]
    related_skills: [grounded-citations]
---

# Blocked-Page Recovery

When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall,
or a bot-detection interstitial — don't give up and don't loop on the same
URL. Third-party services often hold a **copy** of the page. Work down this
ladder, cheapest first.

## The ladder

```
1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
2. archive.today    — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
4. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser     — browser tool as the last, most expensive resort
```

Run it in one shot with the bundled script:

```bash
python3 scripts/recover_page.py "https://example.com/blocked-article" --json
```

The script tries each route in order, validates every body (see "Fake
successes" below), and prints the first genuine hit with its provenance.

## Provenance discipline (non-negotiable)

Every recovered copy carries a provenance you MUST preserve when citing:

| Route | Provenance | How to cite |
|-------|-----------|-------------|
| Wayback / archive.today | `snapshot` | Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale. |
| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |
| Live fetch / browser | `live` | Cite normally. |

If the user needs *current* data (prices, availability, breaking news), a
snapshot is context, not an answer — say so explicitly and note its age.

## Manual routes

### 1. Wayback Machine (best provenance, try first)

```bash
# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url
```

For enumerating many snapshots (or recovering deleted pages), the CDX index:

```bash
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"
```

CDX intermittently returns 503 under load — if it does, fall back to the
`available` API; don't retry-hammer it.

Works for: any publicly crawled URL. Fails for: robots-blocked sites,
never-crawled URLs, JS-only SPAs (snapshots don't render).

### 2. archive.today (paywalls, deleted content)

User-submitted archives — often has paywalled news articles Wayback lacks.
Rate-limits aggressively (429) and rotates domains, so iterate:

```bash
for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o ~/.hermes/cache/scratch/page.html \
    -w "%{http_code}" && break
done
```

**Validate the body, not the status code** — a 429 still ships several KB of
rate-limit HTML that looks like a success to a size check alone.

### 3. Jina Reader (requires JINA_API_KEY)

`r.jina.ai` re-renders the live page in a real browser server-side and
returns markdown. Anonymous access is dead (401 → Turnstile); a key is
required:

```bash
curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"
```

Handles JS SPAs that archives can't. Skip this route entirely when the env
var is unset.

### 4. API-first pivot

WAFs protect the HTML surface far more aggressively than the data endpoints
behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and
look for:

- `/api/...`, `/graphql`, or `.json` variants of the page URL
- An RSS/Atom feed (`/feed`, `/rss`, `<link rel="alternate">` in any copy
  you did recover)
- A sitemap (`/sitemap.xml`) revealing canonical URLs that may not be gated

## Fake successes — routes that LIE

These return HTTP 200 with a plausible body that is NOT the page. The script
rejects them automatically; reject them manually too:

- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`
  returns 200 + tens of KB, but it's a Google Search interstitial with a JS
  redirect, not a cache. Never use it.
- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte
  `<title>Redirecting</title>` meta-refresh stub pointing back at the
  original (blocked) URL. Treating that as success creates a fetch loop.
- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for
  the target's actual content (title words, expected strings), not just size.

Detection heuristics the script applies: body under a per-route byte floor;
meta-refresh/JS-redirect stubs whose target is the original host; interstitial
titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").

## Proxy relays: don't

Generic "web proxy" relays are man-in-the-middle by construction. Never send
cookies or Authorization headers through one, and don't use them for anything
the user will rely on — provenance is unverifiable. Prefer archives, which at
least timestamp their copies.

© Hermes Agent · MIT · NousResearch/hermes-agent

Similar skills

All skills for Browser and scraping →

From the same set61

The whole set →
godmode godmode is a Claude Code skill with three prompt-level routes past model safety filters: templates, word obfuscation, and querying a model pool through OpenRouter. CX Codex +1 214installs competitor-news-monitor competitor-news-monitor — a skill that watches a fixed list of companies and sends a digest of material news, with primary-source links behind each item. CX Codex +1 204installs email-inbox-triage email-inbox-triage sorts mailbox threads by action, drafts replies in the user’s voice, and waits for approval before changing mail. CX Codex +1 178installs document-to-action-items document-to-action-items is a Claude Code skill that extracts obligations, deadlines, and tasks after document extraction, tracing each to its source file and page. CX Codex +1 173installs weekly-review-planning weekly-review-planning — a skill for Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode that runs a structured weekly reset across tasks, calendar, and notes. CX Codex +1 163installs meeting-action-items meeting-action-items — a Claude Code skill that turns meeting notes or a transcript into decisions, owners, and tickets. Run it after a call to draft the follow-up package. CX Codex +1 162installs sdlc-review sdlc-review reviews a task an implementer handed to the Kanban review lane, then approves it, sends it back with findings, or escalates it to a human. CX Codex +1 162installs product-price-monitor product-price-monitor tracks all-in prices for goods, flights, or hotels and triggers alerts upon hitting target thresholds. Filters repeated notifications. CX Codex +1 160installs

Project sponsors

Companies that keep this analytics open