gallery-scraper — Claude Code skill: bulk-download gated gallery images
gallery-scraper — a Claude skill that collects full-size image URLs from a gallery behind a login and downloads them in bulk through an attached browser tab.
MIT ★ 239 updated 2026-08-30
npx skills add jdrhyne/agent-skills --skill gallery-scraper -a claude-codeWhat gallery-scraper does
You attach the tab yourself: sign in to the gallery in Chrome, open the page you want, then click the OpenClaw Browser Relay button until the badge reads ON. The agent then looks for where full-size files live: data-max, data-src, data-full, lightbox markup. Snippets built on querySelectorAll come with the skill, so you are not writing them from scratch.
Pages are walked one at a time, and several galleries can be pulled through a hidden iframe with a load timeout. Downloading goes either as a curl loop with a Referer header and at most eight parallel jobs, or as a Python script on requests and ThreadPoolExecutor. Before the batch starts, the skill advises testing the CDN with curl -I, with and without Referer.
No Chrome with the OpenClaw Browser Relay extension and an attached tab with the badge on means no run at all, because that session is the only place the login exists. A 403 usually calls for Referer or cookies from document.cookie; rate limiting calls for fewer parallel downloads. Cookies and tokens never land in the output files, and the skill stays inside the gallery you picked.
What it helps with
- pull every image from a gallery that only opens after sign-in
- recover full-size URLs hidden behind thumbnails and lightboxes
- walk a multi-page gallery and gather the links into one list
- download a batch in parallel without tripping the rate limit
- work out why the CDN returns 403 and add the right
Referer
Example prompts for the agent
- Scrape every image on this gallery page, I've attached the tab
- Gather the full-size links from all pages and write them to
urls.txt - Check whether the CDN serves files without a
Referer, then download the gallery into~/Downloads/gallery_name
How to install gallery-scraper
1Skills CLI (Vercel)
npx skills add jdrhyne/agent-skills --skill gallery-scraper -a claude-code2Manually: put the files in place
~/.claude/skills/gallery-scraper/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/gallery-scraper/
1Skills CLI (Vercel)
npx skills add jdrhyne/agent-skills --skill gallery-scraper -a codex2Manually: put the files in place
.agents/skills/gallery-scraper/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add jdrhyne/agent-skills --skill gallery-scraper -a cursor2Manually: put the files in place
.cursor/skills/gallery-scraper/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install jdrhyne/agent-skills clawdbot/gallery-scraper/SKILL.md2Manually: put the files in place
.github/skills/gallery-scraper/SKILL.md1Skills CLI (Vercel)
npx skills add jdrhyne/agent-skills --skill gallery-scraper -a gemini-cli2Platform CLI
gemini skills install https://github.com/jdrhyne/agent-skills --path clawdbot/gallery-scraperManually: put the files in place
.devin/skills/gallery-scraper/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about gallery-scraper
How do I install gallery-scraper in Claude Code?
Run npx skills add jdrhyne/agent-skills --skill gallery-scraper -a claude-code. The skill then sits at ~/.claude/skills/gallery-scraper/SKILL.md.
What does gallery-scraper do?
It walks gallery pages in an attached tab, finds the URLs of full-size images, and saves them as files in a local folder.
What do I need before it works?
Chrome with the OpenClaw Browser Relay extension, a signed-in session on the gallery site, and the tab attached so the badge shows ON.
Source text
SKILL.md · MIT
---
name: gallery-scraper
description: Bulk download images from login-protected gallery websites using an attached browser session. Use when asked to scrape, download, or save images from authenticated gallery pages, extract full-size images from thumbnails, or batch download from multi-page galleries.
permissions:
- exec: "Runs local download commands for the URL list gathered from the attached browser session."
- file_write: "Creates URL lists and downloaded image files in the user-approved output directory."
- network: "Uses the attached browser session and direct image downloads against the user-approved gallery domain."
---
# Gallery Scraper
Bulk download images from authenticated gallery websites via browser relay.
## Safety Boundaries
- Do not access gallery sites or user accounts that the user has not explicitly attached and authorized.
- Do not download beyond the selected gallery, profile, or page range without confirmation.
- Do not store cookies, tokens, or hidden form values in local output files.
- Do not keep retrying blocked downloads indefinitely; surface rate limits or auth failures instead.
## Prerequisites
- User must have Chrome with OpenClaw Browser Relay extension
- User must be logged into the target site
- User must attach the browser tab (click relay toolbar button, badge ON)
## Workflow
### 1. Attach Browser Tab
Ask user to:
1. Log into the gallery site in Chrome
2. Navigate to the target gallery/profile page
3. Click the OpenClaw Browser Relay toolbar button (badge shows ON)
### 2. Discover Image URL Pattern
Most gallery sites store full-size URLs in data attributes. Common patterns:
```javascript
// Extract via browser evaluate
() => {
// Try common patterns
const patterns = [
'img[data-max]', // data-max attribute
'img[data-src]', // lazy-load pattern
'img[data-full]', // full-size pattern
'a[data-lightbox] img', // lightbox galleries
'.gallery-item img' // generic gallery
];
for (const sel of patterns) {
const imgs = document.querySelectorAll(sel);
if (imgs.length > 0) {
return {
selector: sel,
count: imgs.length,
sample: imgs[0].outerHTML.substring(0, 200)
};
}
}
return null;
}
```
### 3. Extract Full-Size URLs
Once pattern identified, extract all URLs:
```javascript
// For data-max pattern (common)
() => Array.from(document.querySelectorAll('img[data-max]'))
.map(img => img.dataset.max)
// For thumbnail→full conversion (replace path segment)
() => Array.from(document.querySelectorAll('.gallery img'))
.map(img => img.src.replace('/thumb/', '/full/'))
```
### 4. Handle Pagination
Check for multiple pages:
```javascript
() => {
const pagination = document.querySelectorAll('.pagination a, [class*="page"] a');
return Array.from(pagination).map(a => ({text: a.textContent, href: a.href}));
}
```
Navigate to each page and collect URLs.
### 4b. Batch scrape multiple galleries (iframe trick)
When you need multiple galleries quickly and can’t automate CDP, you can load each gallery in a hidden iframe and extract `data-max` URLs:
```javascript
async () => {
const urls = [
'https://site.example/galleries/view/123',
'https://site.example/galleries/view/456'
];
const results = [];
for (const url of urls) {
const iframe = document.createElement('iframe');
iframe.style.position = 'fixed';
iframe.style.left = '-9999px';
iframe.style.width = '800px';
iframe.style.height = '600px';
iframe.src = url;
document.body.appendChild(iframe);
await new Promise((resolve, reject) => {
const t = setTimeout(() => reject(new Error('timeout load')), 20000);
iframe.onload = () => { clearTimeout(t); resolve(); };
});
const doc = iframe.contentDocument;
const start = Date.now();
let imgs = [];
while (Date.now() - start < 20000) {
imgs = Array.from(doc.querySelectorAll('img[data-max]')).map(i => i.dataset.max);
if (imgs.length) break;
await new Promise(r => setTimeout(r, 500));
}
results.push({ id: url.split('/').pop(), urls: imgs });
iframe.remove();
}
return results;
}
```
### 5. Check CDN Access
Test if CDN requires authentication or just Referer:
```bash
# Test direct access
curl -I "CDN_URL" 2>/dev/null | head -3
# Test with Referer
curl -I -H "Referer: https://SITE_DOMAIN/" "CDN_URL" 2>/dev/null | head -3
```
### 6. Bulk Download
Collect the URLs into a text file, then parallel download:
```bash
# Create output directory
mkdir -p ~/Downloads/gallery_name
# Download with Referer header (parallel)
cd ~/Downloads/gallery_name
while IFS= read -r url; do
filename=$(basename "$url")
curl -s -H "Referer: https://SITE_DOMAIN/" -o "$filename" "$url" &
[ $(jobs -r | wc -l) -ge 8 ] && wait -n
done < urls.txt
wait
```
**Python ThreadPool fallback (avoids shell quoting + wait -n issues):**
```python
import os
import requests
from concurrent.futures import ThreadPoolExecutor
outdir = os.path.expanduser('~/Downloads/gallery_name')
os.makedirs(outdir, exist_ok=True)
headers = {'Referer': 'https://SITE_DOMAIN/', 'User-Agent': 'Mozilla/5.0'}
with open('urls.txt') as f:
urls = [line.strip() for line in f if line.strip()]
def download(url):
filename = os.path.join(outdir, os.path.basename(url))
if os.path.exists(filename) and os.path.getsize(filename) > 0:
return
r = requests.get(url, headers=headers, timeout=60)
r.raise_for_status()
with open(filename, 'wb') as f:
f.write(r.content)
with ThreadPoolExecutor(max_workers=8) as ex:
for url in urls:
ex.submit(download, url)
```
## Handling Lock Buttons
Some galleries have "lock" buttons to reveal hidden content. Look for:
```javascript
// Find lock/unlock buttons
() => {
const locks = document.querySelectorAll(
'[class*="lock"], [class*="unlock"], ' +
'button[title*="lock"], .premium-unlock'
);
return Array.from(locks).map(el => ({
tag: el.tagName,
class: el.className,
text: el.innerText?.substring(0, 30)
}));
}
```
Click each lock button before extracting URLs.
## Output Organization
Optionally organize by gallery:
```bash
# Derive a gallery-specific folder name from the selected URL
mkdir -p "gallery_<id>"
```
## Troubleshooting
- **403 Forbidden**: Add Referer header or extract cookies from browser
- **Rate limited**: Reduce parallel downloads, add delays
- **Missing images**: Check for JavaScript-loaded content, may need scroll injection
- **Login required for CDN**: Extract session cookies via `document.cookie`
© jdrhyne · MIT · jdrhyne/agent-skills