# spider

The spider skill crawls pages, collects links and page data, and uses Gemini for content and security analysis. Use it to inspect a single URL or follow links across a site.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 209 installs · ★ 28 repo stars
- Author: johnlindquist
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/johnlindquist/claude/tree/HEAD/skills/spider/SKILL.md
- Updated: 2025-12-19
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/spider

## What it does

The skill uses `curl` to fetch pages and headers, find links, extract text, and visit multiple URLs. It also covers reading sitemap entries, spotting API addresses in JavaScript, and tracking page changes with a checksum. Shell commands handle the basic scans.

For content analysis and security checks, it sends prompts to Gemini with the `gemini` command. That workflow needs `google-generativeai` and the `GEMINI_API_KEY` environment variable. `puppeteer` and `beautifulsoup4` are listed for more involved HTML parsing; without the Gemini key, the AI analysis steps cannot run.

Security checks examine headers, scripts, forms, and cookie settings in the supplied page material. The skill also advises checking `robots.txt`, limiting request frequency, and reviewing site terms before crawling.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add johnlindquist/claude --skill spider -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/spider/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/spider/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add johnlindquist/claude --skill spider -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/spider/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add johnlindquist/claude --skill spider -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/spider/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install johnlindquist/claude skills/spider
```

2. **Manually: put the files in place**
- File: `.github/skills/spider/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add johnlindquist/claude --skill spider -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/johnlindquist/claude --path skills/spider
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/spider/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add johnlindquist/claude --skill spider -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/johnlindquist/claude/tree/HEAD/skills/spider/SKILL.md

## When to use it
- collect page headers and links
- crawl sitemap URLs and read page titles
- find API addresses in JavaScript
- inspect page material for security issues
- track page changes with a checksum

## Example prompts
- Scan this page and collect its headers, links, and script references.
- Read the site's sitemap and gather titles from its pages.
- Check this page's HTML and headers for security issues.

## Similar
- [Screaming Frog SEO Spider MCP Server (headless)](https://affpapa.org/en/nevibecode/mcp/screaming-frog-seo-spider-mcp-server-headless.md) — MCP server · ★ 97 repo stars
- [competitive-ads-extractor](https://affpapa.org/en/nevibecode/skills/competitive-ads-extractor.md) — skill · 1091 installs
- [scrape](https://affpapa.org/en/nevibecode/skills/scrape.md) — skill · 1002 installs
- [discovering-viral-youtube-videos-by-category](https://affpapa.org/en/nevibecode/skills/discovering-viral-youtube-videos-by-category.md) — skill · 1000 installs
- [finding-instagram-creators-by-location](https://affpapa.org/en/nevibecode/skills/finding-instagram-creators-by-location.md) — skill · 983 installs
- [he4rt-twitter-engagement-tracker](https://affpapa.org/en/nevibecode/skills/he4rt-twitter-engagement-tracker.md) — skill · 876 installs

From the same set: [johnlindquist/claude](https://affpapa.org/en/nevibecode/by/johnlindquist/claude.md) — 7 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
