spider — Claude Code skill for crawling and analysis
The spider skill crawls pages, collects links and page data, and uses Gemini for content and security analysis. Use it to inspect a single URL or follow links across a site.
no license 209 installs ★ 28 updated 2025-12-19
npx skills add johnlindquist/claude --skill spider -a claude-codeWhat spider does
The skill uses curl to fetch pages and headers, find links, extract text, and visit multiple URLs. It also covers reading sitemap entries, spotting API addresses in JavaScript, and tracking page changes with a checksum. Shell commands handle the basic scans.
For content analysis and security checks, it sends prompts to Gemini with the gemini command. That workflow needs google-generativeai and the GEMINI_API_KEY environment variable. puppeteer and beautifulsoup4 are listed for more involved HTML parsing; without the Gemini key, the AI analysis steps cannot run.
Security checks examine headers, scripts, forms, and cookie settings in the supplied page material. The skill also advises checking robots.txt, limiting request frequency, and reviewing site terms before crawling.
What it helps with
- collect page headers and links
- crawl sitemap URLs and read page titles
- find API addresses in JavaScript
- inspect page material for security issues
- track page changes with a checksum
Example prompts for the agent
- Scan this page and collect its headers, links, and script references.
- Read the site's sitemap and gather titles from its pages.
- Check this page's HTML and headers for security issues.
How to install spider
1Skills CLI (Vercel)
npx skills add johnlindquist/claude --skill spider -a claude-code2Manually: put the files in place
~/.claude/skills/spider/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/spider/
1Skills CLI (Vercel)
npx skills add johnlindquist/claude --skill spider -a codex2Manually: put the files in place
.agents/skills/spider/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add johnlindquist/claude --skill spider -a cursor2Manually: put the files in place
.cursor/skills/spider/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install johnlindquist/claude skills/spider2Manually: put the files in place
.github/skills/spider/SKILL.md1Skills CLI (Vercel)
npx skills add johnlindquist/claude --skill spider -a gemini-cli2Platform CLI
gemini skills install https://github.com/johnlindquist/claude --path skills/spiderManually: put the files in place
.devin/skills/spider/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add johnlindquist/claude --skill spider -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about spider
How do I install spider in Claude Code?
Run npx skills add johnlindquist/claude --skill spider -a claude-code. The skill file is at ~/.claude/skills/spider/SKILL.md.
What does the spider skill do?
It helps fetch and crawl web pages, extract links and data, and analyze site content, including security checks.
What is required for Gemini analysis?
The workflow uses the gemini command, the google-generativeai package, and the GEMINI_API_KEY environment variable. HTML parsing options include puppeteer and beautifulsoup4.