Skip to main content
skill CX Codex

spider — Claude Code skill for crawling and analysis

The spider skill crawls pages, collects links and page data, and uses Gemini for content and security analysis. Use it to inspect a single URL or follow links across a site.

no license 209 installs ★ 28 updated 2025-12-19

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add johnlindquist/claude --skill spider -a claude-code
Open on GitHub

What spider does

The skill uses curl to fetch pages and headers, find links, extract text, and visit multiple URLs. It also covers reading sitemap entries, spotting API addresses in JavaScript, and tracking page changes with a checksum. Shell commands handle the basic scans.

For content analysis and security checks, it sends prompts to Gemini with the gemini command. That workflow needs google-generativeai and the GEMINI_API_KEY environment variable. puppeteer and beautifulsoup4 are listed for more involved HTML parsing; without the Gemini key, the AI analysis steps cannot run.

Security checks examine headers, scripts, forms, and cookie settings in the supplied page material. The skill also advises checking robots.txt, limiting request frequency, and reviewing site terms before crawling.

What it helps with

  • collect page headers and links
  • crawl sitemap URLs and read page titles
  • find API addresses in JavaScript
  • inspect page material for security issues
  • track page changes with a checksum

Example prompts for the agent

  • Scan this page and collect its headers, links, and script references.
  • Read the site's sitemap and gather titles from its pages.
  • Check this page's HTML and headers for security issues.

How to install spider

1Skills CLI (Vercel)

npx skills add johnlindquist/claude --skill spider -a claude-code

2Manually: put the files in place

~/.claude/skills/spider/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/spider/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about spider

How do I install spider in Claude Code?

Run npx skills add johnlindquist/claude --skill spider -a claude-code. The skill file is at ~/.claude/skills/spider/SKILL.md.

What does the spider skill do?

It helps fetch and crawl web pages, extract links and data, and analyze site content, including security checks.

What is required for Gemini analysis?

The workflow uses the gemini command, the google-generativeai package, and the GEMINI_API_KEY environment variable. HTML parsing options include puppeteer and beautifulsoup4.

Similar skills

All skills for Browser and scraping →

From the same set7

Project sponsors

Companies that keep this analytics open