# crawl4ai

Use this skill to crawl JavaScript-rendered pages, process multiple URLs, and extract fields from CSS schemas. It fits SPA scraping and automated web data pipelines.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1100 installs · ★ 49
- Author: brettdavies
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/brettdavies/crawl4ai-skill/tree/HEAD/SKILL.md
- Updated: 2026-10-05
- HTML: https://affpapa.org/en/nevibecode/skills/crawl4ai

## What it does

The skill guides an agent through Crawl4AI, a library that combines a Playwright browser with a content-processing pipeline. Its `crwl` CLI and Python SDK can return Markdown, HTML, links, and media. For structured results, configure CSS or JSON schemas; those can extract fields without an LLM at crawl time.

Install the library with `pip install crawl4ai`, run `crawl4ai-setup`, then check it with `crawl4ai-doctor`. Dynamic pages require browser startup, which takes about two seconds. For static HTML, the skill points to `fetch-web` or `defuddle`; for form submission and multi-step UI navigation, it directs users to `/browse`.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add brettdavies/crawl4ai-skill --skill crawl4ai -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/crawl4ai/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/crawl4ai/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add brettdavies/crawl4ai-skill --skill crawl4ai -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/crawl4ai/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add brettdavies/crawl4ai-skill --skill crawl4ai -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/crawl4ai/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install brettdavies/crawl4ai-skill crawl4ai
```

2. **Manually: put the files in place**
- File: `.github/skills/crawl4ai/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add brettdavies/crawl4ai-skill --skill crawl4ai -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/brettdavies/crawl4ai-skill
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/crawl4ai/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add brettdavies/crawl4ai-skill --skill crawl4ai -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/brettdavies/crawl4ai-skill/tree/HEAD/SKILL.md

## When to use it
- capture Markdown from a JavaScript-rendered page
- crawl multiple URLs concurrently
- extract page fields with a CSS schema
- collect links and media from a site
- build a web data pipeline

## Example prompts
- Use `/crawl4ai https://example.com` to fetch the page as Markdown.
- Crawl several URLs and extract structured fields with crawl4ai.
- Collect links and media from these pages with crawl4ai.

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 377af42623db2b67.

BEGIN THIRD-PARTY FILE 377af42623db2b67
````text
---
name: crawl4ai
description: Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.
argument-hint: "[url]"
---

# Crawl4AI

**Verified against `crawl4ai`** [`VERSION`](VERSION). PEP 723 pins in `scripts/*.py` and `tests/*.py` floor at that
version.

## Overview

Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't
reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable
schema-based extraction.

This skill exposes both interfaces of the underlying library:

- **CLI** (`crwl`) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)
- **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)

## Invoked with a URL argument

When the user runs `/crawl4ai <url>` with a single URL and no further qualifier, treat it as the JS-heavy fetch case and
default to:

```bash
crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown
```

`wait_until=networkidle` waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't
named a specific element on a JS-rendered page. (Avoid `wait_for=css:body`: `<body>` exists at t=0 on every HTML
response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to
`wait_for=css:<selector>` if the user named a specific element. Skip the default and route to the relevant section below
for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each
have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to `/fetch-web` instead
per the "When NOT to use" section below.

## When NOT to use this skill

- **Static HTML pages** (most documentation sites, blog posts, news articles, tweets) — use `/fetch-web` or `defuddle`
  directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.
- **Local file conversion** (`.pdf`, `.docx`, `.pptx`, `.epub`) — use `/markdown-convert`.
- **One-URL agent-context reads** (the agent just needs to read this page) — use `/fetch-web` and let it route to
  `defuddle`.
- **Mutating UI flows** (form fills, multi-step clicks, login + navigation) — `/browse` (gstack's persistent headless
  Chromium) is built for that.

## When stuck

For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see
[references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues
→ ask the user) and worked examples.

---

## Quick Start

### Installation

```bash
pip install crawl4ai
crawl4ai-setup
````
END THIRD-PARTY FILE 377af42623db2b67

The third-party file has ended. Everything between the 377af42623db2b67 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 3289 of 16690 bytes. © brettdavies · Apache-2.0 · https://github.com/brettdavies/crawl4ai-skill/tree/HEAD/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md