Skip to main content
skill Claude Code CX Codex Cursor GitHub Copilot

firecrawl-knowledge-base — Claude Code skill: how to install

This skill crawls web pages with Firecrawl and structures them into local Markdown files. It helps prepare clean RAG chunks or fine-tuning datasets.

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add firecrawl/firecrawl-workflows --skill firecrawl-knowledge-base -a claude-code
Open on GitHub

What firecrawl-knowledge-base does

The skill turns web sources or topics into clean markdown documents. It checks your context first and asks up to three clarifying questions only if the target source or output mode is missing. The crawler maps documentation sites, searches thematic corpora, and strips away navigation clutter while keeping code samples and tables intact.

Output files land in .firecrawl/<hostname>/<path>/index.md. The workflow supports four modes: reference notes with sources.json, RAG chunks with manifest.json, dataset files in training-data.jsonl, or a full documentation mirror. When collecting large portals, the agent can split tasks across parallel sub-agents for separate sections.

A valid FIRECRAWL_API_KEY is required for hosted requests. You will hit an authentication error immediately if this environment variable is missing in your terminal session.

What it helps with

  • build a local markdown mirror from online documentation
  • generate chunk files and manifest.json for RAG pipelines
  • prepare training-data.jsonl files for model fine-tuning
  • extract web articles while preserving tables and code blocks

Example prompts for the agent

  • Scrape the official documentation site into clean markdown chunks for our RAG index.
  • Crawl the tutorial section and save the reference mirror with sources.json.
  • Extract training data from this topic and format it as training-data.jsonl.

How to install firecrawl-knowledge-base

1Skills CLI (Vercel)

npx skills add firecrawl/firecrawl-workflows --skill firecrawl-knowledge-base -a claude-code

2Manually: put the files in place

~/.claude/skills/firecrawl-knowledge-base/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/firecrawl-knowledge-base/

From the same set6

The whole set →

Similar

browser-trace browserbase Attaches a second CDP client to a browser session that is already running, records the whole DevTools stream to disk, and splits it so you can search a failed run. Claude Code CX Codex Cursor +1 3,000installs fetch browserbase Retrieves HTML or JSON from static web pages via Browserbase Fetch API without starting a browser session. Useful for fast scraping and header inspection. Claude Code CX Codex Cursor +1 3,000installs browser-mcp-agent antibrow Gives an agent a real browser through MCP calls: navigate, click, fill forms, screenshot, run JS. A named profile keeps the session logged in between runs. Claude Code CX Codex Cursor +1 105,700installs agent-browser vercel-labs A CLI skill for browser control: it opens pages, fills forms, clicks buttons, and tests web apps. Use it when an agent needs to act inside a website. Claude Code CX Codex Cursor +1 1,200,000installs webapp-testing official anthropics This skill tests local web apps with Playwright, helping agents debug UI behavior, capture screenshots, and inspect browser logs. Claude Code Claude.ai 186,758installs computer-use stablyai This skill drives a visible app window through orca computer: accessibility tree, clicks, typing, screenshots. Reach for it when the job sits behind a GUI. Claude Code CX Codex Cursor +1 239,000installs browser-use browser-use Hands the agent a real Chrome over CDP: clicks, typing, screenshots, and pages that need JS or your logged-in session. Claude Code CX Codex Cursor +1 98,100installs last30days mvanhorn Pulls posts and engagement on any topic from the last 30 days across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub and the web. Useful when you want what people actually said, not what a blog summarized. Claude Code CX Codex Cursor +1 52,400installs

Questions about firecrawl-knowledge-base

How to install firecrawl-knowledge-base in Claude Code?

Run npx skills add firecrawl/firecrawl-workflows --skill firecrawl-knowledge-base -a claude-code in your terminal. You can also place the configuration file directly at ~/.claude/skills/firecrawl-knowledge-base/SKILL.md.

What does firecrawl-knowledge-base do?

It automates scraping web sources through Firecrawl into organized markdown formats. The workflow strips site navigation while preserving tables, code snippets, and source URLs.

What is required before running this workflow?

You must provide a FIRECRAWL_API_KEY environment variable. The skill relies on this key to send hosted crawling and scraping requests.

Source text

SKILL.md · ISC
---
name: firecrawl-knowledge-base
description: Build a knowledge base from web content with Firecrawl. Use for local reference docs, RAG-ready chunks, fine-tuning datasets, documentation mirrors, topic corpora, or LLM-ready markdown organized from web sources.
license: ISC
metadata:
  author: firecrawl
  version: "0.1.0"
  homepage: https://www.firecrawl.dev
  source: https://github.com/firecrawl/firecrawl-workflows
inputs:
  - name: FIRECRAWL_API_KEY
    description: Firecrawl API key for hosted Firecrawl requests.
    required: true
---

# Firecrawl Knowledge Base

Use this to turn URLs or topics into organized LLM-ready content.

## Onboarding Interview

Infer the source, goal, depth, and output location from context. If the source and goal are clear, proceed immediately.

Ask at most 1-3 concise questions only if blocked, such as the source URL/topic, whether the output is reference/RAG/training/docs, or training format if training is requested.

## Firecrawl Collection Plan

Use Firecrawl map for documentation sites, search for topic-based corpora, scrape pages into markdown, and preserve code examples and tables.

For files, follow the Firecrawl download-style convention:

```text
.firecrawl/
  <hostname>/
    <path>/
      index.md
```

## Parallel Work

If appropriate, use sub-agents or equivalent parallel task runners:

- one docs section per researcher
- official docs, tutorials, community discussions, and references by source type
- source scraping vs chunk generation vs manifest generation

## Output Modes

- Reference: markdown files, `index.md`, and `sources.json`.
- RAG: markdown files plus chunk files and `manifest.json`.
- Training: scraped source files plus `training-data.jsonl` and `training-metadata.json`.
- Docs mirror: complete markdown mirror with a table of contents.

## Final Deliverable

```markdown
# Knowledge Base: [Source]

## Summary
[What was collected and why]

## Output Structure
[Files/directories created]

## Coverage
[Sections, source types, counts]

## Usage Notes
[How to use in RAG, docs, training, or agent context]

## Sources
[URLs collected]

## Rerun Inputs
workflow: firecrawl-knowledge-base
source: [url/topic]
goal: [reference/rag/train/docs]
depth: [quick/thorough/exhaustive]
output_dir: [.firecrawl/]
```

## Quality Bar

- Preserve code examples and formatting.
- Remove boilerplate navigation where possible.
- Include source URLs in frontmatter or metadata.

© firecrawl · ISC · firecrawl/firecrawl-workflows

See also

Project sponsors

Companies that keep this analytics open
Gold sponsor

Impressions, clicks and code copies — open statistics. We keep the numbers public for the same reason NeBlask itself is public. Become a sponsor