# firecrawl-research

Extracts research topics from markdown notes, scrapes web sources via FireCrawl API, and saves structured summaries. Useful for academic writing and reference management.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 379 installs · ★ 400
- Author: glebis
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/glebis/claude-skills/tree/HEAD/firecrawl-research/SKILL.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/skills/firecrawl-research

## What it does

The skill automates web research via the FireCrawl API. Its Python script `scripts/firecrawl_research.py` parses markdown files for topics using two formats: markdown headings or bullet points tagged with `[research]` and `[search]`. For every topic found, it runs a FireCrawl search and extracts clean markdown content. It pulls up to 5 results per topic by default.

Output files are saved separately with timestamps in the name. Each file stores source URLs, titles, and full page texts. The skill also provides templates for Pandoc and MyST academic papers, alongside `scripts/generate_bibliography.py` to compile scraped sources into a BibTeX file.

Running the tool requires Python packages `requests` and `python-dotenv`, plus a `FIRECRAWL_API_KEY` set in the `.env` file. Because FireCrawl limits free accounts to 5 requests per minute, the script enforces a 12-second delay between queries. Processing twenty topics will take roughly 4 minutes.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add glebis/claude-skills --skill firecrawl-research -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/firecrawl-research/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/firecrawl-research/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add glebis/claude-skills --skill firecrawl-research -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/firecrawl-research/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add glebis/claude-skills --skill firecrawl-research -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/firecrawl-research/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install glebis/claude-skills firecrawl-research
```

2. **Manually: put the files in place**
- File: `.github/skills/firecrawl-research/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add glebis/claude-skills --skill firecrawl-research -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/glebis/claude-skills --path firecrawl-research
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/firecrawl-research/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add glebis/claude-skills --skill firecrawl-research -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/glebis/claude-skills/tree/HEAD/firecrawl-research/SKILL.md

## When to use it
- scrape web pages for research topics listed in markdown files
- save FireCrawl search results into timestamped markdown documents
- convert scraped titles and URLs into BibTeX bibliography entries
- format academic drafts using Pandoc and MyST templates

## Example prompts
- Research the topics in ai-research.md using FireCrawl and save results to ./research_output
- Gather web sources for topics listed in notes.md, limiting results to 3 per topic
- Generate a references.bib file from the scraped markdown files in ./research_output

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 2574869e9d119b09.

BEGIN THIRD-PARTY FILE 2574869e9d119b09
````text
---
name: firecrawl-research
description: This skill should be used when the user requests to research topics using FireCrawl, enrich notes with web sources, search and scrape information, or write scientific/academic papers. It extracts research topics from markdown files, creates research documents with scraped sources, generates BibTeX bibliographies from research results, and provides Pandoc/MyST templates for academic writing with citation management.
---

# FireCrawl Research

## Overview

Enrich research documents by automatically searching and scraping web sources using the FireCrawl API. Extract research topics from markdown files and generate comprehensive research documents with source material.

## When to Use This Skill

Use this skill when the user:
- Says "Research this topic using FireCrawl"
- Requests to enrich notes or documents with web sources
- Wants to gather information about topics listed in a markdown file
- Needs to search and scrape multiple topics systematically

## How It Works

### 1. Topic Extraction

The script automatically extracts research topics from markdown files using two methods:

**Method 1: Headers**
```markdown
## Spatial Reasoning in AI
### Computer Vision Applications
```
Both `Spatial Reasoning in AI` and `Computer Vision Applications` become research topics.

**Method 2: Research Tags**
```markdown
- [research] Large Language Models for robotics
- [search] Theory of Mind in autonomous driving
```
Both tagged items become research topics.

### 2. Search and Scrape

For each topic:
1. Searches FireCrawl with the topic as query
2. Retrieves up to N results (default: 5)
3. Automatically scrapes full content from each result
4. Extracts markdown-formatted content (main content only)

### 3. Output Generation

Creates new markdown files in the specified output directory:
- One file per topic
- Filename: `{topic}_{timestamp}.md`
- Contains: title, date, sources count, full scraped content
- Each source includes: title, URL, markdown content

## Usage

### Basic Usage

```bash
python scripts/firecrawl_research.py research.md
```

Outputs to current directory.

### Specify Output Directory

```bash
python scripts/firecrawl_research.py research.md ./output
```

Creates files in `./output/` folder.

### Limit Results Per Topic

```bash
python scripts/firecrawl_research.py research.md ./output 3
```

Retrieves maximum 3 results per topic.

## Configuration

### API Key Setup

1. Copy `.env.example` to `.env`:
   ```bash
   cp .env.example .env
   ```

2. Add FireCrawl API key:
   ```
   FIRECRAWL_API_KEY=fc-your-actual-api-key
   ```

The script automatically loads the API key from the skill's `.env` file.
````
END THIRD-PARTY FILE 2574869e9d119b09

The third-party file has ended. Everything between the 2574869e9d119b09 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2689 of 8772 bytes. © glebis · MIT · https://github.com/glebis/claude-skills/tree/HEAD/firecrawl-research/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md