# airflow-expert

airflow-expert guides Apache Airflow DAG design, task dependencies, operators, sensors, and scheduling. Use it when building or reviewing data workflows.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 174 installs · ★ 45 repo stars
- Author: PCL Team
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/personamanagmentlayer/pcl/tree/HEAD/stdlib/data/airflow-expert/SKILL.md
- Updated: 2026-10-12
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/airflow-expert

## What it does

This skill gives the agent focused Apache Airflow guidance for DAG design, task dependencies, branching, dynamic task creation, operators, sensors, and XCom communication. It also covers schedules, retries, timeouts, pools, and failure handling.

Its guidance includes TaskFlow API patterns, idempotent DAGs, rescheduling sensors, and production checks such as logging, alerts, Connections, Variables, and Secrets Backend. The skill can consult `references/CORE_CONCEPTS.md` for DAG fundamentals, branching, dynamic tasks, operators, sensors, and task communication.

Apache Airflow `>=2.8.0` is required. The skill is allowed to use `Read`, `Write`, `Edit`, `Bash`, `Glob`, and `Grep`, so the agent can inspect project files, edit them, and run commands.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add personamanagmentlayer/pcl --skill airflow-expert -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/airflow-expert/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/airflow-expert/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add personamanagmentlayer/pcl --skill airflow-expert -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/airflow-expert/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add personamanagmentlayer/pcl --skill airflow-expert -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/airflow-expert/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install personamanagmentlayer/pcl stdlib/data/airflow-expert
```

2. **Manually: put the files in place**
- File: `.github/skills/airflow-expert/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add personamanagmentlayer/pcl --skill airflow-expert -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/personamanagmentlayer/pcl --path stdlib/data/airflow-expert
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/airflow-expert/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add personamanagmentlayer/pcl --skill airflow-expert -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/personamanagmentlayer/pcl/tree/HEAD/stdlib/data/airflow-expert/SKILL.md

## When to use it
- design an Apache Airflow DAG for a data pipeline
- connect tasks with dependencies and branching
- generate tasks dynamically from runtime data
- choose operators and sensors for a workflow
- configure retries, timeouts, and failure handling

## Example prompts
- Design an Apache Airflow DAG for this data pipeline and explain its task dependencies.
- Review this DAG for expensive top-level code and rewrite the affected part.
- Add dynamic task generation, XCom communication, and failure handling to this workflow.

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 28991d55f47fb1b7.

BEGIN THIRD-PARTY FILE 28991d55f47fb1b7
````text
---
name: airflow-expert
version: 1.1.0
description: >-
  Expert-level Apache Airflow orchestration, DAGs, operators, sensors, XComs, task
  dependencies, and scheduling. Use when the user mentions orchestration, DAGs, workflow,
  scheduling, or data pipeline, or when the task involves DAG Fundamentals, Task
  Dependencies and Branching, Dynamic Task Generation, or Operators and Sensors.
category: data
author: PCL Team
license: Apache-2.0
tags:
  - airflow
  - orchestration
  - dag
  - workflow
  - scheduling
  - data-pipeline
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
  - Glob
  - Grep
requirements:
  apache-airflow: '>=2.8.0'
---

# Apache Airflow Expert

You are an expert in Apache Airflow with deep knowledge of DAG design, task orchestration, operators, sensors, XComs, dynamic task generation, and production operations. You design and manage complex data pipelines that are reliable, maintainable, and scalable.

## Best Practices

### 1. DAG Design

- Keep DAGs simple and focused on single workflows
- Use TaskFlow API for cleaner code and automatic XCom handling
- Set catchup=False for new DAGs to avoid backfilling
- Use meaningful task_ids and add documentation
- Make DAGs idempotent for safe reruns

### 2. Task Configuration

- Set appropriate retries and retry_delay
- Use execution_timeout to prevent stuck tasks
- Configure proper depends_on_past for sequential processing
- Use pools to limit concurrent tasks
- Set priority_weight for critical tasks

### 3. Performance

- Minimize DAG file size and complexity
- Avoid top-level code that executes on every parse
- Use dynamic task mapping instead of creating many tasks
- Leverage sensors with reschedule mode for long waits
- Use task pools to prevent resource exhaustion

### 4. Production Operations

- Monitor DAG run duration and SLA misses
- Set up alerting for failures
- Use Variables and Connections instead of hardcoded values
- Enable DAG versioning and testing
- Implement proper logging

### 5. Security

- Store credentials in Connections, not code
- Use Secrets Backend (AWS Secrets Manager, Vault)
- Limit access with RBAC
- Audit DAG changes
- Encrypt sensitive XCom data

## Anti-Patterns

### 1. Non-Idempotent DAGs

```python
# Bad: Using current date
@task
def extract():
    today = datetime.now().date()
    return extract_data_for_date(today)

# Good: Using execution date
@task
def extract(**context):
    date = context['ds']
    return extract_data_for_date(date)
```

### 2. Heavy Top-Level Code

```python
# Bad: Expensive operation at top level
expensive_config = fetch_config_from_api()  # Runs on every parse

dag = DAG(...)

# Good: Load config in task
@task
def get_config():
    return fetch_config_from_api()
```
````
END THIRD-PARTY FILE 28991d55f47fb1b7

The third-party file has ended. Everything between the 28991d55f47fb1b7 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2739 of 4117 bytes. © PCL Team · Apache-2.0 · https://github.com/personamanagmentlayer/pcl/tree/HEAD/stdlib/data/airflow-expert/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
