# verl-rl-training

Provides step-by-step guidance for reinforcement learning post-training of LLMs using verl with FSDP, Megatron-LM, and Ray.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 962 installs · ★ 13956
- Author: Orchestra Research
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/verl/SKILL.md
- Updated: 2026-10-11
- HTML: https://affpapa.org/en/nevibecode/skills/verl-rl-training

## What it does

verl implements the HybridFlow framework from ByteDance's Seed team, separating Ray control flow from distributed workers. This skill instructs the coding agent on configuring algorithms like GRPO, PPO, and DAPO for models ranging from 0.5B to 671B parameters.

It covers dataset formatting in Parquet, rule-based reward functions, and rollout engine configurations using vLLM or SGLang. For models larger than 70B parameters, it provides setup recipes for Megatron-LM tensor and pipeline parallelism via `mbridge`.

Running these workflows demands substantial compute infrastructure. verl relies on `ray>=2.41.0`, `torch>=2.0.0`, and `vllm>=0.8.2`, with standard recipes expecting clusters of 8 or more GPUs.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training -a claude-code
```
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/verl/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/verl/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/verl/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/verl/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Orchestra-Research/AI-Research-SKILLs verl
```

2. **Manually: put the files in place**
- File: `.github/skills/verl/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Orchestra-Research/AI-Research-SKILLs --path 06-post-training/verl
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/verl/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/verl/SKILL.md

## When to use it
- configure GRPO training for reasoning models on math datasets
- write rule-based reward functions extracting answers via regex
- set up PPO post-training with a dedicated critic model and GAE
- orchestrate multi-node training using Megatron-LM and Ray

## Example prompts
- Help me write a verl GRPO config for Qwen2.5-7B on GSM8K.
- How do I configure PPO with GAE and a critic model in verl?
- Set up a verl training script using Megatron backend and vLLM rollouts.

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: b85e443d615d0757.

BEGIN THIRD-PARTY FILE b85e443d615d0757
````text
---
name: verl-rl-training
description: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training]
dependencies: [verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0]
---

# verl: Volcano Engine Reinforcement Learning for LLMs

verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.

## When to Use verl

**Choose verl when you need:**
- Production-ready RL training at scale (tested up to 671B parameters)
- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)
- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)
- Multi-turn rollout with tool calling for agentic workflows
- Vision-language model RL training

**Consider alternatives when:**
- You need Megatron-native training → use **slime** or **miles**
- You want PyTorch-native abstractions with Monarch → use **torchforge**
- You only need simple SFT/DPO → use **TRL** or **Axolotl**

## Key Features

- **Training backends**: FSDP, FSDP2, Megatron-LM
- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers
- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO
- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)
- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools

## Installation

```bash
# Option 1: pip install
pip install verl[vllm]  # or verl[sglang] for SGLang backend

# Option 2: Docker (recommended for production)
docker pull verlai/verl:vllm011.latest

# Option 3: From source
git clone https://github.com/volcengine/verl.git
cd verl && pip install -e .[vllm,math]
```

## Quick Start: GRPO Training

```bash
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=~/data/gsm8k/train.parquet \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    trainer.n_gpus_per_node=8
```

## Core Architecture

verl uses a **HybridFlow** programming model separating control flow from computation:

```
┌─────────────────────────────────────────────────────────┐
│ Single-Process Controller (Ray)                         │
│ - Orchestrates: rollout → reward → train → sync        │
````
END THIRD-PARTY FILE b85e443d615d0757

The third-party file has ended. Everything between the b85e443d615d0757 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2855 of 10323 bytes. © Orchestra Research · MIT · https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/verl/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md