# fine-tuning-with-trl

A skill for post-training language models with TRL: SFT, DPO, PPO and GRPO recipes you can run as-is.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 969 installs · ★ 13956
- Author: Orchestra Research
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/trl-fine-tuning/SKILL.md
- Updated: 2026-10-11
- HTML: https://affpapa.org/en/nevibecode/skills/fine-tuning-with-trl

## What it does

The skill covers four post-training methods built on TRL. SFT trains a base model on instruction-answer pairs, DPO aligns it on chosen/rejected pairs without a separate reward model, and PPO or GRPO optimise the policy against a reward signal. You get snippets for `SFTTrainer`, `DPOTrainer`, `RewardTrainer` and `GRPOTrainer`, plus CLI calls like `trl dpo` and `python -m trl.scripts.ppo`.

Three workflows are laid out step by step. The full RLHF pipeline runs SFT, then reward model training, then PPO. The DPO branch skips the reward model entirely. GRPO targets online training with low memory use, and it expects you to write the reward function yourself: it takes a list of completions and returns a list of scores.

Install the dependencies first: `pip install trl transformers datasets peft accelerate`. You also need `torch`, and without a GPU training a real model will crawl — worth checking your hardware before you start. Data comes as prompt-completion pairs for SFT and chosen/rejected pairs for DPO.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill fine-tuning-with-trl -a claude-code
```
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill trl-fine-tuning -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/trl-fine-tuning/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/trl-fine-tuning/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill trl-fine-tuning -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/trl-fine-tuning/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill trl-fine-tuning -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/trl-fine-tuning/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Orchestra-Research/AI-Research-SKILLs trl-fine-tuning
```

2. **Manually: put the files in place**
- File: `.github/skills/trl-fine-tuning/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill trl-fine-tuning -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Orchestra-Research/AI-Research-SKILLs --path 06-post-training/trl-fine-tuning
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/trl-fine-tuning/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Orchestra-Research/AI-Research-SKILLs --skill trl-fine-tuning -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/trl-fine-tuning/SKILL.md

## When to use it
- fine-tune a base model on instruction data with SFT
- align a model to preferences without training a reward model
- run the full RLHF chain of SFT, reward model and PPO
- write a custom reward function for online RL
- evaluate the aligned model after training

## Example prompts
- Set up SFTTrainer to fine-tune Qwen2.5-0.5B on an instruction dataset
- Build a DPO config with beta 0.1 and run DPOTrainer on chosen/rejected pairs
- Write a reward function for GRPOTrainer and start training

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 7bc5e516c228e46a.

BEGIN THIRD-PARTY FILE 7bc5e516c228e46a
````text
---
name: fine-tuning-with-trl
description: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]
dependencies: [trl, transformers, datasets, peft, accelerate, torch]
---

# TRL - Transformer Reinforcement Learning

## Quick start

TRL provides post-training methods for aligning language models with human preferences.

**Installation**:
```bash
pip install trl transformers datasets peft accelerate
```

**Supervised Fine-Tuning** (instruction tuning):
```python
from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()
```

**DPO** (align with preferences):
```python
from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()
```

## Common workflows

### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

```
RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model
```

**Step 1: Supervised fine-tuning**

Train base model on instruction-following data:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
````
END THIRD-PARTY FILE 7bc5e516c228e46a

The third-party file has ended. Everything between the 7bc5e516c228e46a tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2442 of 11451 bytes. © Orchestra Research · MIT · https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/06-post-training/trl-fine-tuning/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md