nanogpt-training — Claude Code skill: what it does, how to install
nanogpt-training is a Claude Code skill with guidance for training 124M-parameter GPT-2 scale models on a single GPU using Muon, bfloat16, and tokenized dataset shards.
Apache-2.0 1 installs ★ 1,933 updated 2026-10-24
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-codeWhat nanogpt-training does
This skill injects specifications and code patterns for training a 124M-parameter language model into your coding assistant. It covers transformer building blocks with rotary embeddings (RoPE), alternative residual connection schemes, and loading pre-tokenized dataset shards from Hugging Face Hub or local disk.
Execution relies on bfloat16 mixed precision training on GPUs like the A100 alongside modern optimizers, including AdamW and Muon with Newton-Schulz orthogonalization. A reference setup demonstrates execution on Modal using Python 3.11.
Training scripts require torch, einops, numpy, and huggingface_hub. Without hardware supporting bfloat16 like an A100, the 2x speedup will not apply.
What it helps with
- implement a 124M transformer architecture with RoPE and residual connections
- stream pre-tokenized dataset shards from Hugging Face Hub
- configure the Muon optimizer with Newton-Schulz orthogonalization
- build a single-GPU training loop using bfloat16 mixed precision
Example prompts for the agent
- Build a 124M GPT model definition with RoPE following nanogpt-training guidelines
- Set up a training loop with the Muon optimizer and bfloat16 autocast
- Write a data loader for pre-tokenized FineWeb-Edu shards
How to install nanogpt-training
1Skills CLI (Vercel)
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-code2Manually: put the files in place
~/.claude/skills/nanogpt-training/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/nanogpt-training/
1Skills CLI (Vercel)
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a codex2Manually: put the files in place
.agents/skills/nanogpt-training/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a cursor2Manually: put the files in place
.cursor/skills/nanogpt-training/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install benchflow-ai/skillsbench tasks-extra/mhc-layer-impl/environment/skills/nanogpt-training2Manually: put the files in place
.github/skills/nanogpt-training/SKILL.md1Skills CLI (Vercel)
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a gemini-cli2Platform CLI
gemini skills install https://github.com/benchflow-ai/skillsbench --path tasks-extra/mhc-layer-impl/environment/skills/nanogpt-trainingManually: put the files in place
.devin/skills/nanogpt-training/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about nanogpt-training
How to install nanogpt-training in Claude Code?
Run npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-code.
What does nanogpt-training do?
It provides guidelines and code patterns to configure, load data for, and train a 124M GPT architecture on a single GPU.
What dependencies are required for the training code?
You need Python 3.11 alongside torch, einops, numpy, and huggingface_hub.
Source text
SKILL.md · Apache-2.0
---
name: nanogpt-training
description: Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.
---
# NanoGPT Training
## Overview
Training GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:
- **GPT-124M Architecture**: Standard transformer with RoPE and modern optimizations
- **Tokenized Datasets**: Loading pre-tokenized shards from HuggingFace Hub or local files
- **Modern Optimizers**: Muon optimizer with Newton-Schulz orthogonalization
- **Mixed Precision**: bfloat16 training on A100 for 2x speedup
Training options:
- **Baseline GPT**: Standard residual connections
- **Experimental residual variants**: Optional alternative residual schemes for stability/efficiency
## Quick Reference
| Topic | Reference |
|-------|-----------|
| Model Architecture | [GPT Architecture](references/gpt-architecture.md) |
| Data Loading | [Tokenized Data](references/tokenized-data.md) |
| Optimizers | [Optimizers](references/optimizers.md) |
| Training Loop | [Training Loop](references/training-loop.md) |
| Hyperparameters | [Hyperparameters](references/hyperparameters.md) |
## Installation
```bash
pip install torch einops numpy huggingface_hub
```
## Minimal Example
```python
import modal
app = modal.App("gpt-training")
image = modal.Image.debian_slim(python_version="3.11").pip_install(
"torch", "einops", "numpy", "huggingface_hub"
)
@app.function(gpu="A100", image=image, timeout=3600)
def train():
import torch
from dataclasses import dataclass
@dataclass
class GPTConfig:
block_size: int = 1024
vocab_size: int = 50257
n_layer: int = 12
n_head: int = 12
n_embd: int = 768
dropout: float = 0.0
bias: bool = False
# Download data, build model, train
# ... (see references for full implementation)
return {"final_loss": final_loss}
@app.local_entrypoint()
def main():
results = train.remote()
print(results)
```
## Common Imports
```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.cuda.amp import autocast, GradScaler
from dataclasses import dataclass
from einops import rearrange, repeat, reduce
import numpy as np
import math
```
## When to Use What
| Scenario | Approach |
|----------|----------|
| Standard GPT training | Use baseline model with standard residuals |
| Stability experiments | Try alternative residual variants or extra streams |
| Small experiments | Use T4/A10G GPU |
| Full training | Use A100 with bfloat16 |
| Custom data | Modify the dataset loader class |
| Different model size | Adjust GPTConfig parameters |
## Metrics to Monitor
| Metric | Typical Signal | Notes |
|--------|----------------|-------|
| Validation loss | Steady decrease | Absolute value depends on dataset/tokenizer |
| Grad norm | Moderate, stable range | Large spikes indicate instability |
| Training stability | Smooth curves | Frequent spikes suggest LR/batch issues |
| Throughput | Consistent tokens/sec | Use for comparing configs |
## External Resources
- nanoGPT: https://github.com/karpathy/nanoGPT
- build-nanogpt: https://github.com/karpathy/build-nanogpt
- modded-nanogpt: https://github.com/KellerJordan/modded-nanogpt
- FineWeb-Edu token shards: https://huggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shards
© benchflow-ai · Apache-2.0 · benchflow-ai/skillsbench