Skip to main content
skill CX Codex

nanogpt-training — Claude Code skill: what it does, how to install

nanogpt-training is a Claude Code skill with guidance for training 124M-parameter GPT-2 scale models on a single GPU using Muon, bfloat16, and tokenized dataset shards.

Apache-2.0 1 installs ★ 1,933 updated 2026-10-24

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-code
Open on GitHub

What nanogpt-training does

This skill injects specifications and code patterns for training a 124M-parameter language model into your coding assistant. It covers transformer building blocks with rotary embeddings (RoPE), alternative residual connection schemes, and loading pre-tokenized dataset shards from Hugging Face Hub or local disk.

Execution relies on bfloat16 mixed precision training on GPUs like the A100 alongside modern optimizers, including AdamW and Muon with Newton-Schulz orthogonalization. A reference setup demonstrates execution on Modal using Python 3.11.

Training scripts require torch, einops, numpy, and huggingface_hub. Without hardware supporting bfloat16 like an A100, the 2x speedup will not apply.

What it helps with

  • implement a 124M transformer architecture with RoPE and residual connections
  • stream pre-tokenized dataset shards from Hugging Face Hub
  • configure the Muon optimizer with Newton-Schulz orthogonalization
  • build a single-GPU training loop using bfloat16 mixed precision

Example prompts for the agent

  • Build a 124M GPT model definition with RoPE following nanogpt-training guidelines
  • Set up a training loop with the Muon optimizer and bfloat16 autocast
  • Write a data loader for pre-tokenized FineWeb-Edu shards

How to install nanogpt-training

1Skills CLI (Vercel)

npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-code

2Manually: put the files in place

~/.claude/skills/nanogpt-training/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/nanogpt-training/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about nanogpt-training

How to install nanogpt-training in Claude Code?

Run npx skills add benchflow-ai/skillsbench --skill nanogpt-training -a claude-code.

What does nanogpt-training do?

It provides guidelines and code patterns to configure, load data for, and train a 124M GPT architecture on a single GPU.

What dependencies are required for the training code?

You need Python 3.11 alongside torch, einops, numpy, and huggingface_hub.

Source text

SKILL.md · Apache-2.0
---
name: nanogpt-training
description: Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.
---

# NanoGPT Training

## Overview

Training GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:

- **GPT-124M Architecture**: Standard transformer with RoPE and modern optimizations
- **Tokenized Datasets**: Loading pre-tokenized shards from HuggingFace Hub or local files
- **Modern Optimizers**: Muon optimizer with Newton-Schulz orthogonalization
- **Mixed Precision**: bfloat16 training on A100 for 2x speedup

Training options:
- **Baseline GPT**: Standard residual connections
- **Experimental residual variants**: Optional alternative residual schemes for stability/efficiency

## Quick Reference

| Topic | Reference |
|-------|-----------|
| Model Architecture | [GPT Architecture](references/gpt-architecture.md) |
| Data Loading | [Tokenized Data](references/tokenized-data.md) |
| Optimizers | [Optimizers](references/optimizers.md) |
| Training Loop | [Training Loop](references/training-loop.md) |
| Hyperparameters | [Hyperparameters](references/hyperparameters.md) |

## Installation

```bash
pip install torch einops numpy huggingface_hub
```

## Minimal Example

```python
import modal

app = modal.App("gpt-training")

image = modal.Image.debian_slim(python_version="3.11").pip_install(
    "torch", "einops", "numpy", "huggingface_hub"
)

@app.function(gpu="A100", image=image, timeout=3600)
def train():
    import torch
    from dataclasses import dataclass

    @dataclass
    class GPTConfig:
        block_size: int = 1024
        vocab_size: int = 50257
        n_layer: int = 12
        n_head: int = 12
        n_embd: int = 768
        dropout: float = 0.0
        bias: bool = False

    # Download data, build model, train
    # ... (see references for full implementation)

    return {"final_loss": final_loss}

@app.local_entrypoint()
def main():
    results = train.remote()
    print(results)
```

## Common Imports

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.cuda.amp import autocast, GradScaler
from dataclasses import dataclass
from einops import rearrange, repeat, reduce
import numpy as np
import math
```

## When to Use What

| Scenario | Approach |
|----------|----------|
| Standard GPT training | Use baseline model with standard residuals |
| Stability experiments | Try alternative residual variants or extra streams |
| Small experiments | Use T4/A10G GPU |
| Full training | Use A100 with bfloat16 |
| Custom data | Modify the dataset loader class |
| Different model size | Adjust GPTConfig parameters |

## Metrics to Monitor

| Metric | Typical Signal | Notes |
|--------|----------------|-------|
| Validation loss | Steady decrease | Absolute value depends on dataset/tokenizer |
| Grad norm | Moderate, stable range | Large spikes indicate instability |
| Training stability | Smooth curves | Frequent spikes suggest LR/batch issues |
| Throughput | Consistent tokens/sec | Use for comparing configs |

## External Resources

- nanoGPT: https://github.com/karpathy/nanoGPT
- build-nanogpt: https://github.com/karpathy/build-nanogpt
- modded-nanogpt: https://github.com/KellerJordan/modded-nanogpt
- FineWeb-Edu token shards: https://huggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shards

© benchflow-ai · Apache-2.0 · benchflow-ai/skillsbench

Similar skills

All skills for AI and ML →

From the same set22

The whole set →
hex-grid-spatial hex-grid-spatial is a Claude Code skill that calculates neighbours, hex distance, and radius cells on odd-r offset maps for movement or area-of-effect logic. CX Codex +1 1installs python-scala-syntax-mapping python-scala-syntax-mapping is a Claude Code skill that maps Python syntax to Scala for coding assistants translating variables, comprehensions, and control structures. CX Codex +1 1installs image-ocr Use this skill to read text in images with Tesseract OCR through Python. It fits scans, screenshots, and batches of image files. CX Codex +1 1installs audiobook Turns a web article or a text file into an audiobook: pulls the content, cleans it up, and voices it through ElevenLabs, OpenAI TTS or gTTS. CX Codex +1 1installs hierarchical-taxonomy-clustering The hierarchical-taxonomy-clustering skill combines similar category paths from multiple sources into a shared hierarchy. Use it to standardize product categories for cross-source analysis. CX Codex +1 1installs senior-data-engineer senior-data-engineer — a skill for Claude and other agents: it helps design ETL/ELT and streaming pipelines, generates Airflow DAGs, Kafka configs and data quality checks. CX Codex +1 1installs vulnerability-csv-reporting vulnerability-csv-reporting transforms vulnerability scanner JSON outputs into structured CSV files for compliance audits. CX Codex +1 1installs python-scala-functional The python-scala-functional skill guides Python-to-Scala rewrites in a functional style. Use it for higher-order functions, generators, pattern matching, and error handling. CX Codex +1 1installs

Project sponsors

Companies that keep this analytics open