Skip to main content
skill CX Codex

optimize-training-kernels — Claude Code skill for kernel tuning

The optimize-training-kernels Claude Code skill adjusts fusion settings and attention backends while checking numerical tolerances. It is triggered by slow kernels, proposed kernel changes, or numerical drift.

Apache-2.0 ★ 99 updated 2026-11-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add TokenRhythm/InfraRSI --skill optimize-training-kernels -a claude-code
Open on GitHub

What optimize-training-kernels does

The skill guides fusion-flag changes and attention-backend selection; when L2 is open, it can also evaluate approved operator replacements. It checks real inputs against row-wise FP64 references and tracks numerical limits across training steps.

Fusion options have separate rules for Megatron-Core 0.18.2. For example, cross_entropy_loss_fusion accepts the native implementation, while te is rejected for stability. Test flags individually before combining them. Kernel replacements also have precision requirements: BF16 at interfaces, with FP32 used for softmax, LSE, and accumulation.

TF32, reduced-precision reductions, and FP8 are out of bounds for these kernel changes. A CUDA graph alone does not establish a speedup; verification covers numerical accuracy and step time.

What it helps with

  • enable fusion flags while checking numerical tolerances
  • select an attention backend for a given input shape
  • compare a kernel replacement with row-wise FP64 references
  • trace numerical drift after a kernel change

Example prompts for the agent

  • Check whether these fusion flags are safe for this training setup.
  • Compare the new attention kernel with an FP64 reference on real inputs.
  • Investigate numerical drift after replacing a Triton kernel.

How to install optimize-training-kernels

1Skills CLI (Vercel)

npx skills add TokenRhythm/InfraRSI --skill optimize-training-kernels -a claude-code

2Manually: put the files in place

~/.claude/skills/optimize-training-kernels/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/optimize-training-kernels/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about optimize-training-kernels

How do I install optimize-training-kernels in Claude Code?

Run npx skills add TokenRhythm/InfraRSI --skill optimize-training-kernels -a claude-code. The skill file is placed at ~/.claude/skills/optimize-training-kernels/SKILL.md.

What does optimize-training-kernels do?

It guides fusion and attention-backend tuning, and can assess approved kernel replacements when L2 is open. Checks cover FP64 reference comparisons, numerical tolerances, and step time.

How does the skill check kernel replacement accuracy?

It compares forward results element by element and backward results row by row against FP64 references on real inputs. Operations using atomics require two references, and selected rows must stay within tolerance.

Source text

SKILL.md · Apache-2.0
---
name: optimize-training-kernels
description: Use fused kernels and kernel-level changes without breaking numerics - L0 fusion flags, attention backends, and (where L2 is open) whitelisted kernel replacements gated by FP64 row-wise references on first real inputs.
license: Apache-2.0
when_to_use: Fusion flags are off in the start config; attention or elementwise kernels dominate; a custom kernel is proposed; numerics drift after a kernel change; 'fusion', 'rope fusion', 'cross entropy fusion', 'attention backend', 'triton kernel', 'FP64 reference'.
metadata:
  author: InfraRSI A0 (human-authored)
  source: human-authored (A0)
  version: 1
  tags: [kernels, fusion, attention, numerics, triton, dense, moe, long-context]
---

# Optimize training kernels

## Trigger conditions
- FR/ED leave fusions off (apply_rope_fusion, bias_activation_fusion, masked_softmax_fusion, gradient_accumulation_fusion, cross_entropy_loss_fusion all false in fr_v0).
- `profile.fwd_bwd_s` dominates with memory to spare and recompute already minimized.
- L2 is open on this task (only some strong-model dev tasks) and a whitelisted op (swiglu, rmsnorm, rope, cross-entropy) is hot.

## Required observations (receipt fields)
- `step_time`, `profile.fwd_bwd_s`, `numerics` (class, main_ce_rel_max, main_ce_rel_mean_window, grad_norm.first_update_rel, grad_norm.log_bias_window), `invariants.backends_flags`, `invariants.dtype_census`.

## Rules: L0 fusions (Megatron-Core 0.18.2)
- apply_rope_fusion: fused RoPE (TE). bias_activation_fusion: fused bias+activation for gelu/swiglu. masked_softmax_fusion: only affects the unfused attention path. gradient_accumulation_fusion: wgrad GEMM accumulates directly into the fp32 main_grad (TE linears). cross_entropy_loss_fusion with impl 'native' (impl 'te' is rejected for stability).
- These change kernels and reduction orders, not semantics: they are E1 candidates and must pass the numerics tolerance against FR plan A (main CE <= 0.5% per step, window mean CE <= 0.2%, grad norm <= 1% at update 1 and |mean ln(gn_c/gn_r)| <= 0.10 over the window). Per-step grad-norm differences after update 1 are report-only kernel noise (same-config runs differ by up to 56%): never tune against them.
- attention_backend: auto lets TE pick (fused/flash); forcing flash or fused can help or fail per shape; unfused is a debugging path and slow.
- Turn fusions on together only after each is known to work; one at a time when a run fails.

## Rules: exactness gates for kernels (L1 replay / L2)
- Reference: forward elementwise and backward row-wise against an FP64 reference on the first real inputs, with two references when atomics are involved; zero selected-row violations, low and zero rows checked separately.
- BF16 at interfaces, FP32 for softmax/LSE/accumulation/workspaces. Softmax backward: Delta = rowsum(P * dP) must be computed in FP32 from unrounded values; deriving it from the BF16-rounded output broke the gate.
- Bitwise where the contract allows: repeated runs gave O and dQ bitwise; dK/dV with FP32 atomics relL2 4.43e-5 < 1e-4.
- Launch counts and CUDA graphs are not speed by themselves: a selector CUDA graph was exact but missed the 0.16 s/update gate in all 48 full-cost scenarios.
- Precision flags (TF32, reduced-precision reductions) and FP8 are not kernel choices here: they are silent precision drops and invalid.

## Allowed actions
- L0: apply_rope_fusion, bias_activation_fusion, masked_softmax_fusion, gradient_accumulation_fusion, cross_entropy_loss_fusion, cross_entropy_fusion_impl (native), attention_backend.
- L2 (only where the task opens it): whitelisted operator replacement with the gates above.

## Failure signals
- numerics.passed false or class beyond E1; NaN/Inf; backends_flags or dtype_census changed.
- Faster kernel, slower update (extra casts, re-quantization, lost overlap).

## Verification
- First-use exactness gate passes; numerics within tolerance across all updates; StepTime beyond margin.

## Success case (real)
- Route attention v2 on NeMo CP4: Delta = sum(P*dP) in FP32 over the normalized route set, dS = P*(dP - Delta), BF16 interfaces, FP32 softmax/LSE/accumulation/dK-dV workspaces. It passed the frozen FP64 row gate, real-input calibration and a held-back layer: 9.808943 -> 6.699462 s (+46.41%). (MANUSCRIPT.md 4.1, 6.2)
- Smaller: lowering num_warps 8 -> 4 in three Triton kernels made them 37-40% faster, -0.81 s/step, bitwise. (flash_next_mfu_20261003/PROGRESS.md)

## Failure case (real)
- Route v1 passed forward checks and an aggregate L2 test but had 8,676 dQ-row and 43 dK-row stable violations (per CP shard dQ 3229/2011/2224/1212, dK 17/12/8/6) of the bound eB <= max(0.02*normR, 1.25*eA) under two FP64 references. Cause: Delta derived from the BF16-rounded output. Aggregate norms hid row failures. (mfu_core_20260929/ROUTE_ATTENTION_V2_CALIBRATION_RESULT.md; RSI_TRACK.md T2)
- FP8 fusion: see profile-whole-update and the negative-results catalogue (-4.62% / -3.89% vs BF16).

© InfraRSI A0 (human-authored) · Apache-2.0 · TokenRhythm/InfraRSI

Similar skills

All skills for AI and ML →
tilegym-monkey-patch-kernels-to-transformers TileGym Team The tilegym-monkey-patch-kernels-to-transformers skill wires TileGym kernels into Hugging Face Transformers models by swapping components before model creation. Use it to integrate existing kernels or build new cuTile ones. CX Codex +1 1,600installs tilekernels-gpu-kernels reason-machines This skill guides agents through TileKernels GPU kernels for MoE, quantization and other LLM operations. Use it when writing or checking TileLang code. CX Codex +1 370installs cudaq-guide CUDA-Q Team cudaq-guide walks you through CUDA-Q installation, simulation targets, QPU access and Python kernel authoring. Useful when you need to get moving with NVIDIA's quantum stack. CX Codex +1 2,500installs flydsl-kernel-trace-analysis amd The flydsl-kernel-trace-analysis Claude Code skill profiles GPU kernels and analyzes their instruction-level traces to locate stalls. Run it to collect a trace or inspect an existing dispatch directory. CX Codex +1 436installs V3 Performance Optimization ruvnet V3 Performance Optimization — a skill for optimizing claude-flow v3 with Flash Attention, HNSW indexing, and comprehensive benchmarking. Achieves 150–12,500x search improvement and 50–75% memory reduction. CX Codex +1 1,100installs model-serving-kubernetes devops-skills model-serving-kubernetes deploys ML models and LLMs on Kubernetes using KServe and NVIDIA Triton Inference Server. Handles canary rollouts, traffic splitting, and GPU scheduling. CX Codex +1 363installs

From the same set19

The whole set →
optimize-data-path The optimize-data-path Claude Code skill optimizes data and state transfers between the host and GPU without changing the data contract. It is triggered by costly loading, host synchronization, or optimizer-state transfers. CX Codex +1 ★ 99repo stars learn-optimization-skills The learn-optimization-skills Claude Code skill guides an optimizer’s learning between development tasks and records reusable lessons. It runs for post-task reflection or when a search turns up a reusable finding. CX Codex +1 ★ 99repo stars profile-whole-update The profile-whole-update Claude Code skill measures full-update time and checks whether a change improved the end-to-end update. It comes into play when reviewing profiles, comparing candidates, or tracing delays between steps. CX Codex +1 ★ 99repo stars validate-training-lifecycle The validate-training-lifecycle Claude Code skill checks a model-training run from checkpoint save through fresh-process resume, HF export, and forward comparison. CX Codex +1 ★ 99repo stars recompute-policy The recompute-policy Claude Code skill helps choose full, selective, or no recomputation based on memory headroom and model structure. It activates for activation-checkpointing changes, OOMs, or recompute_modules decisions. CX Codex +1 ★ 99repo stars moe-ep-communication The moe-ep-communication Claude Code skill configures expert parallelism and token dispatch for MoE in Megatron-Core 0.18.2. It activates for EP sizing, dispatcher choices, and profiling bottlenecks. CX Codex +1 ★ 99repo stars manage-memory-lifetimes The manage-memory-lifetimes Claude Code skill analyzes GPU memory failures and helps select residency, offload, and allocator settings. It activates for OOMs, allocator retries, or tight memory headroom. CX Codex +1 ★ 99repo stars discover-training-system The discover-training-system Claude Code skill inspects a training setup before performance changes. It runs at task start and when a setting appears to have no effect. CX Codex +1 ★ 99repo stars

Project sponsors

Companies that keep this analytics open