Skip to main content
skill CX Codex

nemotron-asr-finetune — Claude Code skill: what it does

nemotron-asr-finetune coordinates domain and language adaptation for NVIDIA Nemotron Speech and Riva ASR, picking the lowest-cost path from word boosting to full fine-tuning.

Apache-2.0 210 installs ★ 4,106 updated 2026-10-23

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add NVIDIA/skills --skill nemotron-asr-finetune -a claude-code
Open on GitHub

What nemotron-asr-finetune does

The skill manages speech recognition adaptation built on NVIDIA Riva and NeMo. Instead of launching training immediately, it scopes the requirements and tests the lowest-cost options first, such as word boosting or n-gram language model fusion. Fine-tuning runs only when lighter methods cannot satisfy the target accuracy.

Tasks route directly to specialized sub-skills. It assigns synthetic data creation to data-designer, model recipes and execution to nemo-speech-asr-finetune, and final deployment to nemotron-speech. The workflow tracks Word Error Rate (WER) and monitors potential degradation on general speech datasets.

Pre-flight checks are mandatory before training begins: audio files must be validated or converted to 16 kHz, and skipping the latest NeMo main branch or published container risks dependency conflicts.

What it helps with

  • plan the lowest-cost ASR adaptation route for custom vocabulary
  • audit speech datasets and enforce 16 kHz audio prior to training
  • fine-tune NeMo acoustic models for a new target language
  • measure domain WER and prevent performance regression on general speech

Example prompts for the agent

  • What is the cheapest way to adapt our ASR model to medical terms?
  • Design an ASR fine-tuning plan for noisy call center recordings and estimate the required data.
  • Run pre-flight checks on our speech dataset before NeMo training.

How to install nemotron-asr-finetune

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill nemotron-asr-finetune -a claude-code

2Manually: put the files in place

~/.claude/skills/nemotron-asr-finetune/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/nemotron-asr-finetune/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about nemotron-asr-finetune

How to install nemotron-asr-finetune in Claude Code?

Run npx skills add NVIDIA/skills --skill nemotron-asr-finetune -a claude-code. The file will be saved to ~/.claude/skills/nemotron-asr-finetune/SKILL.md.

What does nemotron-asr-finetune do?

It scopes speech adaptation tasks, selects the most cost-efficient path among word boosting, n-gram LM, or fine-tuning, and delegates tasks to specialized NeMo and Riva sub-skills.

How do Nemotron Speech and Riva differ in this skill?

Nemotron Speech is the brand name, but CLI commands, paths, imports, and container tags continue to use the Riva and NeMo naming conventions.

Source text

SKILL.md · Apache-2.0
---
name: "nemotron-asr-finetune"
description: Orchestration skill for NVIDIA Nemotron Speech (Riva) / NeMo ASR domain and language adaptation. Given a goal like "improve/fine-tune ASR for my domain or language", it scopes the task, picks the cheapest sufficient path (word boosting → n-gram LM → fine-tuning), delegates each stage to the right sub-skill (data generation, training, evaluation, deployment), and answers cost/time/data questions along the way.
triggers:
  - fine-tune ASR for my domain
  - adapt ASR to my language
  - improve ASR accuracy
  - customize ASR
  - ASR domain adaptation
  - new language ASR
  - reduce WER
  - my ASR gets jargon wrong
  - noisy call-center ASR
  - how much data to fine-tune ASR
  - cheapest way to improve ASR
  - orchestrate ASR fine-tuning
  - ASR customization pipeline
version: "1.3.0"
license: Apache-2.0
metadata:
  author: "Nemotron Speech Team"
  team: riva
  tags:
    - nvidia
    - nemotron-speech
    - riva
    - nemo
    - asr
    - speech-to-text
    - orchestration
    - customization
    - domain-adaptation
    - fine-tuning
    - word-boosting
    - language-model
    - synthetic-data
    - evaluation
    - deployment
  domain: ml
---

# Nemotron Speech ASR Customization — Orchestration Skill

> **Note:** "Nemotron Speech" is the public-facing name for what NVIDIA documents today as **Riva** / **Riva NIM**; the acoustic models are trained and fine-tuned with **NVIDIA NeMo**. Commands, config paths, imports, and doc URLs still use **"Riva"** / **"NeMo"** — the rename is brand-only. Do not rename them.

## What This Skill Is

This is a **high-level orchestration skill**, not a step-by-step training manual. Its job, given a goal such as *"I want to fine-tune ASR for my domain/language"*, is to:

1. **Scope** the problem (how much real audio, target eval set, latency/hardware budget, language/domain).
2. **Choose the cheapest sufficient path** — word boosting, n-gram LM fusion, or fine-tuning — and escalate only when quality falls short.
3. **Delegate each stage to the right sub-skill** (data generation, training, evaluation, deployment/optimization).
4. **Answer cost/time/data questions along the way** (how many hours to hit X% WER, synthetic vs real, L40S vs H100, expected cost).

It owns the plan and the routing; the sub-skills own the execution. When a needed sub-skill does not exist yet, this skill names it as a **placeholder** and gives interim guidance.

## When to Use

Use for any request to make a Nemotron Speech / Riva ASR model work better on a specific domain or language — improving accuracy, reducing WER, adding a language, or planning a fine-tune. Start here even when the user names a specific technique: treat it as a candidate until the orchestration step confirms it is the cheapest sufficient path, then sequence the right sub-skills.

## Orchestration Workflow

Run the loop below; each stage names the sub-skill it invokes. Full detail in [`references/workflow.md`](references/workflow.md).

| # | Stage | What happens | Sub-skill |
|---|---|---|---|
| 1 | **State the goal** | Capture the target: domain/language, the errors, the metric. | Orchestration (this skill) |
| 2 | **Clarify & scope** | Ask the discovery questions: how much real audio? target eval set? latency/HW budget? deployment target? | Orchestration |
| 3 | **Choose the path** | Pick the cheapest sufficient rung (boosting → n-gram LM → fine-tune). Escalate only if quality is short; experiment while proposing the full plan. | Orchestration → Research/Training |
| 4 | **Get the data right** | If data is scarce/noisy: synthetic (TTS), TTS-friendly formatting, noise profiling/harvest, blend, score vendor samples; align customer data to training format; flag missing real data. | SDG / Data |
| 5 | **Train** | Apply the recipe (configs, hyperparameters, replay/curriculum, GPU/OOM preflight) and run. | Research / Training |
| 6 | **Evaluate** | Normalized WER on the domain set + A/B forgetting check on a general set; error-driven analysis to find the next lever. | Evaluation |
| 7 | **Loop or ship** | If short of target, loop to 4/5 with targeted data; else select/average checkpoints. Consult the user before more cycles. | Orchestration |
| 8 | **Deploy** | Export to NIM/HF, hot-swap the checkpoint, serve. | Deployment / Optimization |

**Stages 4–8 are the fine-tune path** (`data → NeMo train → NeMo eval → Riva deploy`). Cheaper rungs (boosting, custom vocab, n-gram LM) take a **shorter branch owned by a single sub-skill** — don't force them through the full loop. See the branch-by-rung table in [`references/workflow.md`](references/workflow.md) (§3b).

Before training (Stage 5), run the **pre-flight dataset quality check** (§4a): verify/convert audio to 16 kHz, and, on user request, audit transcript quality by running a reference pretrained model and checking WER against the provided ground truth. See [`references/workflow.md`](references/workflow.md) §4a.

Also before training (Stage 5), run the **pre-flight environment/dependency check** (§4b): if no NeMo is provided, pull the latest `main` (local execution) or the latest published container tag (container execution). If a NeMo checkout/install is already provided, check its version, but still recommend switching to latest `main` (staleness risk) and ask the user — if they insist on the provided one, proceed with it and only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. See [`references/workflow.md`](references/workflow.md) §4b.

Throughout, answer the **"along the way"** questions (data volume, synthetic vs real, hours to reach a WER target, cost, GPU choice) — see [`references/planning-answers.md`](references/planning-answers.md).

## Sub-Skills This Skill Calls

Detailed registry, invocation, and handoff contracts in [`references/sub-skills.md`](references/sub-skills.md).

| Role (per the architecture) | Purpose | Sub-skill to invoke |
|---|---|---|
| **Research / Training** | NeMo configs, recipes, fine-tuning, checkpoint averaging; also owns the NeMo-side word-boosting and n-gram LM pilots | `nemo-speech-asr-finetune` |
| **SDG / Data Designer** | Synthetic transcripts/text, noise profiling, vendor-data impact, blends | `data-designer` (synthetic **text**; audio via TTS in `nemotron-speech`); *placeholder:* `asr-data-profiling` |
| **Evaluation** | Normalized WER, A/B forgetting, error analysis | Offline file WER → `nemo-speech-asr-finetune`; **served-endpoint WER → `nemotron-speech`** |
| **Deployment / Optimization** | NIM/Riva export, checkpoint swap, NIM-build optimization, serving | `nemotron-speech` |

If a sub-skill is unavailable, say so, give the interim guidance from the reference, and continue the plan.

## Choosing The Path (cheapest first)

The scoping in Stage 3 selects the lowest-cost rung that can meet the target. Summary; full docs-grounded ladder in [`references/path-selection.md`](references/path-selection.md).

- **Word boosting** — a bounded set of known words/names/jargon. **Two realizations that are different artifacts:**
  *pilot (NeMo)* to prove lift offline via GPU-PB context biasing (`nemo-speech-asr-finetune`), or *deploy (Riva)* to
  ship it via runtime `boosted_lm_score` (`nemotron-speech`) — different score systems, don't reuse one for the
  other. Runtime, no training either way. See [`references/path-selection.md`](references/path-selection.md).
- **Custom vocabulary / pronunciation** — OOV or consistently mispronounced terms. Deploy-time. → Deployment sub-skill.
- **N-gram (KenLM) LM** — domain phrasing/word-sequences when you have text but little audio. **Two realizations, built the same way but deployed differently by architecture:** *pilot (NeMo, NGPU-LM)* to prove lift offline (`nemo-speech-asr-finetune`), or *deploy (Riva)* to ship it (`nemotron-speech`). For **CTC**, rebuild the pilot corpus into a Riva word-level LM — don't ship the pilot artifact as-is. For **RNN-T/TDT**, the opposite: hand the pilot's `.nemo` artifact to Riva unchanged, no rebuild. See [`references/path-selection.md`](references/path-selection.md).
- **Fine-tune** — real acoustic gaps (accents, noise, channel) with enough transcribed audio (NIM guide: 100+ h; ~10 h floor only if mixed to avoid catastrophic forgetting). **Below ~10 h, do not recommend fine-tuning — recommend word boosting instead** (severe catastrophic-forgetting/overfitting risk). → Research/Training.
- **Train from scratch / cross-language transfer** — a new language with no suitable checkpoint (last resort). → Research/Training.

Ordering and per-model support follow the NVIDIA Speech NIM ASR customization guide:
<https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html>.

## Key Principles

- **Scope before you pick.** Don't recommend fine-tuning before the discovery questions and a measured baseline.
- **Cheapest sufficient path.** Escalate rungs only when the current one provably can't hit the target; you may experiment on a cheap rung while presenting the full fine-tuning plan.
- **Measure with a contract.** Report normalized WER on the domain set plus an A/B forgetting check on a general set — never in-training logs alone.
- **Verify before you train.** Run the pre-flight dataset quality check before Stage 5: verify/convert audio to 16 kHz, and, when the user asks, audit ground-truth transcripts by comparing them to a reference model's WER. Don't train on an unresampled or unaudited-on-request dataset. See [`references/workflow.md`](references/workflow.md) §4a.
- **Know your environment before you train.** If no NeMo is provided, pull the latest `main` (or latest published container tag). If one is provided, check its version but still recommend latest `main` and ask the user first — a provided checkout always carries staleness risk. If they insist on keeping it, don't pull preemptively; only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. Record what was resolved in the ledger. See [`references/workflow.md`](references/workflow.md) §4b.
- **Delegate, don't reimplement.** Route execution to the sub-skills; keep this skill focused on the plan, sequencing, and cost/time/data answers.
- **Real target-domain audio is the usual bottleneck.** Prefer real data; use synthetic to fill measured gaps, kept separately weighted so it can be ablated.
- **Consult the user before extra tuning cycles**, and when a needed sub-skill is a placeholder.

## Source of Truth

| Topic | Location |
|---|---|
| NIM Speech docs home | https://docs.nvidia.com/nim/speech/latest/index.html |
| ASR customization guide (methods, per-model support) | https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html |
| ASR support matrix (models & features) | https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html |
| NeMo fine-tuning (flags/config) | `docs/source/asr/fine_tuning.rst`, and the `nemo-speech-asr-finetune` sub-skill |
| Riva ASR tutorials (boosting, LM, fine-tune) | https://github.com/nvidia-riva/tutorials |
| Tokenizer extension to new language + acoustic fine-tune | https://github.com/nvidia-riva/tutorials/blob/main/asr-extend-tokenizer-to-newlang-ft-acoustic-model.ipynb |

## Limitations

- Orchestration only — execution happens in the sub-skills. Where a sub-skill is a placeholder, guidance is interim until it exists.
- GPU required for the training rungs; deployment/serving is owned by the `nemotron-speech` sub-skill.
- Model names, config paths, flags, and per-model feature support drift across NeMo/Riva releases — verify against the support matrix and the current checkout.
- Public branding is **"Nemotron Speech"**; commands, imports, config paths, and doc URLs still use **"Riva"** / **"NeMo"** — do not rename.

© Nemotron Speech Team · Apache-2.0 · NVIDIA/skills

Similar skills

All skills for AI and ML →

From the same set312

The whole set →
earth2studio-create-prognostic earth2studio-create-prognostic guides an agent through building an Earth2Studio wrapper for a weather model that forecasts future states over successive time steps. CX Codex +1 579installs nemo-relay-install The nemo-relay-install skill identifies your installation target and deploys the current stable NeMo Relay package, resolving prerequisites and verifying availability. Works across agent runtimes, frameworks, and language bindings. CX Codex +1 227installs nemo-relay-instrument-calls nemo-relay-instrument-calls wraps tool and LLM provider call sites with NeMo Relay execution scopes. Keeps business logic intact while attaching lifecycle events, intercepts, and guardrails. CX Codex +1 224installs nemo-relay-instrument-context-isolation The nemo-relay-instrument-context-isolation skill keeps NeMo Relay scope stacks separate across concurrent work and preserves event ancestry as tasks cross execution boundaries. CX Codex +1 211installs nemo-relay-debug-runtime-integration The nemo-relay-debug-runtime-integration skill traces NeMo Relay runtime faults across loading, scopes, events, plugins, and adaptive wiring. Use it when the package is present but app behavior is wrong. CX Codex +1 197installs nemo-relay-instrument-typed-wrappers nemo-relay-instrument-typed-wrappers configures NeMo Relay typed wrappers and provider codecs while keeping JSON middleware semantics intact. CX Codex +1 182installs nemo-relay-plugin-build nemo-relay-plugin-build is a skill for packaging reusable NeMo Relay behavior as config components or dynamic plugins, with validation and rollback for plugins shared across apps or teams. CX Codex +1 181installs nemo-relay-plugin-adaptive-tuning nemo-relay-plugin-adaptive-tuning is a Claude Code skill that configures and evaluates NeMo Relay adaptive behavior from runtime signals, including telemetry, parallelism, hints, and prompt-cache planning. CX Codex +1 178installs

Project sponsors

Companies that keep this analytics open