offline-voice — Claude Code skill: what it does, how to install
offline-voice is a Claude Code skill that guides the setup of VieNeu-TTS narration and voice cloning. Activates when working on /api/tts, /api/voices, or narration debugging.
MIT ★ 147 updated 2026-11-13
npx skills add notivn/AIEVv2 --skill offline-voice -a claude-codeWhat offline-voice does
Documents the on-device VieNeu-TTS v3 Turbo synthesis setup alongside Gemini API narration. The local engine handles Vietnamese narration with English code-switching and clones voices from 3–8 seconds of reference audio.
Enforces two isolated setup tiers. Preset synthesis only demands pip install vieneu on ONNX Runtime taking ~30 MB without Torch. Voice cloning requires torch and torchaudio, pulling an extra 2–3 GB. Initial generation downloads roughly 1 GB of weights from Hugging Face.
Supplies verified fixes for dependency traps: monkeypatching torchaudio.load with soundfile to prevent TorchCodec loading failures on static FFmpeg, pinning av>=11,<17 for faster-whisper, and strict positional parsing for voice metadata labels.
What it helps with
- configure local VieNeu-TTS synthesis without API charges
- bypass TorchCodec import errors during voice cloning
- resolve PyAV version conflicts in faster-whisper transcriptions
- parse Vietnamese voice metadata without confusing gender and region
Example prompts for the agent
- Fix the 503 LOAD_FAILED error on the VieNeu local narration worker.
- How do I set up voice cloning without forcing full Torch on preset speech users?
- Apply the soundfile monkeypatch for torchaudio reference loading.
How to install offline-voice
1Skills CLI (Vercel)
npx skills add notivn/AIEVv2 --skill offline-voice -a claude-code2Manually: put the files in place
~/.claude/skills/offline-voice/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/offline-voice/
1Skills CLI (Vercel)
npx skills add notivn/AIEVv2 --skill offline-voice -a codex2Manually: put the files in place
.agents/skills/offline-voice/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add notivn/AIEVv2 --skill offline-voice -a cursor2Manually: put the files in place
.cursor/skills/offline-voice/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install notivn/AIEVv2 .claude/skills/offline-voice2Manually: put the files in place
.github/skills/offline-voice/SKILL.md1Skills CLI (Vercel)
npx skills add notivn/AIEVv2 --skill offline-voice -a gemini-cli2Platform CLI
gemini skills install https://github.com/notivn/AIEVv2 --path .claude/skills/offline-voiceManually: put the files in place
.devin/skills/offline-voice/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about offline-voice
How to install offline-voice in Claude Code?
Run npx skills add notivn/AIEVv2 --skill offline-voice -a claude-code. The file will be placed in ~/.claude/skills/offline-voice/SKILL.md.
What does offline-voice do?
It gives instructions for on-device VieNeu-TTS integration, worker runtime handling, and workarounds for torchaudio and audio library bugs.
What is required to run offline-voice?
Basic synthesis requires Python and vieneu running via ONNX Runtime. Cloning additionally requires torch, torchaudio, and 1 GB of disk cache for Hugging Face model files.
Source text
SKILL.md · MIT
---
name: offline-voice
description: How the on-device narration engine (VieNeu-TTS) and voice cloning work in this system - install tiers, the worker protocol, and the verified traps around torchaudio, voice metadata parsing and duration. Read when working on Text to video narration, the Voices page, the /api/voices or /api/tts endpoints, or when a user reports that offline voices or voice cloning fail to install or sound wrong.
---
# Offline voice - VieNeu-TTS and voice cloning
The system has **two narration engines running side by side**. They are deliberately not merged.
| | `gemini` | `vieneu` |
|---|---|---|
| Where it runs | Google API | The user's own machine |
| Cost | Per call | Free |
| Needs network | Yes | No |
| Voices | 30 preset | 14 Vietnamese preset + unlimited cloned |
| Voice cloning | **No** (API only accepts preset names) | **Yes** |
| Speed | A few seconds | RTF ~1.0 (about real time) |
| Setup | `GEMINI_API_KEY` | `pip install vieneu` |
Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while Gemini costs money on every preview click.
## Why VieNeu and not F5-TTS / XTTS / Chatterbox
Checked against the requirement "Vietnamese + English, cloning, and shippable in an open-source repo":
- **Chatterbox** (MIT, excellent English) has **no Vietnamese**. Disqualified on language.
- **F5-TTS** code is MIT but the **weights are CC-BY-NC**, and every Vietnamese finetune is CC-BY-NC-SA. Disqualified on licence.
- **viXTTS / XTTS-v2** inherit Coqui's CPML non-commercial licence. Disqualified on licence.
- **VietTTS** (dangvansam) is Apache-2.0 code but the installer is Linux-only, and its weights are CC-BY-NC too.
- **VieNeu-TTS** is Apache 2.0 on the code and on the **v3 Turbo weights** (`pnnbao-ump/VieNeu-TTS-v3-Turbo`, trained from scratch on ~10k hours, not a finetune), plus Apache 2.0 on its codec dependency `OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano`. Vietnamese-first with English code-switching, torch-free on CPU via ONNX, clones from 3-8 seconds of reference audio.
If a future engine is evaluated, check the **weights** licence separately from the code licence. That is where every other candidate failed.
**Licence trap inside the package itself.** `vieneu/assets/voices.json` carries `"license": "CC BY-NC 4.0"` and `"Model and voices are for non-commercial use only"`. That notice is scoped to **that file**, which is the **v2** preset pack: 6 voices named `Binh, Tuyen, Vinh, Doan, Ly, Ngoc`, default `Binh`. We run `mode="v3turbo"`, which reads `voices_v3_turbo.json` (14 Vietnamese voices, default `Phạm Tuyên`). The two sets have **zero overlap** - verified by comparing the key sets - so nothing on our path is NC-licensed. If anyone ever switches the engine mode away from `v3turbo`, or a preset named `Binh`/`Tuyen`/`Vinh`/`Doan`/`Ly`/`Ngoc` shows up in the picker, the commercial position changes and must be rechecked.
## Install tiers - keep them separate
Two independent levels. Never collapse them into one check:
1. **Speech only**: `pip install vieneu`. Torch-free (ONNX Runtime), ~30 MB of wheels. This is enough for all 14 preset voices.
2. **Cloning**: additionally `pip install torch torchaudio`, 2-3 GB.
`start/doctor.mjs` reports these as `vieneu` and `vieneu-clone`, and only asks about torch once `vieneu` is present. Forcing a 2-3 GB torch download on someone who only wants offline narration is the mistake to avoid.
The first synthesis after a server start downloads ~1 GB of model from HuggingFace and takes 30s; subsequent process starts take ~15s to load. Always surface this in the UI, otherwise the first click looks like a hang.
## Known issues
**Symptom:** every transcription (video projects, Text to video, narration) fails with `TypeError: open() got an unexpected keyword argument 'metadata_errors'` from `faster_whisper/audio.py`.
**Cause:** PyAV 17+ dropped `metadata_errors`, but faster-whisper 1.2.x still passes it to `av.open()` (verified 2026-10: `av` 19.0.1 + `faster-whisper` 1.2.1). An unpinned `pip install faster-whisper` pulls the newest `av`.
**Fix:** `.runtime/venv/Scripts/python.exe -m pip install "av>=11,<17"` (16.1.0 verified). `start/doctor.mjs` now pins it in the faster-whisper fix, but a venv that already has av 19 must be downgraded by hand - the doctor's probe only checks `import faster_whisper`, which still succeeds.
**Symptom:** every VieNeu call (narration, Text to video, voice preview) returns `503 LOAD_FAILED` for the rest of the server's life, yet the same Python loads `Vieneu()` and synthesizes fine from the command line.
**Cause:** one failed model load (seen once as `External data path escapes model directory` while the ~1GB model was still being fetched on first use) set `loadFailure` in `ttsLocal.ts`. That flag overrode every later probe, and the only way to clear it was `probeLocalEngine(true)` - which no route called. An agent mid-edit had no way out but asking the user to restart the backend.
**Fix:** the flag now expires after 2 minutes (the next call retries the real load), and `GET /api/tts/engines?refresh=1` - the Voices page "Kiểm tra lại" button - clears it immediately. To check a machine by hand, run the worker with the server's env (`HF_HOME` and `HUGGINGFACE_HUB_CACHE` both = `.runtime/models`) and send `{"id":1,"cmd":"synth","text":"Xin chào","voice":"Mai Anh","out":"<abs path>.wav"}` on stdin.
**Symptom:** Cloning fails with `ImportError: TorchCodec is required for load_with_torchcodec`, or `RuntimeError: Could not load libtorchcodec`.
**Cause:** `add_voice()` reads the reference through `torchaudio.load()`. From torchaudio 2.9 every backend routes through `torchcodec`, which needs FFmpeg **shared libraries** - the usual Windows FFmpeg builds are static, so it can never load. Installing `torchcodec` does not fix it. Passing `backend="soundfile"` does not fix it either (verified: all three backends raise the same error).
**Fix:** monkeypatch `torchaudio.load` to use `soundfile` **before importing vieneu**. `soundfile` is already a vieneu dependency, so this adds nothing to install. Guard it in try/except so that a machine without torch still gets preset voices.
```python
import torch, torchaudio, soundfile as sf
def _load(uri, *a, **kw):
data, sr = sf.read(str(uri), dtype="float32", always_2d=True)
return torch.from_numpy(data.T).contiguous(), sr
torchaudio.load = _load
```
**Symptom:** Voices are filed under the wrong region, or every male voice looks southern.
**Cause:** `list_preset_voices()` returns labels shaped `Name — <Nam|Nữ> · <Bắc|Trung|Nam> · Phong cách <...>`. Field 1 is **gender**, field 2 is **region**, and the word **"Nam" means both "male" and "southern"**.
**Fix:** parse strictly by position, never by searching for the substring "Nam". Regression cases: `Thái Sơn` = gender nam / region nam; `Mai Anh` = gender nu / region bac.
**Symptom:** Cloned voices vanish after a restart, or a stray voice appears in the preset list and leaks into vieneu's own error messages.
**Cause:** `tts.save_voices()` (and `add_voice(..., save=True)`) rewrites `vieneu/assets/voices_v3_turbo.json` inside site-packages, permanently adding your test voice as a 15th "preset". This happened for real during development.
**Fix:** never call either. The store lives at `assets/voices/` (`library.json` plus `<id>/ref.wav`) and voices are re-registered into the worker on demand with `add_voice(..., save=False)`. Registration costs ~3s and is lost whenever the worker restarts, so track what the live process actually has. As defence in depth the worker drops any preset whose description does not parse into `gender · region · style`, and `listLocalVoices()` drops presets colliding with a store id - a contaminated install still yields a clean 14. To repair one by hand, delete the offending key from `voices_v3_turbo.json` or `pip install --force-reinstall vieneu`.
**Symptom:** A storytelling voice reads like a newsreader.
**Cause:** `infer()` has its own `style=` argument defaulting to `"tu_nhien"`, which is independent of the voice you picked.
**Fix:** look up each preset's own `style` (`get_preset_voice(name)` returns `{gender, style, description, ...}` as real fields) and pass it into `infer()`. Only **region** has to be parsed out of the description string.
**Symptom:** Timeline drifts against the narration.
**Cause:** estimating duration from character count.
**Fix:** every duration is measured with ffprobe, exactly as on the Gemini path. This is not engine-specific advice - see the `video-pipeline` skill.
## Architecture notes
- `apps/server/python/vieneu_worker.py` is a long-lived worker speaking JSON-lines on stdin/stdout. **stdout is protocol only**; all logging goes to stderr. It must `sys.stdout.reconfigure(encoding="utf-8")` first thing or Vietnamese corrupts on the Windows cp1252 console.
- `ping` must answer **without loading the model** - it backs the availability check.
- One worker, requests serialised (the model is not re-entrant), 10-minute idle shutdown because it holds ~1 GB of RAM.
- Output is already **48 kHz mono**, which equals the pipeline's `OUT_SAMPLE_RATE`, so no resampling. It still goes through the same `TRIM_FILTER` and gap-insertion path as Gemini in `synthScript()` - two engines with two joining strategies would drift audibly at the seams.
- The Perth watermark is left at its default (on). It is inaudible and is the responsible default for cloned speech.
## Reference audio guidance for users
3-8 seconds, clean speech, no background music, no noise. Longer is **not** better - the model only uses the head, so anything past ~8s is discarded. Below 3s the embedding is too weak and the clone stops resembling the speaker.
© notivn · MIT · notivn/AIEVv2