heartmula — Claude Code skill: what it does, how to install
This skill guides local song creation with HeartMuLa from lyrics and tags. Use it when you want an open-source workflow for AI music generation.
MIT 230 installs ★ 265,216 updated 2026-10-25
npx skills add NousResearch/hermes-agent --skill heartmula -a claude-codeWhat heartmula does
heartmula guides an agent through HeartMuLa, which creates full songs from a lyrics file and a tags file. The setup covers HeartMuLa for music generation, HeartCodec for audio reconstruction, HeartTranscriptor for lyrics transcription, and HeartCLAP for audio-text alignment. The generated track is saved as an MP3.
The skill walks through cloning heartlib, creating a Python 3.10 environment, downloading three checkpoints, and running run_music_generation.py. It also covers upgrading datasets and transformers, plus two source patches needed with newer transformers versions. Tags go into tags.txt; lyrics can use sections such as [Verse] and [Chorus] in lyrics.txt.
A comfortable single-GPU setup calls for 16GB+ VRAM, while --lazy_load true lets the 3B model run with about 8GB VRAM. CPU generation can take 30-60+ minutes, and Triton does not provide GPU acceleration on macOS. Check those constraints before preparing the environment.
What it helps with
- generate a song from lyrics and tags
- prepare
heartlibwith its Python environment - download the HeartMuLa and HeartCodec checkpoints
- configure CUDA or CPU generation
- save the result as an MP3
Example prompts for the agent
- Generate a song from
lyrics.txtandtags.txtwith HeartMuLa. - Set up
heartlib, download the checkpoints, and write the track to./assets/output.mp3. - Show me how to use
--lazy_load trueand split HeartMuLa and HeartCodec across two GPUs.
How to install heartmula
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill heartmula -a claude-code2Manually: put the files in place
~/.claude/skills/heartmula/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/heartmula/
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill heartmula -a codex2Manually: put the files in place
.agents/skills/heartmula/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill heartmula -a cursor2Manually: put the files in place
.cursor/skills/heartmula/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install NousResearch/hermes-agent optional-skills/creative/heartmula2Manually: put the files in place
.github/skills/heartmula/SKILL.md1Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill heartmula -a gemini-cli2Platform CLI
gemini skills install https://github.com/NousResearch/hermes-agent --path optional-skills/creative/heartmulaManually: put the files in place
.devin/skills/heartmula/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add NousResearch/hermes-agent --skill heartmula -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about heartmula
How do I install heartmula in Claude Code?
Run npx skills add NousResearch/hermes-agent --skill heartmula -a claude-code, then check ~/.claude/skills/heartmula/SKILL.md.
What does heartmula do?
It guides HeartMuLa song generation from lyrics and tags, including environment setup, model downloads, and MP3 output.
What does heartmula need to run?
It requires Python 3.10 and the heartlib project. GPU use needs an NVIDIA setup with CUDA support in PyTorch; --lazy_load true targets a minimum of 8GB VRAM.
Source text
SKILL.md · MIT
---
name: heartmula
description: "HeartMuLa: Suno-like song generation from lyrics + tags."
version: 1.0.0
author: Teknium (teknium1), Hermes Agent
license: MIT
platforms: [linux, macos]
metadata:
hermes:
tags: [music, audio, generation, ai, heartmula, heartcodec, lyrics, songs]
related_skills: [songwriting-and-ai-music]
---
# HeartMuLa - Open-Source Music Generation
## Overview
HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags, with multilingual support. Generates full songs from lyrics + tags. Comparable to Suno for open-source. Includes:
- **HeartMuLa** - Music language model (3B/7B) for generation from lyrics + tags
- **HeartCodec** - 12.5Hz music codec for high-fidelity audio reconstruction
- **HeartTranscriptor** - Whisper-based lyrics transcription
- **HeartCLAP** - Audio-text alignment model
## When to Use
- User wants to generate music/songs from text descriptions
- User wants an open-source Suno alternative
- User wants local/offline music generation
- User asks about HeartMuLa, heartlib, or AI music generation
## Hardware Requirements
- **Minimum**: 8GB VRAM with `--lazy_load true` (loads/unloads models sequentially)
- **Recommended**: 16GB+ VRAM for comfortable single-GPU usage
- **Multi-GPU**: Use `--mula_device cuda:0 --codec_device cuda:1` to split across GPUs
- 3B model with lazy_load peaks at ~6.2GB VRAM
## Installation Steps
### 1. Clone Repository
```bash
cd ~/ # or desired directory
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
```
### 2. Create Virtual Environment (Python 3.10 required)
```bash
uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .
```
### 3. Fix Dependency Compatibility Issues
**IMPORTANT**: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:
```bash
# Upgrade datasets (old version incompatible with current pyarrow)
uv pip install --upgrade datasets
# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
uv pip install --upgrade transformers
```
### 4. Patch Source Code (Required for transformers 5.x)
**Patch 1 - RoPE cache fix** in `src/heartlib/heartmula/modeling_heartmula.py`:
In the `setup_caches` method of the `HeartMuLa` class, add RoPE reinitialization after the `reset_caches` try/except block and before the `with device:` block:
```python
# Re-initialize RoPE caches that were skipped during meta-device loading
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
module.rope_init()
module.to(device)
```
**Why**: `from_pretrained` creates model on meta device first; `Llama3ScaledRoPE.rope_init()` skips cache building on meta tensors, then never rebuilds after weights are loaded to real device.
**Patch 2 - HeartCodec loading fix** in `src/heartlib/pipelines/music_generation.py`:
Add `ignore_mismatched_sizes=True` to ALL `HeartCodec.from_pretrained()` calls (there are 2: the eager load in `__init__` and the lazy load in the `codec` property).
**Why**: VQ codebook `initted` buffers have shape `[1]` in checkpoint vs `[]` in model. Same data, just scalar vs 0-d tensor. Safe to ignore.
### 5. Download Model Checkpoints
```bash
cd heartlib # project root
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'
```
All 3 can be downloaded in parallel. Total size is several GB.
## GPU / CUDA
HeartMuLa uses CUDA by default (`--mula_device cuda --codec_device cuda`). No extra setup needed if the user has an NVIDIA GPU with PyTorch CUDA support installed.
- The installed `torch==2.4.1` includes CUDA 12.1 support out of the box
- `torchtune` may report version `0.4.0+cpu` — this is just package metadata, it still uses CUDA via PyTorch
- To verify GPU is being used, look for "CUDA memory" lines in the output (e.g. "CUDA memory before unloading: 6.20 GB")
- **No GPU?** You can run on CPU with `--mula_device cpu --codec_device cpu`, but expect generation to be **extremely slow** (potentially 30-60+ minutes for a single song vs ~4 minutes on GPU). CPU mode also requires significant RAM (~12GB+ free). If the user has no NVIDIA GPU, recommend using a cloud GPU service (Google Colab free tier with T4, Lambda Labs, etc.) or the online demo at https://heartmula.github.io/ instead.
## Usage
### Basic Generation
```bash
cd heartlib
. .venv/bin/activate
python ./examples/run_music_generation.py \
--model_path=./ckpt \
--version="3B" \
--lyrics="./assets/lyrics.txt" \
--tags="./assets/tags.txt" \
--save_path="./assets/output.mp3" \
--lazy_load true
```
### Input Formatting
**Tags** (comma-separated, no spaces):
```
piano,happy,wedding,synthesizer,romantic
```
or
```
rock,energetic,guitar,drums,male-vocal
```
**Lyrics** (use bracketed structural tags):
```
[Intro]
[Verse]
Your lyrics here...
[Chorus]
Chorus lyrics...
[Bridge]
Bridge lyrics...
[Outro]
```
### Key Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--max_audio_length_ms` | 240000 | Max length in ms (240s = 4 min) |
| `--topk` | 50 | Top-k sampling |
| `--temperature` | 1.0 | Sampling temperature |
| `--cfg_scale` | 1.5 | Classifier-free guidance scale |
| `--lazy_load` | false | Load/unload models on demand (saves VRAM) |
| `--mula_dtype` | bfloat16 | Dtype for HeartMuLa (bf16 recommended) |
| `--codec_dtype` | float32 | Dtype for HeartCodec (fp32 recommended for quality) |
### Performance
- RTF (Real-Time Factor) ≈ 1.0 — a 4-minute song takes ~4 minutes to generate
- Output: MP3, 48kHz stereo, 128kbps
## Pitfalls
1. **Do NOT use bf16 for HeartCodec** — degrades audio quality. Use fp32 (default).
2. **Tags may be ignored** — known issue (#90). Lyrics tend to dominate; experiment with tag ordering.
3. **Triton not available on macOS** — Linux/CUDA only for GPU acceleration.
4. **RTX 5080 incompatibility** reported in upstream issues.
5. The dependency pin conflicts require the manual upgrades and patches described above.
## Links
- Repo: https://github.com/HeartMuLa/heartlib
- Models: https://huggingface.co/HeartMuLa
- Paper: https://arxiv.org/abs/2601.10547
- License: Apache-2.0
© Teknium (teknium1), Hermes Agent · MIT · NousResearch/hermes-agent