Skip to main content
skill CX Codex

audio-reactive-video-composition — Claude Code skill

audio-reactive-video-composition is a Claude Code skill that maps audio features to deterministic video timing. It governs cuts, motion, and typography driven by beats, onsets, or spectral changes.

MIT 170 installs ★ 252 updated 2026-07-14

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add calesthio/generative-media-skills --skill audio-reactive-video-composition -a claude-code
Open on GitHub

What audio-reactive-video-composition does

The skill translates audio sources into auditable cue maps and deterministic visual behavior. It treats detector outputs conservatively: onsets are not automatically beats, and acoustic clusters do not establish verse or chorus labels. Workflows route through rhythmic tracking when tempo is stable, mixed confidence, or non-rhythmic cues based on speech phrases, silence, and spectral change.

Frame presentation times rely on exact rational numbers such as 30000/1001, anchoring visual state to frame numbers rather than playback clocks. Continuous values like RMS loudness and spectral centroid require normalization across track sections before mapping to graphics.

What it helps with

  • build an auditable audio cue map for video timelines
  • align cuts and motion to beats, onsets, or speech phrases
  • normalize energy and spectral measurements for visual parameters
  • calculate exact frame presentation numbers from rational frame rates

Example prompts for the agent

  • Generate a cue map from track.wav and mark strong onsets for visual accents
  • Map spectral centroid values from analysis.json to layer scale at 30000/1001 fps
  • Configure non-rhythmic routing for a spoken word audio track

How to install audio-reactive-video-composition

1Skills CLI (Vercel)

npx skills add calesthio/generative-media-skills --skill audio-reactive-video-composition -a claude-code

2Manually: put the files in place

~/.claude/skills/audio-reactive-video-composition/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/audio-reactive-video-composition/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about audio-reactive-video-composition

How do I install audio-reactive-video-composition in Claude Code?

Run npx skills add calesthio/generative-media-skills --skill audio-reactive-video-composition -a claude-code. Alternatively, place the file at ~/.claude/skills/audio-reactive-video-composition/SKILL.md.

What does the audio-reactive-video-composition skill do?

It provides production guidelines for turning measured audio features into deterministic frame-accurate animation. The skill establishes cue-map schemas and routes features into rhythmic or non-rhythmic visual timelines.

How does the skill ensure deterministic video timing?

It requires output frame rates as rational numbers and drives visual state from cue data and frame index rather than wall-clock playback.

Source text

SKILL.md · MIT
---
name: audio-reactive-video-composition
description: Provider-independent production guidance for translating measured audio features into deterministic video timing and motion. Use for beat-, onset-, phrase-, energy-, silence-, or spectrum-reactive visualizers, edits, typography, and generated compositions; not for music-video concept direction, audio generation, or a runtime-specific recipe.
---

# Audio-reactive video composition

Use this skill to turn an approved audio source into an auditable cue map and then into bounded visual behavior. The central contract is:

$$
\text{source audio} \rightarrow \text{measured features} \rightarrow
\text{confidence-bearing cues} \rightarrow \text{reviewed anchors} \rightarrow
\text{deterministic visual timeline}
$$

Do not equate a detector output with editorial meaning. An onset is not automatically a beat, cut, drop, or lyric accent; an acoustic cluster is not automatically a verse or chorus.

## Evidence stance

- **Documented fact:** behavior stated by official analysis libraries, standards, or cited research.
- **Production heuristic:** a practical mapping that must be tested on the track and audience.
- **Empirical observation:** a measured result from the supplied audio, analyzer run, render, or playback.

Analysis algorithms, defaults, and model behavior are volatile. Facts were verified **2026-07-12**. Pin the decoder, library, model, parameters, and random seeds used for each production.

## Scope

This skill owns source custody, analysis policy, confidence interpretation, rhythmic/non-rhythmic routing, cue promotion, feature-to-visual mappings, rational frame alignment, accessibility, and render QA.

It does not own music-video narrative or artist branding, music generation, mastering, source separation, transcription, lyric writing, or a HyperFrames/Remotion/FFmpeg-specific implementation.

## Freeze the audio contract

Record before analysis:

- source path/URI, SHA-256, acquisition source, rights basis, and restrictions;
- selected stream, codec, native sample rate, channels/layout, start timestamp, and duration;
- decoder/resampler and versions;
- canonical PCM format, channel/downmix policy, analysis sample rate, window/hop sizes, centering, and padding;
- analyzer/library/model versions, priors, thresholds, and seeds;
- trim offsets and source time origin;
- master output frame rate as a rational number;
- transcript/lyrics source, language, timing provenance, review status, and separate rights basis.

Use sample indices as the primary analysis clock where possible. Derived seconds should not replace exact source positions.

## Interpret features conservatively

**Documented facts:**

- `librosa.beat.beat_track` estimates tempo from onset strength and selects beat positions consistent with it; it does not return calibrated beat confidence.
- `librosa.onset.onset_detect` peak-picks an onset-strength envelope. Onsets estimate event attacks, not semantic accents.
- Predominant local pulse can model changing tempo, but remains an estimate.
- Essentia confidence values are algorithm-specific; they are not universal probabilities, and some routes return an unusable zero confidence.
- Beat evaluation commonly accounts for half/double-tempo metrical ambiguity and uses tolerances rather than exact equality.
- Structural boundaries and structural labels are separate tasks. Cluster labels such as A/B/C do not establish verse/chorus meaning.
- RMS is an energy measure. EBU R128 loudness uses defined weighting and windows. Neither is a direct emotion score.
- Spectral centroid, bandwidth, and contrast describe spectrum distribution; they do not mean happiness, tension, or quality.
- Silence detection is relative to a declared reference and threshold.
- pYIN voicing means pitch periodicity, not proof that a person is speaking or singing.

Keep raw candidates distinct from human-promoted anchors.

## Cue-map contract

Each cue should include:

```json
{
  "id": "accent-014",
  "type": "onset",
  "time_samples": 417312,
  "time_seconds": 9.46399,
  "value": 0.82,
  "units": "normalized-onset-strength",
  "analyzer": "record exact library and algorithm",
  "confidence": null,
  "confidence_semantics": "not supplied by analyzer",
  "profile_hash": "sha256:...",
  "status": "promoted-anchor",
  "editorial_role": "single visual accent",
  "reason": "reviewed strong transient before section change"
}
```

For interval features, include exact end samples/times. Record rejected candidates rather than deleting evidence. Keep acoustic section names neutral until lyrics, metadata, or human review supports functional labels.

## Choose a routing mode

### Rhythmic

Use when the selected tracker has useful evidence, local tempo is stable enough, and beat/onset results agree. Periodic motion may follow beats; selected strong accents or reviewed section changes may motivate cuts.

### Mixed confidence

Use reliable rhythmic windows locally. Elsewhere shift to onsets, transcript phrases, energy, silence, or manual anchors. Never extrapolate a grid through a failed interval.

### Non-rhythmic

Use reviewed speech/lyric phrases, silence, energy contour, spectral change, and manually confirmed macro anchors. This suits rubato, ambient work, spoken word, sparse recordings, and free improvisation.

## Map features to visual behavior

Production heuristics:

- Give each feature a limited visual responsibility.
- Normalize continuous values with robust per-track or per-section statistics, not one extreme maximum.
- Smooth noise and use hysteresis before switching visual states.
- Clamp every parameter and define neutral behavior for missing/invalid values.
- Use macro section anchors, medium phrase anchors, and sparse accents rather than cutting on every event.
- Reserve low-energy or silent spans for holds, reading, resets, and visual breathing room.
- Protect lyric lines and important vocal phrases from competing cuts and overlays.
- Preserve one cue map across aspect ratios; recompose layout, not timing.

Avoid quantitative-looking mappings that imply false measurement. If energy maps to scale, define the exact bounded range and do not call it emotion.

## Deterministic frame timing

Keep exact rates such as $30000/1001$ rational. For each cue, define a rounding policy. A conservative visual-response policy assigns the cue to the first output frame whose presentation time is at or after the audio event:

$$
f = \left\lceil t_{event} \cdot \frac{fps_{num}}{fps_{den}} \right\rceil
$$

Record both the original event time and mapped frame. Verify actual frame presentation timestamps after encoding because an encoder may duplicate or drop frames to satisfy constant-frame-rate output.

The visual state at frame $f$ must derive from cue data and frame time, not wall-clock playback. Seed procedural mappings and freeze analyzer output before distributed rendering.

## Lyrics, vocals, and captions

Treat transcript or lyric timing as a separate evidence stream. Do not use pitch voicing as vocal detection. Human-review names, lyrics, line boundaries, and timing before they control typography.

Prerecorded synchronized media with meaningful speech needs accurate captions. Captions should include meaningful non-speech sound where needed. A stylized lyric layer does not automatically replace accessible captions or transcript.

## Safety, rights, and provenance

WCAG 2.2 SC 2.3.1 limits flashing above three times in one second unless below general/red-flash thresholds. Test loops while looping and at the largest intended scale. Reduced motion does not make unsafe flashing safe.

Provide a lower-motion version when large displacement, zoom, shake, or dense event response may cause discomfort. Reduce event density and travel, not merely output FPS.

The musical work, lyrics, and sound recording can carry separate rights. Possession of a file does not grant synchronization, adaptation, or distribution rights. Record source and transformation provenance; do not claim metadata proves authenticity or permission.

## QA

1. Re-run identical input/configuration and compare canonical cue-map hashes.
2. Audition click-marked candidate and promoted anchors.
3. Inspect tempo-level ambiguity and failed-confidence windows.
4. Compare source timing against decoded/trimmed master and non-zero timestamps.
5. Inspect frames immediately before, on, and after every macro anchor.
6. Verify aspect variants share cue IDs and frame assignments.
7. Probe output frame PTS, duration, and audio sync.
8. Review captions/lyrics, reduced motion, and flash safety.
9. Preserve source hash, profile, cue map, mapping table, render versions, and approvals.

## Example 1: constant-tempo electronic visualizer

This is a complete example, not a mandatory formula.

**Intent:** 30-second 9:16 and 16:9 visualizer from an authorized 44.1 kHz stereo instrumental at 30 fps.

**Approach:** confirm stable tempo with a documented analyzer and inspect local pulse. Retain all beat/onset candidates. Promote every fourth beat for medium choreography, reviewed top-strength transients for sparse accents, and reviewed recurrence changes for macro transitions. Beat phase drives scale only from 1.000 to 1.035; low-band energy controls bounded depth; centroid controls a narrow texture-density range. A six-beat low-energy span holds title copy.

Map all anchors to the first frame at or after their sample time. Both aspect variants use identical cue IDs. QA click tracks, rerun hashes, PTS, duration, final-size text, and full-screen flashing.

**Likely failure:** half/double-tempo ambiguity. Repair by documenting the metrical level selected for visual periodicity without rewriting the raw detections.

## Example 2: rubato spoken word

This is a complete example, not a mandatory formula.

**Intent:** 75-second poem with ambient bed at 24 fps.

**Approach:** global beat trackers disagree, so use the non-rhythmic route. Reviewed transcript line starts/ends are primary anchors; pauses reset the field; sustained loudness rises control subtle expansion; acoustic-change candidates remain review prompts. Each stanza establishes a stable visual field, and punctuation settles motion. No beat cuts or word-by-word scaling.

The vertical variant reflows text but retains timing. QA muted-caption comprehension, audio-only clarity, every line boundary, breath-adjacent silence, reduced motion, flashing, and separate text/recording rights.

**Likely failure:** an acoustic boundary lands inside a sentence. Keep it as a low-confidence event and reject it as an editorial anchor.

## Sources

Verified 2026-07-12:

- librosa beat, onset, PLP, tempo, segmentation, RMS, spectral, split, and pYIN documentation: https://librosa.org/doc/latest/
- Essentia rhythm and loudness references: https://essentia.upf.edu/reference/
- `mir_eval` beat, onset, and segment metrics: https://mir-eval.readthedocs.io/
- Ellis, Dynamic Programming Beat Tracking: https://www.ee.columbia.edu/~dpwe/pubs/Ellis07-beattrack.pdf
- FFmpeg and ffprobe documentation: https://ffmpeg.org/documentation.html
- EBU R 128: https://tech.ebu.ch/publications/r128
- WCAG flashing and captions: https://www.w3.org/WAI/WCAG22/Understanding/three-flashes-or-below-threshold.html and https://www.w3.org/WAI/WCAG22/Understanding/captions-prerecorded.html
- U.S. Copyright Office guidance for musicians: https://www.copyright.gov/engage/musicians/

© calesthio · MIT · calesthio/generative-media-skills

Similar skills

All skills for Design →

From the same set149

The whole set →
real-estate-content-production real-estate-content-production is a Claude Code skill for producing and reviewing real estate marketing content. It applies to listing photos, video, copy, virtual staging, and other property visuals. CX Codex +1 178installs product-ad-production The product-ad-production skill for Claude Code designs, scripts, and quality-controls video ads from evidence-backed briefs. CX Codex +1 177installs resemble-chatterbox resemble-chatterbox is a Claude Code skill that guides text-to-speech and voice cloning workflows with Resemble AI models. It directs model selection between local weights and hosted API routes. CX Codex +1 175installs explainer-video-production explainer-video-production is a Claude Code skill that turns a topic, brief, or research set into an explainer video plan. It covers claims, storyboards, captions, localization, and QA. CX Codex +1 173installs threejs-scene-composition threejs-scene-composition is a Claude Code skill for assembling and animating Three.js scenes for rendered media. It covers 3D product shots, title sequences, procedural worlds, and render QA. CX Codex +1 169installs google-veo google-veo is a Claude Code skill that orchestrates Veo 3.1 video generation via the Gemini Developer API and Google Cloud. It structures shot parameters, audio, frame controls, and async jobs. CX Codex +1 168installs brand-launch-film-production brand-launch-film-production is a Claude Code skill for planning and producing brand launch films. It supports work from the first concept through review and delivery across placements. CX Codex +1 167installs saas-product-demo-production saas-product-demo-production is a Claude Code skill that structures SaaS demo video production. It handles planning and scripting for feature launches, onboarding, and hero walkthroughs. CX Codex +1 165installs

Project sponsors

Companies that keep this analytics open