multimodal-llm — Claude Code skill: what it does, how to install
multimodal-llm — a skill for Claude Code and Codex that routes tasks across vision, speech, and video models. The agent activates it when handling image analysis, audio transcription, or video generation APIs.
MIT 243 installs ★ 305 updated 2026-11-21
npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-codeWhat multimodal-llm does
The skill provides nine implementation rules for pipelines spanning vision, audio processing, and AI video generation. It covers image captioning, visual question answering, document OCR, speech-to-text, text-to-speech, and video generation APIs including Kling v3, Sora 2, Veo 3.1, and Runway Gen-4.5.
Included reference tables define canonical model identifiers and trade-offs. For example, claude-opus-5-5 handles fine-grained visual inspection, while gemini-3.1-pro-preview processes multi-page documents and audio files up to 9.5 hours. Video integration sections cover async task polling, webhook callbacks, fal.ai and Vercel AI SDK integration, and multi-shot storyboard consistency.
Because user-invocable is set to false, the agent loads this skill automatically when encountering multimodal requirements rather than via manual slash commands. It requires Claude Code 2.1.277 or newer.
What it helps with
- select vision models for OCR and PDF table extraction
- configure long-form speech transcription with speaker diarization
- integrate video generation APIs for Kling v3, Veo 3.1, or Sora 2
- set up multi-scene video storyboards with consistent characters
Example prompts for the agent
- Compare multiple PDF invoices and extract the total amounts into a JSON table
- Write Python code to transcribe a meeting recording via Gemini with speaker diarization
- Implement an async polling pipeline for Kling v3 video generation
How to install multimodal-llm
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code2Manually: put the files in place
~/.claude/skills/multimodal-llm/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/multimodal-llm/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill multimodal-llm -a codex2Manually: put the files in place
.agents/skills/multimodal-llm/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill multimodal-llm -a cursor2Manually: put the files in place
.cursor/skills/multimodal-llm/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install yonatangross/orchestkit src/skills/multimodal-llm2Manually: put the files in place
.github/skills/multimodal-llm/SKILL.md1Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill multimodal-llm -a gemini-cli2Platform CLI
gemini skills install https://github.com/yonatangross/orchestkit --path src/skills/multimodal-llmManually: put the files in place
.devin/skills/multimodal-llm/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add yonatangross/orchestkit --skill multimodal-llm -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about multimodal-llm
How to install multimodal-llm in Claude Code?
Run npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code. This installs the definition to ~/.claude/skills/multimodal-llm/SKILL.md.
What does multimodal-llm do?
It equips the coding agent with architectural patterns, token limits, and API polling logic for vision analysis, voice agents, transcription, and generative video providers.
What is required to run multimodal-llm?
It requires Claude Code 2.1.277 or later. It runs autonomously in background context when the agent encounters image, audio, or video generation tasks.
Source text
SKILL.md · MIT
---
name: multimodal-llm
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Vision, audio, video generation, and multimodal LLM patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1, Runway Gen-4.5), or building multimodal AI pipelines.
user-invocable: false
effort: high
metadata:
archive-candidate: "pending an outside usage signal"
owner-agent: "multimodal-specialist"
category: mcp-enhancement
version: "2.1.1"
author: "OrchestKit"
complexity: "high"
tags: "vision, audio, video, multimodal, image, speech, transcription, tts, kling, sora, veo, video-generation"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---
# Multimodal LLM Patterns
Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via `gen4_turbo`).
> **Canonical model IDs** (Google IDs checked against ai.google.dev/gemini-api/docs/models on 2026-09-23):
>
> | Provider | Model IDs |
> |----------|-----------|
> | Anthropic | `claude-opus-5-5` (recommended, the default Opus since CC 2.1.280, same 2,576 px budget as Opus 5), `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-5-5`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`. `claude-fable-5` is Anthropic's **frontier tier above Opus** (GA 2026-07). Premium cost, never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spend |
> | OpenAI | `gpt-5.5` (current flagship) |
> | Google | `gemini-3.1-pro-preview` (flagship), `gemini-3.1-flash-lite` (cost) |
> | Veo | `veo-3.1-generate-preview` / `veo-3.1-lite-generate-preview` / `veo-3.1-fast-generate-preview` |
> | Kling | `kling-v3` (model_name field in Kling API) |
> | Runway | `gen4_turbo` (product label: Gen-4.5) |
## Quick Reference
| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Vision: Image Analysis](#vision-image-analysis) | 1 | HIGH | Image captioning, VQA, multi-image comparison, object detection |
| [Vision: Document Understanding](#vision-document-understanding) | 1 | HIGH | OCR, chart/diagram analysis, PDF processing, table extraction |
| [Vision: Model Selection](#vision-model-selection) | 1 | MEDIUM | Choosing provider, cost optimization, image size limits |
| [Audio: Speech-to-Text](#audio-speech-to-text) | 1 | HIGH | Transcription, speaker diarization, long-form audio |
| [Audio: Text-to-Speech](#audio-text-to-speech) | 1 | MEDIUM | Voice synthesis, expressive TTS, multi-speaker dialogue |
| [Audio: Model Selection](#audio-model-selection) | 1 | MEDIUM | Real-time voice agents, provider comparison, pricing |
| [Video: Model Selection](#video-model-selection) | 1 | HIGH | Choosing video gen provider (Kling, Sora, Veo, Runway) |
| [Video: API Patterns](#video-api-patterns) | 1 | HIGH | Async task polling, SDK integration, webhook callbacks |
| [Video: Multi-Shot](#video-multi-shot) | 1 | HIGH | Storyboarding, character elements, scene consistency |
**Total: 9 rules across 3 categories (Vision, Audio, Video Generation)**
## Vision: Image Analysis
Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set `max_tokens` and resize images before encoding.
| Rule | File | Key Pattern |
|------|------|-------------|
| Image Analysis | `rules/vision-image-analysis.md` | Base64 encoding, multi-image, bounding boxes |
## Vision: Document Understanding
Extract structured data from documents, charts, and PDFs using vision models.
| Rule | File | Key Pattern |
|------|------|-------------|
| Document Vision | `rules/vision-document.md` | PDF page ranges, detail levels, OCR strategies |
## Vision: Model Selection
Choose the right vision provider based on accuracy, cost, and context window needs.
| Rule | File | Key Pattern |
|------|------|-------------|
| Vision Models | `rules/vision-models.md` | Provider comparison, token costs, image limits |
## Audio: Speech-to-Text
Convert audio to text with speaker diarization, timestamps, and sentiment analysis.
| Rule | File | Key Pattern |
|------|------|-------------|
| Speech-to-Text | `rules/audio-speech-to-text.md` | Gemini long-form, GPT-4o-Transcribe, AssemblyAI features |
## Audio: Text-to-Speech
Generate natural speech from text with voice selection and expressive cues.
| Rule | File | Key Pattern |
|------|------|-------------|
| Text-to-Speech | `rules/audio-text-to-speech.md` | Gemini TTS, voice config, auditory cues |
## Audio: Model Selection
Select the right audio/voice provider for real-time, transcription, or TTS use cases.
| Rule | File | Key Pattern |
|------|------|-------------|
| Audio Models | `rules/audio-models.md` | Real-time voice comparison, STT benchmarks, pricing |
## Video: Model Selection
Choose the right video generation provider based on use case, duration, and budget.
| Rule | File | Key Pattern |
|------|------|-------------|
| Video Models | `rules/video-generation-models.md` | Kling vs Sora vs Veo vs Runway, pricing, capabilities |
## Video: API Patterns
Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.
| Rule | File | Key Pattern |
|------|------|-------------|
| API Integration | `rules/video-generation-patterns.md` | Kling REST, fal.ai SDK, Vercel AI SDK, task polling |
## Video: Multi-Shot
Generate multi-scene videos with consistent characters using storyboarding and character elements.
| Rule | File | Key Pattern |
|------|------|-------------|
| Multi-Shot | `rules/video-multi-shot.md` | Kling v3 character elements, 6-shot storyboards, identity binding |
## Key Decisions
| Decision | Recommendation |
|----------|----------------|
| High accuracy vision | `claude-opus-5-5` (default Opus since CC 2.1.280, 2,576 px vision budget, 3× what Opus 4.6 allotted; give it crop/analyze tools rather than more thinking, which is the cheaper lever on this model). (`claude-fable-5` is the frontier SOTA option, GA 2026-07 — premium cost, use only with explicit consent via the fable-spend-consent gate) |
| Long documents | `gemini-3.1-pro-preview` (1M+ context) |
| Cost-efficient vision | `gemini-3.1-flash-lite` (GA successor of the Flash-Lite preview, which was shut down 2026-05-25; shutdown scheduled 2027-05-07) |
| Video analysis | `gemini-3.1-pro-preview` (native video, supersedes 2.5 Pro) |
| Voice assistant | Grok Voice Agent on Grok 4.20 (fastest, <1s) |
| Emotional voice AI | Gemini Live API |
| Long audio transcription | `gemini-3.1-pro-preview` (9.5hr) |
| Speaker diarization | AssemblyAI or Gemini |
| Self-hosted STT | Whisper Large V3 |
| Character-consistent video | `kling-v3` (Character Elements 3.0) |
| Narrative video / storytelling | Sora 2 (best cause-and-effect coherence) |
| Cinematic B-roll | `veo-3.1-generate-preview` (camera control + polished motion) |
| Budget drafts | `veo-3.1-lite-generate-preview` (~$0.05/s, 720/1080p) |
| Mid-tier fast renders | `veo-3.1-fast-generate-preview` |
| Professional VFX | Runway `gen4_turbo` (Act-Two motion transfer) |
| High-volume social video | `kling-v3` Standard (~$0.20/video) |
| Open-source video gen | Wan 2.6 or LTX-2 |
| Lip-sync / avatar video | `kling-v3` (native lip-sync API) |
## Example
```python
import anthropic, base64
client = anthropic.Anthropic()
with open("image.png", "rb") as f:
b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.messages.create(
model="claude-opus-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
{"type": "text", "text": "Describe this image"}
]}]
)
```
## Common Mistakes
1. Not setting `max_tokens` on vision requests (responses truncated)
2. Sending oversized images without resizing (>2048px)
3. Using `high` detail level for simple yes/no classification
4. Using STT+LLM+TTS pipeline instead of native speech-to-speech
5. Not leveraging barge-in support for natural voice conversations
6. Using deprecated models (GPT-4V, Whisper-1)
7. Ignoring rate limits on vision and audio endpoints
8. Calling video generation APIs synchronously (they're async — poll or use callbacks)
9. Generating separate clips without character elements (characters look different each time)
10. Using Sora for high-volume social content (expensive, slow — use Kling Standard instead)
## Related Skills
- `ork-extras:rag-retrieval` - Multimodal RAG with image + text retrieval
- `ork-extras:llm-integration` - General LLM function calling patterns
- `streaming-api-patterns` - WebSocket patterns for real-time audio
- `ork:demo-producer` - Terminal demo videos (VHS, asciinema) — not AI video gen
© OrchestKit · MIT · yonatangross/orchestkit