Skip to main content
skill CX Codex

multimodal-llm — Claude Code skill: what it does, how to install

multimodal-llm — a skill for Claude Code and Codex that routes tasks across vision, speech, and video models. The agent activates it when handling image analysis, audio transcription, or video generation APIs.

MIT 243 installs ★ 305 updated 2026-11-21

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code
Open on GitHub

What multimodal-llm does

The skill provides nine implementation rules for pipelines spanning vision, audio processing, and AI video generation. It covers image captioning, visual question answering, document OCR, speech-to-text, text-to-speech, and video generation APIs including Kling v3, Sora 2, Veo 3.1, and Runway Gen-4.5.

Included reference tables define canonical model identifiers and trade-offs. For example, claude-opus-5-5 handles fine-grained visual inspection, while gemini-3.1-pro-preview processes multi-page documents and audio files up to 9.5 hours. Video integration sections cover async task polling, webhook callbacks, fal.ai and Vercel AI SDK integration, and multi-shot storyboard consistency.

Because user-invocable is set to false, the agent loads this skill automatically when encountering multimodal requirements rather than via manual slash commands. It requires Claude Code 2.1.277 or newer.

What it helps with

  • select vision models for OCR and PDF table extraction
  • configure long-form speech transcription with speaker diarization
  • integrate video generation APIs for Kling v3, Veo 3.1, or Sora 2
  • set up multi-scene video storyboards with consistent characters

Example prompts for the agent

  • Compare multiple PDF invoices and extract the total amounts into a JSON table
  • Write Python code to transcribe a meeting recording via Gemini with speaker diarization
  • Implement an async polling pipeline for Kling v3 video generation

How to install multimodal-llm

1Skills CLI (Vercel)

npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code

2Manually: put the files in place

~/.claude/skills/multimodal-llm/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/multimodal-llm/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about multimodal-llm

How to install multimodal-llm in Claude Code?

Run npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code. This installs the definition to ~/.claude/skills/multimodal-llm/SKILL.md.

What does multimodal-llm do?

It equips the coding agent with architectural patterns, token limits, and API polling logic for vision analysis, voice agents, transcription, and generative video providers.

What is required to run multimodal-llm?

It requires Claude Code 2.1.277 or later. It runs autonomously in background context when the agent encounters image, audio, or video generation tasks.

Source text

SKILL.md · MIT
---
name: multimodal-llm
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Vision, audio, video generation, and multimodal LLM patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1, Runway Gen-4.5), or building multimodal AI pipelines.
user-invocable: false
effort: high
metadata:
  archive-candidate: "pending an outside usage signal"
  owner-agent: "multimodal-specialist"
  category: mcp-enhancement
  version: "2.1.1"
  author: "OrchestKit"
  complexity: "high"
  tags: "vision, audio, video, multimodal, image, speech, transcription, tts, kling, sora, veo, video-generation"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---

# Multimodal LLM Patterns

Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via `gen4_turbo`).

> **Canonical model IDs** (Google IDs checked against ai.google.dev/gemini-api/docs/models on 2026-09-23):
>
> | Provider | Model IDs |
> |----------|-----------|
> | Anthropic | `claude-opus-5-5` (recommended, the default Opus since CC 2.1.280, same 2,576 px budget as Opus 5), `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-5-5`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`. `claude-fable-5` is Anthropic's **frontier tier above Opus** (GA 2026-07). Premium cost, never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spend |
> | OpenAI    | `gpt-5.5` (current flagship) |
> | Google    | `gemini-3.1-pro-preview` (flagship), `gemini-3.1-flash-lite` (cost) |
> | Veo       | `veo-3.1-generate-preview` / `veo-3.1-lite-generate-preview` / `veo-3.1-fast-generate-preview` |
> | Kling     | `kling-v3` (model_name field in Kling API) |
> | Runway    | `gen4_turbo` (product label: Gen-4.5) |

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Vision: Image Analysis](#vision-image-analysis) | 1 | HIGH | Image captioning, VQA, multi-image comparison, object detection |
| [Vision: Document Understanding](#vision-document-understanding) | 1 | HIGH | OCR, chart/diagram analysis, PDF processing, table extraction |
| [Vision: Model Selection](#vision-model-selection) | 1 | MEDIUM | Choosing provider, cost optimization, image size limits |
| [Audio: Speech-to-Text](#audio-speech-to-text) | 1 | HIGH | Transcription, speaker diarization, long-form audio |
| [Audio: Text-to-Speech](#audio-text-to-speech) | 1 | MEDIUM | Voice synthesis, expressive TTS, multi-speaker dialogue |
| [Audio: Model Selection](#audio-model-selection) | 1 | MEDIUM | Real-time voice agents, provider comparison, pricing |
| [Video: Model Selection](#video-model-selection) | 1 | HIGH | Choosing video gen provider (Kling, Sora, Veo, Runway) |
| [Video: API Patterns](#video-api-patterns) | 1 | HIGH | Async task polling, SDK integration, webhook callbacks |
| [Video: Multi-Shot](#video-multi-shot) | 1 | HIGH | Storyboarding, character elements, scene consistency |

**Total: 9 rules across 3 categories (Vision, Audio, Video Generation)**

## Vision: Image Analysis

Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set `max_tokens` and resize images before encoding.

| Rule | File | Key Pattern |
|------|------|-------------|
| Image Analysis | `rules/vision-image-analysis.md` | Base64 encoding, multi-image, bounding boxes |

## Vision: Document Understanding

Extract structured data from documents, charts, and PDFs using vision models.

| Rule | File | Key Pattern |
|------|------|-------------|
| Document Vision | `rules/vision-document.md` | PDF page ranges, detail levels, OCR strategies |

## Vision: Model Selection

Choose the right vision provider based on accuracy, cost, and context window needs.

| Rule | File | Key Pattern |
|------|------|-------------|
| Vision Models | `rules/vision-models.md` | Provider comparison, token costs, image limits |

## Audio: Speech-to-Text

Convert audio to text with speaker diarization, timestamps, and sentiment analysis.

| Rule | File | Key Pattern |
|------|------|-------------|
| Speech-to-Text | `rules/audio-speech-to-text.md` | Gemini long-form, GPT-4o-Transcribe, AssemblyAI features |

## Audio: Text-to-Speech

Generate natural speech from text with voice selection and expressive cues.

| Rule | File | Key Pattern |
|------|------|-------------|
| Text-to-Speech | `rules/audio-text-to-speech.md` | Gemini TTS, voice config, auditory cues |

## Audio: Model Selection

Select the right audio/voice provider for real-time, transcription, or TTS use cases.

| Rule | File | Key Pattern |
|------|------|-------------|
| Audio Models | `rules/audio-models.md` | Real-time voice comparison, STT benchmarks, pricing |

## Video: Model Selection

Choose the right video generation provider based on use case, duration, and budget.

| Rule | File | Key Pattern |
|------|------|-------------|
| Video Models | `rules/video-generation-models.md` | Kling vs Sora vs Veo vs Runway, pricing, capabilities |

## Video: API Patterns

Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.

| Rule | File | Key Pattern |
|------|------|-------------|
| API Integration | `rules/video-generation-patterns.md` | Kling REST, fal.ai SDK, Vercel AI SDK, task polling |

## Video: Multi-Shot

Generate multi-scene videos with consistent characters using storyboarding and character elements.

| Rule | File | Key Pattern |
|------|------|-------------|
| Multi-Shot | `rules/video-multi-shot.md` | Kling v3 character elements, 6-shot storyboards, identity binding |

## Key Decisions

| Decision | Recommendation |
|----------|----------------|
| High accuracy vision | `claude-opus-5-5` (default Opus since CC 2.1.280, 2,576 px vision budget, 3× what Opus 4.6 allotted; give it crop/analyze tools rather than more thinking, which is the cheaper lever on this model). (`claude-fable-5` is the frontier SOTA option, GA 2026-07 — premium cost, use only with explicit consent via the fable-spend-consent gate) |
| Long documents | `gemini-3.1-pro-preview` (1M+ context) |
| Cost-efficient vision | `gemini-3.1-flash-lite` (GA successor of the Flash-Lite preview, which was shut down 2026-05-25; shutdown scheduled 2027-05-07) |
| Video analysis | `gemini-3.1-pro-preview` (native video, supersedes 2.5 Pro) |
| Voice assistant | Grok Voice Agent on Grok 4.20 (fastest, <1s) |
| Emotional voice AI | Gemini Live API |
| Long audio transcription | `gemini-3.1-pro-preview` (9.5hr) |
| Speaker diarization | AssemblyAI or Gemini |
| Self-hosted STT | Whisper Large V3 |
| Character-consistent video | `kling-v3` (Character Elements 3.0) |
| Narrative video / storytelling | Sora 2 (best cause-and-effect coherence) |
| Cinematic B-roll | `veo-3.1-generate-preview` (camera control + polished motion) |
| Budget drafts | `veo-3.1-lite-generate-preview` (~$0.05/s, 720/1080p) |
| Mid-tier fast renders | `veo-3.1-fast-generate-preview` |
| Professional VFX | Runway `gen4_turbo` (Act-Two motion transfer) |
| High-volume social video | `kling-v3` Standard (~$0.20/video) |
| Open-source video gen | Wan 2.6 or LTX-2 |
| Lip-sync / avatar video | `kling-v3` (native lip-sync API) |

## Example

```python
import anthropic, base64

client = anthropic.Anthropic()
with open("image.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
        {"type": "text", "text": "Describe this image"}
    ]}]
)
```

## Common Mistakes

1. Not setting `max_tokens` on vision requests (responses truncated)
2. Sending oversized images without resizing (>2048px)
3. Using `high` detail level for simple yes/no classification
4. Using STT+LLM+TTS pipeline instead of native speech-to-speech
5. Not leveraging barge-in support for natural voice conversations
6. Using deprecated models (GPT-4V, Whisper-1)
7. Ignoring rate limits on vision and audio endpoints
8. Calling video generation APIs synchronously (they're async — poll or use callbacks)
9. Generating separate clips without character elements (characters look different each time)
10. Using Sora for high-volume social content (expensive, slow — use Kling Standard instead)

## Related Skills

- `ork-extras:rag-retrieval` - Multimodal RAG with image + text retrieval
- `ork-extras:llm-integration` - General LLM function calling patterns
- `streaming-api-patterns` - WebSocket patterns for real-time audio
- `ork:demo-producer` - Terminal demo videos (VHS, asciinema) — not AI video gen

© OrchestKit · MIT · yonatangross/orchestkit

Similar skills

All skills for AI and ML →

From the same set86

The whole set →
code-review-playbook code-review-playbook supplies structured review workflows, conventional comment formats, and security checklists. Triggers during pull request evaluations or author self-reviews. CX Codex +1 254installs python-backend python-backend — a skill for Claude Code and Codex providing async Python patterns when editing *.py or package files. Enforces conventions for FastAPI, SQLAlchemy 2.0, and asyncio. CX Codex +1 250installs accessibility accessibility — a Claude Code skill for building accessible web interfaces. It activates for work on screen reader support, keyboard navigation, focus handling, and ARIA components. CX Codex +1 246installs memory-fabric memory-fabric is a Claude Code skill that orchestrates knowledge graph retrieval, parses queries, deduplicates facts, and ranks results. It runs when designing or debugging agent memory mechanics. CX Codex +1 244installs database-patterns The database-patterns skill guides schema design, Alembic migrations, and database versioning. It fits work on normalization, migration branches, and schema drift. CX Codex +1 238installs architecture-patterns architecture-patterns enforces clean architecture, backend layer boundaries, and directory limits. Triggers when designing module boundaries or running refactoring tasks. CX Codex +1 236installs performance This skill helps diagnose slow web pages, React renders, backend requests, and LLM inference across a performance review. CX Codex +1 233installs github-operations The github-operations skill handles GitHub issues, pull requests, milestones, and Projects v2 with gh, REST, and GraphQL patterns. It activates for GitHub resource management. CX Codex +1 230installs

Project sponsors

Companies that keep this analytics open