# multimodal-llm

multimodal-llm — a skill for Claude Code and Codex that routes tasks across vision, speech, and video models. The agent activates it when handling image analysis, audio transcription, or video generation APIs.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 243 installs · ★ 305 repo stars
- Author: OrchestKit
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/yonatangross/orchestkit/tree/HEAD/src/skills/multimodal-llm/SKILL.md
- Updated: 2026-11-21
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/multimodal-llm

## What it does

The skill provides nine implementation rules for pipelines spanning vision, audio processing, and AI video generation. It covers image captioning, visual question answering, document OCR, speech-to-text, text-to-speech, and video generation APIs including Kling v3, Sora 2, Veo 3.1, and Runway Gen-4.5.

Included reference tables define canonical model identifiers and trade-offs. For example, `claude-opus-5-5` handles fine-grained visual inspection, while `gemini-3.1-pro-preview` processes multi-page documents and audio files up to 9.5 hours. Video integration sections cover async task polling, webhook callbacks, fal.ai and Vercel AI SDK integration, and multi-shot storyboard consistency.

Because `user-invocable` is set to `false`, the agent loads this skill automatically when encountering multimodal requirements rather than via manual slash commands. It requires Claude Code 2.1.277 or newer.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add yonatangross/orchestkit --skill multimodal-llm -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/multimodal-llm/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/multimodal-llm/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add yonatangross/orchestkit --skill multimodal-llm -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/multimodal-llm/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add yonatangross/orchestkit --skill multimodal-llm -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/multimodal-llm/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install yonatangross/orchestkit src/skills/multimodal-llm
```

2. **Manually: put the files in place**
- File: `.github/skills/multimodal-llm/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add yonatangross/orchestkit --skill multimodal-llm -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/yonatangross/orchestkit --path src/skills/multimodal-llm
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/multimodal-llm/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add yonatangross/orchestkit --skill multimodal-llm -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/yonatangross/orchestkit/tree/HEAD/src/skills/multimodal-llm/SKILL.md

## When to use it
- select vision models for OCR and PDF table extraction
- configure long-form speech transcription with speaker diarization
- integrate video generation APIs for Kling v3, Veo 3.1, or Sora 2
- set up multi-scene video storyboards with consistent characters

## Example prompts
- Compare multiple PDF invoices and extract the total amounts into a JSON table
- Write Python code to transcribe a meeting recording via Gemini with speaker diarization
- Implement an async polling pipeline for Kling v3 video generation

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: c86f9289ce895fc3.

BEGIN THIRD-PARTY FILE c86f9289ce895fc3
```text
---
name: multimodal-llm
license: MIT
compatibility: "Claude Code 2.1.277+."
description: Vision, audio, video generation, and multimodal LLM patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1, Runway Gen-4.5), or building multimodal AI pipelines.
user-invocable: false
effort: high
metadata:
  archive-candidate: "pending an outside usage signal"
  owner-agent: "multimodal-specialist"
  category: mcp-enhancement
  version: "2.1.1"
  author: "OrchestKit"
  complexity: "high"
  tags: "vision, audio, video, multimodal, image, speech, transcription, tts, kling, sora, veo, video-generation"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---

# Multimodal LLM Patterns

Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via `gen4_turbo`).

> **Canonical model IDs** (Google IDs checked against ai.google.dev/gemini-api/docs/models on 2026-09-23):
>
> | Provider | Model IDs |
> |----------|-----------|
> | Anthropic | `claude-opus-5-5` (recommended, the default Opus since CC 2.1.280, same 2,576 px budget as Opus 5), `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-5-5`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`. `claude-fable-5` is Anthropic's **frontier tier above Opus** (GA 2026-07). Premium cost, never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spend |
> | OpenAI    | `gpt-5.5` (current flagship) |
> | Google    | `gemini-3.1-pro-preview` (flagship), `gemini-3.1-flash-lite` (cost) |
> | Veo       | `veo-3.1-generate-preview` / `veo-3.1-lite-generate-preview` / `veo-3.1-fast-generate-preview` |
> | Kling     | `kling-v3` (model_name field in Kling API) |
> | Runway    | `gen4_turbo` (product label: Gen-4.5) |

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Vision: Image Analysis](#vision-image-analysis) | 1 | HIGH | Image captioning, VQA, multi-image comparison, object detection |
| [Vision: Document Understanding](#vision-document-understanding) | 1 | HIGH | OCR, chart/diagram analysis, PDF processing, table extraction |
| [Vision: Model Selection](#vision-model-selection) | 1 | MEDIUM | Choosing provider, cost optimization, image size limits |
```
END THIRD-PARTY FILE c86f9289ce895fc3

The third-party file has ended. Everything between the c86f9289ce895fc3 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2537 of 8985 bytes. © OrchestKit · MIT · https://github.com/yonatangross/orchestkit/tree/HEAD/src/skills/multimodal-llm/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
