# tao-finetune-clip

A skill that walks an agent through CLIP training, zero-shot classification, embedding extraction and ONNX or TensorRT export in NVIDIA TAO.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1600 installs · ★ 3735
- Author: NVIDIA Corporation
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-finetune-clip/SKILL.md
- Updated: 2026-10-26
- HTML: https://affpapa.org/en/nevibecode/skills/tao-finetune-clip

## What it does

Five CLIP actions live in this skill: `train`, `evaluate`, `inference`, `export` and `gen_trt_engine`. The agent reads the model's `references/` folder for the image, command and spec template that fit the chosen action, and you hand over dataset paths through `spec_overrides`. Fine-tuning reshapes the shared image-text embedding space toward your own image-caption pairs.

Validation on paired captions reports `val/t2i_mAP`; for PAS metadata matching the metric is `val/pas/overall_mAP`. AutoML is on by default, so training goes through `tao-skill-bank:tao-run-automl` unless you ask for plain training without HPO or the packaged train schema is missing — then the skill runs the model directly.

Docker plus nvidia-container-toolkit are prerequisites, and a container will not start without them. One trap worth knowing: `inference.checkpoint: null` does not fall back to pretrained weights, since `clip inference` calls the loader with `None` and fails before extracting embeddings. On PyTorch actions that load a checkpoint from a trusted job, set `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` so PyTorch 2.6 can read the Lightning metadata.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-finetune-clip -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/tao-finetune-clip/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/tao-finetune-clip/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-finetune-clip -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/tao-finetune-clip/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-finetune-clip -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/tao-finetune-clip/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install NVIDIA/skills tao-finetune-clip
```

2. **Manually: put the files in place**
- File: `.github/skills/tao-finetune-clip/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-finetune-clip -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/NVIDIA/skills --path skills/tao-finetune-clip
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/tao-finetune-clip/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-finetune-clip -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-finetune-clip/SKILL.md

## When to use it
- fine-tune CLIP on custom image-caption pairs
- run zero-shot classification without retraining
- extract image embeddings for retrieval
- build a TensorRT engine from an exported ONNX file
- score retrieval quality with `val/t2i_mAP`

## Example prompts
- Train CLIP on my image and caption folders, paths are in `spec_overrides`
- Export the fine-tuned CLIP checkpoint to ONNX, then generate the TensorRT engine
- Run zero-shot classification on my prompt set and show the metrics

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 3d208f3f714797f5.

BEGIN THIRD-PARTY FILE 3d208f3f714797f5
```text
---
name: tao-finetune-clip
description: CLIP vision-language model for image-text retrieval, zero-shot classification, embedding extraction, ONNX
  export, and TensorRT deployment. Use when fine-tuning or training CLIP, running zero-shot classification, computing image
  embeddings, or deploying CLIP to ONNX/TensorRT. This is a single-action model skill; do not use it for an iterative
  weak-attribute improvement loop that keeps retraining and evaluating until progress stops, which belongs to tao-run-deft-pas.
license: Apache-2.0
compatibility: Requires docker + nvidia-container-toolkit.
metadata:
  author: NVIDIA Corporation
  version: "0.1.0"
allowed-tools: Read Bash
tags:
- vision-language
- classification
- embedding
- zero-shot
- deployment
---

# CLIP

> **Standalone install?** Read [CLIP standalone setup](references/standalone-setup.md) and verify the bank checkout before running `tao-setup` or launching a CLIP action.

Contrastive Language-Image Pre-training model for zero-shot and fine-tuned image classification, image-text retrieval, and embedding extraction. Fine-tuning adapts CLIP's shared image-text embedding space to domain-specific image-caption data.

No default NGC pretrained checkpoint is required for spec construction, but unset checkpoint behavior is action-specific. In the validation-fixes PyTorch image, `export.checkpoint: null` exports the selected CLIP architecture and may initialize weights when pretrained weights are unavailable. Do not assume `inference.checkpoint: null` loads pretrained weights: `clip inference` currently calls the checkpoint loader with `None` and fails before embedding extraction. For PyTorch inference, checkpoint-backed evaluation/export, resume, and retrain flows, resolve and pass an exact checkpoint from the parent train output. For trusted TAO checkpoints produced by the current run or a known parent job, set `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` on checkpoint-dependent PyTorch actions so PyTorch 2.6 can load the Lightning checkpoint metadata; do not set this for untrusted checkpoints.

Supported actions: `train`, `evaluate`, `inference`, `export`, `gen_trt_engine`.

## Train Action Policy
```
END THIRD-PARTY FILE 3d208f3f714797f5

The third-party file has ended. Everything between the 3d208f3f714797f5 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2178 of 19901 bytes. © NVIDIA Corporation · Apache-2.0 · https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-finetune-clip/SKILL.md

## Similar
- [clip](https://affpapa.org/en/nevibecode/skills/clip.md) — skill · 957 installs
- [llava](https://affpapa.org/en/nevibecode/skills/llava.md) — skill · 940 installs
- [opus-clip](https://affpapa.org/en/nevibecode/skills/opus-clip.md) — skill · 593 installs
- [vision-sft](https://affpapa.org/en/nevibecode/skills/vision-sft.md) — skill · 2600 installs
- [scikit-learn](https://affpapa.org/en/nevibecode/skills/k-dense-ai-scikit-learn.md) — skill · 2200 installs

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md