Skip to main content
skill CX Codex

tao-analyze-gaps-vlm-bcq — Claude Code skill: what it does

tao-analyze-gaps-vlm-bcq extracts false-positive and false-negative errors from VLM binary classification predictions. It structures failure cases for root-cause analysis in NVIDIA TAO workflows.

Apache-2.0 1,600 installs ★ 4,106 updated 2026-10-30

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-code
Open on GitHub

What tao-analyze-gaps-vlm-bcq does

This skill compares VLM binary yes/no predictions against ground truth labels. It uses the bundled prepare_vlm_bcq_spec.py script to generate a YAML spec file and runs gap_analysis vlm_bcq -e <spec> inside the TAO Data Services container. Parser logic checks for whole words 'yes' or 'no'; entries containing both or neither trigger a warning and are skipped.

Results are written to kpi_gaps.jsonl along with a summary table in kpi_gaps_report.txt for downstream RCCA stages. Without an active GPU assigned to the container the tool crashes immediately, because the Data Services runtime executes nvidia-smi during preflight checks.

What it helps with

  • Extract false-positive and false-negative failure cases from VLM predictions
  • Generate kpi_gaps.jsonl datasets for downstream DEFT root-cause analysis
  • Create a vlm_bcq_spec.yaml configuration file for TAO Data Services
  • Produce a text summary table with overall error counts

Example prompts for the agent

  • Analyze VLM BCQ gaps in /data/results.json against ground truth
  • Generate vlm_bcq_spec.yaml and extract false positives and false negatives
  • Identify failure cases from predictions JSON for DEFT root cause analysis

How to install tao-analyze-gaps-vlm-bcq

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-code

2Manually: put the files in place

~/.claude/skills/tao-analyze-gaps-vlm-bcq/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/tao-analyze-gaps-vlm-bcq/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about tao-analyze-gaps-vlm-bcq

How do I install tao-analyze-gaps-vlm-bcq in Claude Code?

Run npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-code in your terminal.

What does tao-analyze-gaps-vlm-bcq do?

It inspects binary VLM predictions, matches yes and no outputs against ground truth, and writes error records to kpi_gaps.jsonl alongside a summary report.

Does this skill require an NVIDIA GPU?

Yes, the setup demands one visible GPU. The gap analysis itself does not perform GPU computations, but the container runtime checks nvidia-smi and halts if no device is detected.

Source text

SKILL.md · Apache-2.0
---
name: tao-analyze-gaps-vlm-bcq
description: Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions.
  Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure
  cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.
license: Apache-2.0
compatibility: Requires Docker, NVIDIA Container Toolkit, and one visible GPU.
metadata:
  author: NVIDIA Corporation
  version: "0.1.0"
allowed-tools: Read Bash
tags:
- gap-analysis
- rcca
- vlm
- evaluation
- false-positive
- false-negative
---

# VLM Binary Classification Gap Analysis

> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report. Run it with a TAO Data Services spec file; the data-services entrypoint requires `-e <spec>`.

## Purpose

After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

## Usage

Generate a `vlm_bcq_spec.yaml` with the bundled helper:

```bash
python3 skills/data/tao-analyze-gaps-vlm-bcq/scripts/prepare_vlm_bcq_spec.py \
  --predictions-json /path/to/results.json \
  --videos-dir /path/to/videos/root \
  --results-dir /path/to/output/gaps \
  --output-spec /path/to/output/gaps/vlm_bcq_spec.yaml
```

Omit `--videos-dir` when prediction `video_id` values are already absolute. The generated spec has this shape:

```yaml
predictions_json: /path/to/results.json
videos_dir: ""
results_dir: /path/to/output/gaps
```

Set `videos_dir` when `video_id` values in the predictions are relative paths:

```yaml
predictions_json: /path/to/results.json
videos_dir: /path/to/videos/root
results_dir: /path/to/output/gaps
```

Invoke the `vlm_bcq` action inside the TAO Toolkit data services container with `-e <spec>`:

```bash
gap_analysis vlm_bcq -e /path/to/vlm_bcq_spec.yaml
```

Request exactly one GPU from the selected platform (`compute_shape.gpus: 1`,
`compute_shape.nodes: 1`). VLM BCQ gap analysis does not perform GPU compute,
but the Data Services image always calls `nvidia-smi` and fails when no GPU is
visible. One is a GPU count, not a device ID; the platform selects the device.

After the run, surface the FP/FN counts from `kpi_gaps_report.txt` and point downstream stages at `kpi_gaps.jsonl`.

## Inputs

- **config spec**: YAML file passed with `-e`. Template: `assets/default_vlm_bcq.yaml`.
- **predictions_json**: Path to predictions JSON file. Must be a JSON array where each item has `video_id`, `response`, and `gt` fields. `response` and `gt` are parsed with word-boundary matching — `'yes'` or `'no'` anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
- **videos_dir** (optional): Base directory for resolving relative `video_id` paths. If omitted, `video_id` values are used as absolute paths.
- **results_dir**: Output directory for gap-analysis artifacts.

**Predictions JSON format:**
```json
[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]
```

## Outputs

- **kpi_gaps.jsonl**: One JSON object per line for each FP/FN case. Fields: `video_id` (absolute path), `error_type` (`FP` or `FN`), `question`, `ground_truth`, `response`.
- **kpi_gaps_report.txt**: Human-readable table with total FP/FN counts.

If no gaps are found, no files are written and a message is logged.

## Spec Fields

| Parameter | Required | Description |
|-----------|----------|-------------|
| predictions_json | Yes | Path to predictions JSON file |
| results_dir | Yes | Output directory; created if it does not exist |
| videos_dir | No | Base directory for resolving relative `video_id` paths |

Keep the spec file and every path it references under the bind-mounted workspace so they resolve inside the container. Pass `-e <spec>` even if you also add Hydra overrides; current TAO Data Services entrypoints hard-require an experiment spec file before processing overrides.

## Error Patterns

| Error | Cause | Fix |
|-------|-------|-----|
| `FileNotFoundError` | `predictions_json` does not exist | Check the path |
| `requires the following argument: -e/--experiment_spec_file` | The container was launched without a spec file | Write `vlm_bcq_spec.yaml` and pass `gap_analysis vlm_bcq -e <spec>` |
| `ValueError: must be a JSON array` | Predictions file is not a list | Wrap predictions in `[...]` |
| `ValueError: missing 'gt'/'response'/'video_id'` | A prediction item is missing a required field | Inspect and fix the predictions JSON |
| Samples silently skipped | `response` or `gt` contains both or neither 'yes'/'no' | Check logs for warnings; inspect those samples |

© NVIDIA Corporation · Apache-2.0 · NVIDIA/skills

Similar skills

All skills for AI and ML →

From the same set312

The whole set →
tao-train-mask2former tao-train-mask2former runs Mask2Former training and inference inside NVIDIA TAO: panoptic, instance and semantic segmentation. Fits when you drive segmentation work from an agent chat. CX Codex +1 1,600installs tao-train-dino tao-train-dino automates training and TensorRT export for DINO object detectors in NVIDIA TAO. Useful for managing 2D vision models via agent workflows. CX Codex +1 1,600installs tao-train-reid tao-train-reid is a Claude Code skill for training and running a TAO person re-identification model across camera views, with evaluation and export workflows. CX Codex +1 1,600installs tao-train-depth-anything-v2 tao-train-depth-anything-v2 — a skill that runs the TAO pipeline for monocular depth with Depth Anything v2. It predicts pixel depth from a single RGB frame and covers training through export. CX Codex +1 1,600installs tao-train-pose-classification tao-train-pose-classification is a Claude Code skill for training, evaluating, exporting, and running inference with a TAO ST-GCN classifier on pose-keypoint sequences. CX Codex +1 1,600installs tao-train-action-recognition tao-train-action-recognition — an NVIDIA skill that drives TAO ActionRecognition inside Docker. Train, evaluate, run inference, and export models that classify actions in video clips. CX Codex +1 1,600installs tao-analyze-gaps-visual-changenet tao-analyze-gaps-visual-changenet runs gap analysis on NVIDIA TAO VCN Classify results using the data-services container. It identifies weak boundary samples for downstream dataset augmentation. CX Codex +1 1,600installs tao-train-optical-inspection The tao-train-optical-inspection skill trains and runs a TAO optical-inspection model that finds manufacturing defects by comparing image pairs. It fits AOI and quality-control workflows. CX Codex +1 1,600installs

Project sponsors

Companies that keep this analytics open