tao-analyze-gaps-vlm-bcq — Claude Code skill: what it does
tao-analyze-gaps-vlm-bcq extracts false-positive and false-negative errors from VLM binary classification predictions. It structures failure cases for root-cause analysis in NVIDIA TAO workflows.
Apache-2.0 1,600 installs ★ 4,106 updated 2026-10-30
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-codeWhat tao-analyze-gaps-vlm-bcq does
This skill compares VLM binary yes/no predictions against ground truth labels. It uses the bundled prepare_vlm_bcq_spec.py script to generate a YAML spec file and runs gap_analysis vlm_bcq -e <spec> inside the TAO Data Services container. Parser logic checks for whole words 'yes' or 'no'; entries containing both or neither trigger a warning and are skipped.
Results are written to kpi_gaps.jsonl along with a summary table in kpi_gaps_report.txt for downstream RCCA stages. Without an active GPU assigned to the container the tool crashes immediately, because the Data Services runtime executes nvidia-smi during preflight checks.
What it helps with
- Extract false-positive and false-negative failure cases from VLM predictions
- Generate
kpi_gaps.jsonldatasets for downstream DEFT root-cause analysis - Create a
vlm_bcq_spec.yamlconfiguration file for TAO Data Services - Produce a text summary table with overall error counts
Example prompts for the agent
- Analyze VLM BCQ gaps in /data/results.json against ground truth
- Generate vlm_bcq_spec.yaml and extract false positives and false negatives
- Identify failure cases from predictions JSON for DEFT root cause analysis
How to install tao-analyze-gaps-vlm-bcq
1Skills CLI (Vercel)
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-code2Manually: put the files in place
~/.claude/skills/tao-analyze-gaps-vlm-bcq/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/tao-analyze-gaps-vlm-bcq/
1Skills CLI (Vercel)
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a codex2Manually: put the files in place
.agents/skills/tao-analyze-gaps-vlm-bcq/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a cursor2Manually: put the files in place
.cursor/skills/tao-analyze-gaps-vlm-bcq/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install NVIDIA/skills skills/tao-analyze-gaps-vlm-bcq2Manually: put the files in place
.github/skills/tao-analyze-gaps-vlm-bcq/SKILL.md1Skills CLI (Vercel)
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a gemini-cli2Platform CLI
gemini skills install https://github.com/NVIDIA/skills --path skills/tao-analyze-gaps-vlm-bcqManually: put the files in place
.devin/skills/tao-analyze-gaps-vlm-bcq/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about tao-analyze-gaps-vlm-bcq
How do I install tao-analyze-gaps-vlm-bcq in Claude Code?
Run npx skills add NVIDIA/skills --skill tao-analyze-gaps-vlm-bcq -a claude-code in your terminal.
What does tao-analyze-gaps-vlm-bcq do?
It inspects binary VLM predictions, matches yes and no outputs against ground truth, and writes error records to kpi_gaps.jsonl alongside a summary report.
Does this skill require an NVIDIA GPU?
Yes, the setup demands one visible GPU. The gap analysis itself does not perform GPU computations, but the container runtime checks nvidia-smi and halts if no device is detected.
Source text
SKILL.md · Apache-2.0
---
name: tao-analyze-gaps-vlm-bcq
description: Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions.
Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure
cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.
license: Apache-2.0
compatibility: Requires Docker, NVIDIA Container Toolkit, and one visible GPU.
metadata:
author: NVIDIA Corporation
version: "0.1.0"
allowed-tools: Read Bash
tags:
- gap-analysis
- rcca
- vlm
- evaluation
- false-positive
- false-negative
---
# VLM Binary Classification Gap Analysis
> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).
Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report. Run it with a TAO Data Services spec file; the data-services entrypoint requires `-e <spec>`.
## Purpose
After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.
## Usage
Generate a `vlm_bcq_spec.yaml` with the bundled helper:
```bash
python3 skills/data/tao-analyze-gaps-vlm-bcq/scripts/prepare_vlm_bcq_spec.py \
--predictions-json /path/to/results.json \
--videos-dir /path/to/videos/root \
--results-dir /path/to/output/gaps \
--output-spec /path/to/output/gaps/vlm_bcq_spec.yaml
```
Omit `--videos-dir` when prediction `video_id` values are already absolute. The generated spec has this shape:
```yaml
predictions_json: /path/to/results.json
videos_dir: ""
results_dir: /path/to/output/gaps
```
Set `videos_dir` when `video_id` values in the predictions are relative paths:
```yaml
predictions_json: /path/to/results.json
videos_dir: /path/to/videos/root
results_dir: /path/to/output/gaps
```
Invoke the `vlm_bcq` action inside the TAO Toolkit data services container with `-e <spec>`:
```bash
gap_analysis vlm_bcq -e /path/to/vlm_bcq_spec.yaml
```
Request exactly one GPU from the selected platform (`compute_shape.gpus: 1`,
`compute_shape.nodes: 1`). VLM BCQ gap analysis does not perform GPU compute,
but the Data Services image always calls `nvidia-smi` and fails when no GPU is
visible. One is a GPU count, not a device ID; the platform selects the device.
After the run, surface the FP/FN counts from `kpi_gaps_report.txt` and point downstream stages at `kpi_gaps.jsonl`.
## Inputs
- **config spec**: YAML file passed with `-e`. Template: `assets/default_vlm_bcq.yaml`.
- **predictions_json**: Path to predictions JSON file. Must be a JSON array where each item has `video_id`, `response`, and `gt` fields. `response` and `gt` are parsed with word-boundary matching — `'yes'` or `'no'` anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
- **videos_dir** (optional): Base directory for resolving relative `video_id` paths. If omitted, `video_id` values are used as absolute paths.
- **results_dir**: Output directory for gap-analysis artifacts.
**Predictions JSON format:**
```json
[
{
"video_id": "/path/to/video.mp4",
"response": "Yes, there is a collision.",
"gt": "B. No",
"question": "Is there a collision?"
}
]
```
## Outputs
- **kpi_gaps.jsonl**: One JSON object per line for each FP/FN case. Fields: `video_id` (absolute path), `error_type` (`FP` or `FN`), `question`, `ground_truth`, `response`.
- **kpi_gaps_report.txt**: Human-readable table with total FP/FN counts.
If no gaps are found, no files are written and a message is logged.
## Spec Fields
| Parameter | Required | Description |
|-----------|----------|-------------|
| predictions_json | Yes | Path to predictions JSON file |
| results_dir | Yes | Output directory; created if it does not exist |
| videos_dir | No | Base directory for resolving relative `video_id` paths |
Keep the spec file and every path it references under the bind-mounted workspace so they resolve inside the container. Pass `-e <spec>` even if you also add Hydra overrides; current TAO Data Services entrypoints hard-require an experiment spec file before processing overrides.
## Error Patterns
| Error | Cause | Fix |
|-------|-------|-----|
| `FileNotFoundError` | `predictions_json` does not exist | Check the path |
| `requires the following argument: -e/--experiment_spec_file` | The container was launched without a spec file | Write `vlm_bcq_spec.yaml` and pass `gap_analysis vlm_bcq -e <spec>` |
| `ValueError: must be a JSON array` | Predictions file is not a list | Wrap predictions in `[...]` |
| `ValueError: missing 'gt'/'response'/'video_id'` | A prediction item is missing a required field | Inspect and fix the predictions JSON |
| Samples silently skipped | `response` or `gt` contains both or neither 'yes'/'no' | Check logs for warnings; inspect those samples |
© NVIDIA Corporation · Apache-2.0 · NVIDIA/skills