# tao-train-nvpanoptix3d

Automates training, evaluation, inference, and export for the NVPanoptix3D model using posed RGB images. Useful for 3D panoptic segmentation on Matterport and Front3D datasets.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1600 installs · ★ 3735
- Author: NVIDIA Corporation
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-train-nvpanoptix3d/SKILL.md
- Updated: 2026-10-26
- HTML: https://affpapa.org/en/nevibecode/skills/tao-train-nvpanoptix3d

## What it does

This skill handles the NVPanoptix3D workflow built on a VGGT backbone with a Mask2Former-style head and 3D frustum reconstruction. The model performs panoptic 3D scene reconstruction and occupancy completion from posed RGB images. It generates semantic, instance, and panoptic masks. Initialization relies on two-stage weights defined in `train.checkpoint_2d` and `train.checkpoint_3d`. Training, evaluation, inference, and export execute inside the TAO PyTorch container `nvcr.io/nvidia/tao/tao-toolkit:7.3.0-pyt`. You do not need the TAO SDK or Python installed directly on the host.

Without Docker and `nvidia-container-toolkit`, the container fails to start. Standalone sessions require running the `tao-setup` skill first to verify credentials and host preflight settings. AutoML is enabled by default. Hyperparameter tuning uses `PRQ` as the primary optimization metric with `direction=maximize`. Status logs also report `RSQ` and `RRQ`, but `train_loss` is not exposed reliably for model selection.

## Requirements
- Works with: Docker

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-train-nvpanoptix3d -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/tao-train-nvpanoptix3d/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/tao-train-nvpanoptix3d/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-train-nvpanoptix3d -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/tao-train-nvpanoptix3d/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-train-nvpanoptix3d -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/tao-train-nvpanoptix3d/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install NVIDIA/skills tao-train-nvpanoptix3d
```

2. **Manually: put the files in place**
- File: `.github/skills/tao-train-nvpanoptix3d/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-train-nvpanoptix3d -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/NVIDIA/skills --path skills/tao-train-nvpanoptix3d
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/tao-train-nvpanoptix3d/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-train-nvpanoptix3d -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-train-nvpanoptix3d/SKILL.md

## When to use it
- train NVPanoptix3D models on Front3D or Matterport datasets
- reconstruct 3D scenes with occupancy completion from posed RGB frames
- evaluate panoptic segmentation accuracy using PRQ, RSQ, and RRQ metrics
- export trained checkpoints via /specs/export.yaml specifications
- configure AutoML hyperparameter searches maximizing the PRQ metric

## Example prompts
- Train NVPanoptix3D using `/specs/train.yaml` with AutoML disabled
- Evaluate the NVPanoptix3D checkpoint on the validation split with `/specs/evaluate.yaml`
- Export the trained model according to `/specs/export.yaml`

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: adba87473f70caf2.

BEGIN THIRD-PARTY FILE adba87473f70caf2
````text
---
name: tao-train-nvpanoptix3d
description: NVPanoptix3D for panoptic 3D scene reconstruction from posed RGB images. Produces 3D panoptic segmentation
  (semantic, instance, and panoptic masks) with occupancy completion. Built on a VGGT backbone with a Mask2Former-style head
  and 3D frustum reconstruction. Use when training, evaluating, exporting, or running inference for a TAO NVPanoptix3D model.
  Trigger phrases include "train NVPanoptix3D", "panoptic 3D reconstruction", "3D scene segmentation", "occupancy completion".
license: Apache-2.0
compatibility: Requires docker + nvidia-container-toolkit.
metadata:
  version: "0.1.0"
  author: NVIDIA Corporation
allowed-tools: Read Bash
tags:
- panoptic
- 3d
- reconstruction
---

# NVPanoptix3D

> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).

NVPanoptix3D for panoptic 3D scene reconstruction from posed RGB images. Produces 3D panoptic segmentation (semantic, instance, and panoptic masks) with occupancy completion. Built on VGGT backbone with Mask2Former-style head and 3D frustum reconstruction.

Uses 2D and 3D stage checkpoints. Set train.checkpoint_2d and train.checkpoint_3d for staged initialization.

## Quick Start (docker run)

Docker-native launch — no TAO SDK and no Python on the host. Use the local
Docker/platform skill instead when it gives a stricter environment-specific
command (non-root UID mapping, cache redirects, remote daemons).

```bash
TAO_PYT_IMAGE_DEFAULT=nvcr.io/nvidia/tao/tao-toolkit:7.3.0-pyt  # versions-key: images.tao_toolkit.pyt
TAO_PYT_IMAGE="${TAO_PYT_IMAGE:-$TAO_PYT_IMAGE_DEFAULT}"
RUN_ROOT="${RUN_ROOT:-$PWD}"
DOCKER_COMMON=(
  --rm --gpus all --shm-size=8g
  --shm-size=8g
  --ulimit memlock=-1
  --ulimit stack=67108864
  -v "$RUN_ROOT/data:/data:ro"
  -v "$RUN_ROOT/specs:/specs:ro"
  -v "$RUN_ROOT/results:/results"
)
```

Train:

```bash
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
  python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d train -e /specs/train.yaml
```

Evaluate:

```bash
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
  python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d evaluate -e /specs/evaluate.yaml
```

Inference:

```bash
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
  python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d inference -e /specs/inference.yaml
```

Export:
````
END THIRD-PARTY FILE adba87473f70caf2

The third-party file has ended. Everything between the adba87473f70caf2 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2504 of 17033 bytes. © NVIDIA Corporation · Apache-2.0 · https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-train-nvpanoptix3d/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md