# tao-run-on-kubernetes

Turns a TAO spec-bundle into a Kubernetes Job and submits it with plain kubectl. For training runs and inference on EKS, GKE, AKS or an on-prem GPU cluster.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 1600 installs · ★ 3735
- Author: NVIDIA Corporation
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-run-on-kubernetes/SKILL.md
- Updated: 2026-10-26
- HTML: https://affpapa.org/en/nevibecode/skills/tao-run-on-kubernetes

## What it does

The skill consumes a spec-bundle and produces a Kubernetes Job from it. There is no `nvidia-tao-sdk` here and nothing imports `tao_sdk`; the entire path is `kubectl apply`, so it works against any cluster your kubeconfig can reach, or through a service account when the client already runs inside a pod. One pod is the default. Set `num_nodes > 1` and you get distributed training across nodes, built on an Indexed Job plus a headless Service.

Checks run before anything is applied. On self-managed GPU workers the host script `setup-nvidia-gpu-host.sh --backend kubernetes --check-only` is mandatory and expects driver branch 580, CUDA Toolkit 13.0 and NVIDIA Container Toolkit 1.19.0. Managed clusters skip that step via `TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1` when the provider or GPU Operator owns the driver lifecycle. Cluster reachability and allocatable `nvidia.com/gpu` are checked next, and the `submit` verb makes the GPU count a hard gate: there is no gang scheduling, so an oversized Job would sit `Pending` indefinitely.

Dataset I/O comes in two tiers. Tier A mounts a bound PVC or NFS and fetches nothing, which is the air-gap answer. Tier C fills a shared `emptyDir` from S3 in an initContainer, then uploads results before TTL. Pick C with care: the pod holds `nvidia.com/gpu` for its whole life, initContainers included, so a big download or a first multi-GB image pull burns GPU time you pay for.

## Requirements
- Works with: Kubernetes

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-run-on-kubernetes -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/tao-run-on-kubernetes/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/tao-run-on-kubernetes/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-run-on-kubernetes -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/tao-run-on-kubernetes/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-run-on-kubernetes -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/tao-run-on-kubernetes/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install NVIDIA/skills tao-run-on-kubernetes
```

2. **Manually: put the files in place**
- File: `.github/skills/tao-run-on-kubernetes/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-run-on-kubernetes -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/NVIDIA/skills --path skills/tao-run-on-kubernetes
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/tao-run-on-kubernetes/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add NVIDIA/skills --skill tao-run-on-kubernetes -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-run-on-kubernetes/SKILL.md

## When to use it
- submit a TAO job to a cluster from kubeconfig
- run distributed training across several nodes
- verify GPU node readiness before scheduling
- pull a dataset from S3 into an ephemeral volume
- stop an oversized Job from hanging in Pending

## Example prompts
- Submit this TAO bundle to namespace `ml`, the data sits on a bound PVC
- Run the training across two nodes with tao-run-on-kubernetes
- Check whether allocatable `nvidia.com/gpu` covers this bundle

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: de57ad62b5617332.

BEGIN THIRD-PARTY FILE de57ad62b5617332
````text
---
name: tao-run-on-kubernetes
description: Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod
  for one node, Indexed Jobs for multi-node distributed training. Use when running on EKS / GKE / AKS / on-prem clusters
  with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.
license: Apache-2.0
compatibility: Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; a `kubectl` client authenticated to the cluster; and the NVIDIA GPU Operator or device plugin. No nvidia-tao-sdk required — jobs are submitted with plain `kubectl`.
metadata:
  author: NVIDIA Corporation
  version: "0.1.0"
allowed-tools: Read Bash
tags:
- kubernetes
- k8s
- gpu
- compute
- container
---

# Kubernetes

> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).

Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod).

Single-pod by default; opt into multi-node distributed training via `num_nodes > 1` (uses Indexed Job + headless Service, see [Multi-node training](#multi-node-training-distributed) below).

## Preflight

Three checks: GPU host runtime ready, cluster reachable via `kubectl`, GPU
Operator/device plugin present.

```bash
# 0. GPU node host runtime.
# Run this on each self-managed GPU worker node or in the node image build.
# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose
# driver/toolkit lifecycle is owned by the cloud provider or GPU Operator policy.
if [ "${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}" != "1" ]; then
  TAO_SKILL_BANK_ROOT="${TAO_SKILL_BANK_ROOT:-$PWD}"
  SETUP_SCRIPT="${TAO_SKILL_BANK_ROOT}/skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh"

  bash "$SETUP_SCRIPT" --backend kubernetes --check-only || {
    echo "MISSING: TAO Kubernetes GPU node runtime is not ready."
    echo "For self-managed GPU nodes, run after user approval:"
````
END THIRD-PARTY FILE de57ad62b5617332

The third-party file has ended. Everything between the de57ad62b5617332 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2222 of 19805 bytes. © NVIDIA Corporation · Apache-2.0 · https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-run-on-kubernetes/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md