gke-ai-troubleshooting-tpu-partition-overlap: Claude Code skill

---
name: gke-ai-troubleshooting-tpu-partition-overlap
description: >-
  Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed)
  caused by partition overlap conflicts where requested TPU partitions or their
  nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU
  slice creation failures with SliceCreationFailed partition or node conflicts.
  Don't use for general slice lifecycle states or slice provisioning failures
  without node overlap (use
  gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or
  unresponsive TPU nodes.
metadata:
  version: "1.0.1"
  category: Containers
---

# Troubleshoot TPU slice creation failures caused by partition overlap

A dynamic slice fails to form with the status reason `SliceCreationFailed` when,
among other documented causes, any of its nodes are already used by another
slice. This skill confirms the failure, rules out the other documented causes,
finds the slice that uses the same nodes, and routes the fix through GKE.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)` to `User-Agent` headers.

## Prerequisites

- **Tools**:
  - For installing and initializing `gcloud` (which covers authentication and
    project selection), follow
    [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).
  - For installing `kubectl` and required plugins, follow
    [Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
    and
    [Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).
- **Cloud Billing & Project Configuration**: Ensure the project has an active
  Cloud Billing account. Follow
  [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
  to initialize `gcloud` and enable the Google Kubernetes Engine API
  (`container.googleapis.com`), ensure the Cloud Monitoring API
  (`monitoring.googleapis.com`) is enabled, and follow
  [Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
  to configure `kubectl` cluster credentials.
- **Required IAM Roles**:
  - Kubernetes Engine Viewer (`roles/container.viewer`)
  - Monitoring Viewer (`roles/monitoring.viewer`)
- **Documentation**:
  - [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
    (Sections: "Before you begin", "Target your node pool to belong to a block
    or a sub-block", "Verify the status of the nodes and the partitions",
    "Create a Slice custom resource", "Monitor the status of the Slice custom
    resource", "Delete the slice")
  - [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
    (Sections: "Install kubectl", "Install required plugins", "Store cluster
    information for kubectl")
  - [Slice](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)
    (Sections: "Slice status", "Slice specification")
  - [Schedule dynamic slices with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)
    (Sections: "Target your node pool to belong to a block or a sub-block",
    "Create a dynamic slice with Kueue and TAS", "Define the partition health
    selection", "Optimize workload scheduling with slice reuse", "Enable Slice
    reuse", "Enforce slice deletion per workload", "Monitor the status of the
    slice", "Monitor the metrics of the slice", "Clean up")
  - [GKE system metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt)
    (Sections: "Kubernetes metrics")
  - [Monitored resource types](https://docs.cloud.google.com/monitoring/api/resources.md.txt)
    (Sections: "k8s_node_pool")
  - [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)
    (Sections: "Dynamic slicing configurations")
  - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
    (Sections: "Before you contact support", "Recommendations for efficient
    support")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

## Diagnostic workflow

### Step 0: Collect context `[Low Risk]`

Collect the target parameters:

- `{project_id}`: Google Cloud project ID
- `{cluster_name}`: GKE cluster name
- `{location}`: Cluster region or zone
- `{slice_name}`: Name of the Slice custom resource that failed to form
- `{node_pool_name}`: Name of the TPU node pool (or node pools) backing the
  partitions in `{slice_name}`, if known. If the user doesn't know
  `{node_pool_name}`, after Step 1 run the Cloud Monitoring query in Step 3
  for the partition IDs in `{partition_ids}` and read the `node_pool_name`
  label on the
  [k8s_node_pool](https://docs.cloud.google.com/monitoring/api/resources.md.txt)
  monitored resource.
- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known. If the user
  doesn't know `{issue_time}`, use the `Last Transition Time`
  (`lastTransitionTime`, "A timestamp of the most recent status change" in
  [Slice status](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt))
  of the `Ready` condition from Step 1. By default, query Cloud Monitoring over
  a 60-minute window around `{issue_time}`.

---

### Step 1: Confirm the `SliceCreationFailed` status `[Low Risk]`

To describe `{slice_name}`, follow the instructions in the section
[Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).

- In `Status` > `Conditions`, check that the `Reason` of the `Ready` condition
  is `SliceCreationFailed`, and record its `Last Transition Time` (to use as
  `{issue_time}` if needed).
- Record the IDs in `Spec` > `Partition Ids` as `{partition_ids}`, the
  `Topology`, and any `Labels` and `Annotations` on `{slice_name}`.
- If the `Ready` reason isn't `SliceCreationFailed`, this skill doesn't apply.
  See
  [Slice status](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)
  for what each reason means, and hand off to
  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.

---

### Step 2: Rule out the other documented causes `[Low Risk]`

Per
[Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
slice creation also fails with `SliceCreationFailed` when:

- The selected nodes don't exist.
- The topology doesn't match the number of partitions. A dynamic sub-slice lists
  exactly one partition ID. A dynamic super-slice lists `(A × B × C) / 64`
  partition IDs.

If either of those causes applies, hand off to
`gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.

Otherwise, follow
[Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
on `{node_pool_name}`:

- Verify that the nodes exist and carry each ID in `{partition_ids}`.
- On the nodes that carry `{partition_ids}`, record **every** partition ID of
  **every** shape (`4x4x4`, `2x4x4`, `2x2x4`, `2x2x2`, and `2x2x1`) from the
  `cloud.google.com/gke-tpu-partition-.*-id` labels. Per
  [Dynamic slicing configurations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt),
  dynamic sub-slicing partitions a `4x4x4` sub-block into smaller topologies on
  the same physical node pool, so partitions of different shapes share the same
  nodes.
- Read each partition's `cloud.google.com/gke-tpu-partition-.*-state` label. Per
  [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
  and
  [Define the partition health selection](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),
  count `HEALTHY` partitions and `DEGRADED` `4x4x4` partitions ("still usable
  for dynamic slice formation"; "Smaller topologies don't have a degraded
  state") as available. Partitions in `UNHEALTHY` ("can't be used to form a
  slice"), `UNSET`, or `INCOMPLETE` aren't available. If a requested partition
  is in `UNHEALTHY`, `UNSET`, or `INCOMPLETE`, suggest an available partition
  whose nodes no other slice uses (Step 4, item 2), hand off to
  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or escalate to Cloud
  Customer Care (Step 4, item 4).

---

### Step 3: Find the slice that uses the same nodes `[Low Risk]`

Per
[Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
slice creation fails with `SliceCreationFailed` when any nodes on the custom
resource are used by another slice. Because partitions of different shapes on
the same sub-block share nodes (see
[Dynamic slicing configurations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)),
another slice conflicts with `{slice_name}` if it lists **any** of the partition
IDs (of any shape) recorded on `{slice_name}`'s nodes in Step 2.

1. List the `Slice` custom resources in the cluster (as shown in
   [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt))
   and describe each one following
   [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
   to read `Spec` > `Partition Ids`, `Metadata` > `Labels`, and `Status` >
   `Conditions` (`Ready` reason):
   - Treat any slice other than `{slice_name}` whose `Spec` > `Partition Ids`
     includes **any** partition ID recorded on `{slice_name}`'s nodes in Step 2
     (whether the exact ID in `{partition_ids}` or another shape's partition ID
     on the same nodes) as conflicting. Record its name, its `Ready` reason
     (such as `ACTIVE`), and whether Kueue manages it (Step 4, item 1).
2. To check partition-to-slice associations in Cloud Monitoring (including
   around `{issue_time}`), query the `kubernetes.io/accelerator/partition/state`
   metric listed in
   [Monitor the metrics of the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt).
   Per
   [Kubernetes metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt),
   this metric "is 1 if the partition state is detected, 0 otherwise", and its
   labels include `partition_id`, `slice_id` ("ID of the associated slice"),
   and `state` ("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the
   query for each `{partition_id}` recorded on `{slice_name}`'s nodes in Step 2:

```promql
kubernetes_io:accelerator_partition_state{
  monitored_resource="k8s_node_pool",
  cluster_name="{cluster_name}",
  partition_id="{partition_id}"
}
```

   - Read only time series whose value is `1` (where `state="HEALTHY"` means the
     partition is healthy and functional, per
     [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)).
   - Because
     [Kubernetes metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt)
     defines `slice_id` only as "ID of the associated slice", always confirm the
     conflicting slice from the `Slice` custom resources in the cluster rather
     than from `slice_id` alone: list the `Slice` custom resources as shown in
     [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
     describe each one following
     [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
     and check which slice's `Spec` > `Partition Ids` includes the partition ID
     (or any other partition ID of any shape on the same nodes from Step 2).
3. **Next steps based on Step 3 results**:
   - **Another slice uses the nodes now**: Proceed to Step 4.
   - **No other slice uses the nodes now, but the metric (value `1`) showed an
     associated `slice_id` around `{issue_time}`**: The conflict may already be
     over. Proceed to Step 4, item 1 (to re-create or retry `{slice_name}`) and
     then Step 4, item 4.
   - **Neither the current `Slice` custom resources nor the metric around
     `{issue_time}` shows another slice**: Node overlap isn't confirmed. Hand
     off to `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` or escalate
     to Cloud Customer Care (Step 4, item 4).

---

### Step 4: Resolution `[High Risk]`

> **Guardrails**:
> - **Never** delete or change the other slice, or stop its workload, to free
>   the nodes. Deleting a `Slice` custom resource starts dismantling that slice
>   (`DEACTIVATING`, see
>   [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)).
>   The owner of that slice decides when to release it.
> - **Never** manually modify or delete `Slice` custom resources created by
>   Kueue. Per
>   [Target your node pool to belong to a block or a sub-block](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
>   "Avoid manually modifying Slice custom resources created by Kueue."
> - **Never** change GKE-managed instance groups or VMs directly through Compute
>   Engine, and never delete node pools to free a partition. Manage node pools
>   and slices through GKE.

First, check whether Kueue manages `{slice_name}` or the conflicting slice:

- Per
  [Monitor the status of the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),
  when a slice is created or actively used by a workload, the slice controller
  attaches the `kueue.gke.io/tpu-slice-used-by-namespace` and
  `kueue.gke.io/tpu-slice-used-by-workload` labels.
- Per
  [Optimize workload scheduling with slice reuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)
  and
  [Enable Slice reuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),
  when slice reuse is enabled, "Kueue marks the slice as idle at the end of a
  job instead of deleting it", and "Reusable slices are named dynamically with
  an `s-<topology>` prefix, decoupling them from specific workload names."

Then apply the matching option:

1. **Wait for the conflicting slice to release the nodes and retry**:
   - **If Kueue manages the conflicting slice** (it has the `kueue.gke.io/...`
     labels or an `s-<topology>` name prefix): Do not suggest deleting the
     `Slice` custom resource directly. Point the workload owner to
     [Clean up](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)
     (deleting the `JobSet` triggers Kueue to delete the associated `Slice`
     custom resources) and, if slice reuse keeps an idle `s-<topology>` slice
     allocated after a job ends, to
     [Enforce slice deletion per workload](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)
     (`kueue.gke.io/tpu-enforce-slice-deletion: "true"`).
   - **If a custom scheduler or user created the conflicting slice directly**:
     Its owner can delete it when finished, as described in
     [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).
   - **How `{slice_name}` retries**: Check whether `{slice_name}` already has
     `slice.gke.io/retry-on-failure: "true"` in `Metadata` > `Annotations`. Per
     [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt),
     when a `Slice` custom resource is created with this annotation, "If the
     slice isn't created because of the `SliceCreationFailed` status reason, the
     controller will retry until the slice is successfully formed." Because the
     documentation only shows configuring
     `slice.gke.io/retry-on-failure: "true"` in the `Slice` manifest when
     creating a `Slice` custom resource (and doesn't state that adding the
     annotation to an existing slice already in `SliceCreationFailed` starts a
     retry), if `{slice_name}` wasn't created by Kueue and lacks the annotation,
     tell the user to delete `{slice_name}` following
     [Delete the slice](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
     and re-create it with `slice.gke.io/retry-on-failure: "true"` in its
     manifest following
     [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).
2. **Use available partitions whose nodes no other slice uses**: If the user
   manages `Slice` custom resources directly (not through Kueue) and can't wait,
   direct them to
   [Verify the status of the nodes and the partitions](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
   and
   [Create a Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
   so they can pick available partitions (`HEALTHY` partitions, or `DEGRADED`
   `4x4x4` partitions) whose nodes aren't used by any other slice (across all
   partition shapes on those nodes) and create the `Slice` custom resource with
   them.
3. **Schedule dynamic slices with Kueue and TAS instead of creating `Slice`
   custom resources directly**: Per
   [Target your node pool to belong to a block or a sub-block](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)
   and
   [Create a dynamic slice with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt),
   instead of creating `Slice` custom resources directly, teams can schedule
   GKE workloads with Kueue and Topology Aware Scheduling (TAS), where "Kueue
   automatically handles the creation and deletion of Slice custom resources"
   and admits a workload after its slice becomes active.
4. **Verify or escalate**: Repeat Step 1 following
   [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt):
   - **`ACTIVE`**: The slice is fully formed, healthy, and ready to execute
     workloads.
   - **`ACTIVE_DEGRADED`**: The slice is formed, and workloads can run, but
     performance might be impacted.
   - **Still `SliceCreationFailed` after no other slice uses the nodes, or
     `FAILED`**: Hand off to
     `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or advise the user
     to contact Cloud Customer Care following
     [Before you contact support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
     and
     [Recommendations for efficient support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt).