К основному содержанию
скилл CX Codex

nvshmem-troubleshoot-and-report-bugs — скилл для Claude Code

Скилл nvshmem-troubleshoot-and-report-bugs помогает разбирать сбои NVSHMEM по выводу всех PE и готовить текст отчёта об ошибке. Запрос на диагностику сам по себе не разрешает менять состояние системы.

Apache-2.0 ★ 4 106 обновлён 2026-10-23

Установить в Claude Code · Через Skills CLI (Vercel) Все способы и платформы ↓
npx skills add NVIDIA/skills --skill nvshmem-troubleshoot-and-report-bugs -a claude-code
Открыть на GitHub

Что делает nvshmem-troubleshoot-and-report-bugs

Скилл начинает с точной команды запуска и полного stdout и stderr от каждого PE. Для диагностики сбоя внутри NVSHMEM ему нужно подтверждение INFO-логирования; если его нет, он просит повторить запуск с переменными NVSHMEM_DEBUG=INFO, NVSHMEM_DEBUG_SUBSYS=ALL и NVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log. Исключение — сбой до старта NVSHMEM, например ошибка разбора команды или запуска загрузчика.

Дальше скилл отделяет первичную ошибку от каскадных сбоев и сверяет версионные выводы с официальной документацией. Диагностика и подготовка отчёта остаются только для чтения: скилл не запускает процессы, не меняет настройки и не отправляет отчёты. Он выдаёт готовый текст Markdown для передачи дальше. Это значит, что воспроизвести сбой или исправить среду за вас он не сможет.

Какие задачи решает

  • собрать данные запуска NVSHMEM для диагностики
  • найти первичное расхождение между PE
  • проверить, подтверждает ли вывод INFO-логирование
  • подготовить Markdown-отчёт об ошибке для передачи

Примеры запросов агенту

  • Разбери сбой NVSHMEM по этой команде запуска и полному stdout и stderr каждого PE.
  • Проверь, подтверждают ли логи INFO-уровень; если нет, попроси повторить запуск с нужными переменными.
  • Подготовь Markdown-отчёт по этим логам, не меняя файлы и не отправляя его.

Как установить nvshmem-troubleshoot-and-report-bugs

1Через Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill nvshmem-troubleshoot-and-report-bugs -a claude-code

2Вручную: положить файлы

~/.claude/skills/nvshmem-troubleshoot-and-report-bugs/SKILL.md

Папка скилла целиком; для одного проекта — .claude/skills/nvshmem-troubleshoot-and-report-bugs/

По шагам, с командами для каждой платформы: Как установить скилл в Claude Code: пять способов

Вопросы о nvshmem-troubleshoot-and-report-bugs

Как установить nvshmem-troubleshoot-and-report-bugs в Claude Code?

Выполните npx skills add NVIDIA/skills --skill nvshmem-troubleshoot-and-report-bugs -a claude-code, затем используйте скилл из ~/.claude/skills/nvshmem-troubleshoot-and-report-bugs/SKILL.md.

Что делает nvshmem-troubleshoot-and-report-bugs?

Скилл разбирает сбои NVSHMEM по команде запуска и полным логам PE, сверяет версионные выводы с официальной документацией и готовит текст отчёта.

Какие логи нужны для диагностики NVSHMEM?

Нужны точная команда запуска и полный stdout и stderr каждого PE. Для сбоя внутри NVSHMEM вывод должен подтверждать INFO-логирование; иначе скилл запросит повторный запуск с переменными отладки.

Исходный текст

SKILL.md · Apache-2.0
---
name: nvshmem-troubleshoot-and-report-bugs
description: Diagnose NVSHMEM runtime failures and prepare bug reports for launch, crashes, hangs, correctness, transport, or topology issues.
license: Apache-2.0
metadata:
  version: "1.0.0"
  author: NVIDIA NVSHMEM Team <[email protected]>
  tags:
    - nvshmem
    - troubleshooting
    - bug
---

# NVSHMEM Troubleshooting and Bug Reporting

## Purpose

Diagnose NVSHMEM runtime failures from high-signal evidence. Match documented problems conservatively, verify version-sensitive claims with live official documentation, separate NVSHMEM defects from application or environment failures, and prepare a portable report when escalation is useful.

## Prerequisites

- Require no credentials, API keys, or privileged access.
- Work from the user's exact launch command, complete output, and generated logs. Use direct read-only inspection only when the user supplies access and asks for it.
- Use `$nvshmem-docs` when it is registered to verify version-specific behavior, and `$nvshmem-install` when it is registered for installation or build handoffs. Otherwise use the live official documentation linked from the bundled knowledge base.
- Keep [troubleshooting-knowledge-base.md](references/troubleshooting-knowledge-base.md) available for diagnosis and [bug-report-template.md](references/bug-report-template.md) available for escalation.

## Inputs

- **Required:** Obtain the exact command or complete batch-script launch stanza and complete stdout and stderr from every PE, including lines before the first error and any per-PE logs.
- **Conditional:** Obtain output that confirms INFO-level NVSHMEM logging before diagnosing an in-process NVSHMEM failure. Apply the documented pre-start exception only when NVSHMEM demonstrably cannot emit logs.
- **Optional:** Obtain the full NVSHMEM version, CUDA and driver versions, launcher and bootstrap, transport settings, PE/node/GPU counts, rank-to-GPU mapping, topology, smallest failing scale, and last known-good configuration only when they distinguish plausible causes.
- **Source precedence:** Prefer raw artifacts from the current run, then explicit values in the current user prompt or invocation, then non-conflicting agent context, then focused follow-up answers. Report conflicts and prefer the current raw artifact unless the user confirms a correction.

## Limitations

- Do not confirm a root cause from paraphrased errors, incomplete PE output, a partial signature match, or timing changes alone.
- Treat the bundled knowledge base as a routing snapshot, not current authority. Require live official documentation for version-specific claims.
- Do not claim that a fixed-in release proves every earlier release is affected. Treat upgrade compatibility as unverified until it has been checked against the collected environment.
- Do not perform privileged platform changes or directly submit reports. Provide read-only checks, administrator questions, and paste-ready Markdown instead.
- Follow the human-in-the-loop execution boundary below for every command or tool action.

## Instructions

### Human-in-the-Loop Execution Boundary

Keep diagnosis, artifact inspection, command construction, and report drafting read-only. The default tool allowlist is limited to:

- read-only inspection and search of user-provided artifacts in authorized paths, using `rg`, `sed -n`, `head`, `tail`, `wc`, `stat`, or `file`;
- read-only retrieval of official NVSHMEM documentation through `$nvshmem-docs` or web search/open operations; and
- drafting diagnostic commands and report text in the response.

All other tool actions are denied by default. In particular, do not invoke file-write or deletion operations, process or job launch/control, package management, configuration changes, privileged operations, uploads, connectors, or external create/update/submit actions on the user's behalf.

If the user asks for a non-destructive state-changing diagnostic action that remains within this skill's scope, pause immediately before it and:

1. Show the exact command or action and its resolved targets.
2. Explain its expected effects, whether it is reversible, and any data and destination involved.
3. Obtain the user's explicit confirmation for that exact action immediately before execution.

A request to diagnose, reproduce, fix, prepare, or report a bug is not by itself confirmation to perform a state-changing action. Do not infer confirmation from credentials, tool availability, shell access, or earlier approval of a different action. If the command, target, data, destination, or scope changes, disclose the change and confirm again.

Destructive or irreversible operations, direct report submission, artifact upload, and privileged platform changes remain outside this skill's scope even after confirmation. Return paste-ready report text and administrator handoff instructions instead.

### Start With the Command and Output

Ask first for both of the following when either is missing, then stop:

1. The exact command or complete batch-script launch stanza, including environment settings, launcher options, container invocation, and module loads.
2. The complete stdout and stderr from every PE, including lines before the first error and any per-PE log files.

Do not begin with a broad environment questionnaire. Do not diagnose from a paraphrased error when raw output can be obtained.

### Require INFO Logging

Determine whether the captured output contains `NVSHMEM_DEBUG=INFO` or clear INFO-level NVSHMEM records. The command line alone is not sufficient because a launcher may fail to propagate its environment. If the output does not confirm INFO logging, ask the user to repeat the same run with:

```text
NVSHMEM_DEBUG=INFO
NVSHMEM_DEBUG_SUBSYS=ALL
NVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log
```

Preserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Put the variables where the selected launcher propagates them to every PE. For a simple direct launch, prefix them to the command. For `srun`, `mpirun`, `nvshmrun`, or another launcher, invoke `$nvshmem-docs` for the detected NVSHMEM version before giving launcher-specific syntax when propagation is uncertain.

Ask for the repeated command, stdout and stderr, and every generated `nvshmem-<host>-<pid>.log`. Wait for that evidence before analyzing NVSHMEM behavior.

Apply one exception: diagnose immediately when the failure demonstrably occurs before NVSHMEM starts and cannot emit NVSHMEM logs, such as shell parsing, executable lookup, loader startup, or launcher admission failure. State why the debug rerun cannot add NVSHMEM evidence. Do not use the exception merely because an existing error looks familiar.

### Build the Evidence Record

After the logging gate, extract before asking for more information:

- the first causal error or divergence on each affected PE, separate from cascading aborts;
- the NVSHMEM version and any host/device library mismatch reported by INFO logs;
- launcher and bootstrap, selected transports, relevant `NVSHMEM_*` settings, PE/node/GPU counts, and rank-to-GPU mapping;
- whether the failure changes with one PE, one node, different GPU pairs, or multiple nodes;
- the operation and phase: launch, initialization, allocation, communication, synchronization, collective, finalization, or process teardown.

Ask only for missing facts that distinguish plausible causes. Never fabricate a version, topology, environment setting, or reproduction result.

### Diagnose Conservatively

Read [troubleshooting-knowledge-base.md](references/troubleshooting-knowledge-base.md) for every diagnosis after the logging gate or pre-start exception. Use its topic index to select the relevant section, then search exact log signatures within that section before falling back to semantic matches.

Classify the result as exactly one of:

1. **Confirmed bundled knowledge-base match**: the signature and required conditions match an entry.
2. **Version-specific known issue or later-release fix**: official release notes establish applicability to the detected version.
3. **Likely NVSHMEM diagnosis not in the knowledge base**: evidence points into NVSHMEM, but no bundled or documented issue matches.
4. **Likely external issue**: evidence points to the application, CUDA, launcher, loader, network, driver, scheduler, container, or platform configuration.

Use class 1 whenever a bundled entry is confirmed, even when that entry assigns ownership to an external component; state ownership separately in `Diagnosis`. Use class 4 only when no bundled or versioned documented issue matches. For partial signature matches, use class 3 or 4 rather than calling the match confirmed. Treat a similar release-note symptom as a lead, not proof that the user has the same defect. A release note that says only that a bug was fixed establishes the fix release, not every affected earlier release. Describe an older installed version as predating a candidate fix unless official evidence explicitly establishes that it is affected or a controlled upgrade test confirms the match.

Use `$nvshmem-docs` with the complete detected version before making version-sensitive claims. Ask for the version if it is missing; if live verification is unavailable, label those claims unverified and non-actionable.

Before recommending a target upgrade, verify that release's compatibility and known issues against the collected CUDA, driver, GPU, CPU, NCCL, bootstrap, and transport context. If that context is incomplete, present the release as a candidate test and name the missing compatibility checks instead of calling it suitable.

Separate:

- **Observation**: what the command and logs directly show.
- **Documentation evidence**: the matched FAQ, environment-variable entry, limitation, known issue, or fixed issue.
- **Inference**: the most likely explanation and its confidence.
- **Verification**: the smallest safe test that can confirm or reject the inference.

If no entry matches, say exactly: `No matching entry was found in the bundled NVSHMEM troubleshooting knowledge base.` Then provide a likely diagnosis, confidence, competing explanations, and targeted next checks.

For a likely external issue, identify the likely owner and give concrete next checks. Do not stop at “not NVSHMEM.” Route installation and build failures to `$nvshmem-install` when registered; otherwise identify the appropriate installation/build handoff without attempting an unregistered-skill fallback. Route performance-only regressions to an available NVSHMEM performance skill, or to `$nvshmem-docs` when none is available.

Do not perform privileged platform changes. For IMEX, driver, fabric-manager, device-node, kernel-module, NIC, or scheduler configuration, provide read-only checks and tell the user what to ask the administrator to verify.

### Present the Diagnosis

Return these headings:

- `Evidence Collected`
- `Diagnosis`
- `Knowledge-Base Match`
- `Version Applicability`
- `Next Actions`

Include direct official source links for documentation-backed claims. Keep commands copyable and mark placeholders. Prefer one discriminating test at a time over a long generic checklist.

### Prepare a Bug Report or Question

Offer a report when the issue remains unresolved, is likely an NVSHMEM defect, or the user explicitly requests one. Do not routinely offer escalation for a resolved local configuration problem or a clearly external failure.

When INFO-level NVSHMEM logs are absent, explicitly ask the user to reproduce the failure before support submission with:

```text
NVSHMEM_DEBUG=INFO
NVSHMEM_DEBUG_SUBSYS=ALL
NVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log
```

Preserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Ask for the exact repeated command, complete stdout/stderr from every PE, and all generated log files. A draft may still be generated when the user asks to preserve the investigation, but label it incomplete and include this rerun as a required pre-submission step; do not present it as support-ready.

When preparing one, read [bug-report-template.md](references/bug-report-template.md) and fill it from collected evidence. Mirror the `NVSHMEM issue or bug` GitHub form: start the title with `[Issue]:`, use its exact field headings, and select exactly one supported impact value. Preserve exact commands and the first causal error. Use `Not provided` for missing facts and list those gaps; never invent them. Remind the user to redact credentials, tokens, internal hostnames and IPs, and user-identifying data before sharing logs.

Return the Markdown in a fenced block so it can be pasted into GitHub, NVONLINE, or email. Do not require or directly submit through a connector.

## Troubleshooting

Use these patterns when the investigation stalls:

| Error or signal | Likely cause | Solution |
| --- | --- | --- |
| No INFO records appear after setting debug variables | The launcher did not propagate the environment, or the process failed before NVSHMEM started | Verify propagation with version-matched launcher syntax. Use the pre-start exception only when the emitting component proves NVSHMEM did not start. |
| Logs show only peer aborts or launcher cleanup | The visible lines are cascading consequences | Compare every PE log and use the earliest causal or divergent line, including preceding context. |
| A knowledge-base signature matches but its required conditions do not | The match is only a hypothesis | Classify the issue as likely NVSHMEM or likely external, state the mismatch, and run one discriminating test. |
| The NVSHMEM version is missing or live documentation is unavailable | Version applicability cannot be verified | Ask for the complete version and label version-sensitive guidance unverified; do not infer applicability from the snapshot. |
| INFO logging changes or removes the failure | The failure may be timing-sensitive | Preserve both commands and outcomes, repeat enough trials to characterize frequency, and do not call the logging change a fix. |

## Examples

- **Missing evidence:** Given only “NVSHMEM hangs during init,” ask for the exact launch stanza and complete output from every PE, then stop.
- **Missing INFO logs:** Given a complete command and non-INFO output from an in-process failure, request the same run with the three debug settings and all generated logs; preserve scale and arguments.
- **Pre-start failure:** Given `error while loading shared libraries` before any PE enters NVSHMEM, state why INFO logging cannot help, identify the loader as the emitting component, and route an NVSHMEM-library installation problem to `$nvshmem-install`.
- **Confirmed bundled match:** Given `No more teams available` plus matching team-usage conditions, classify it as a confirmed bundled match, verify `NVSHMEM_MAX_TEAMS` for the complete version, and propose one minimal rerun.
- **No match:** State the required no-match sentence, separate observation from inference, give one discriminating test, and offer a filled report only if the issue remains unresolved or looks like an NVSHMEM defect.

<!-- ci trigger -->

© NVIDIA NVSHMEM Team · Apache-2.0 · NVIDIA/skills

Похожие скиллы

Все скиллы по теме «DevOps и инфраструктура» →
troubleshoot-ssi datadog-labs Скилл troubleshoot-ssi для Claude Code находит причины отсутствия APM-трейсов Datadog и сбоев Single Step Instrumentation в Kubernetes. Диагностирует цепочку, когда агент настроен, но трейсы не поступают. CX Codex +1 472установок troubleshoot-observability Microsoft Скилл troubleshoot-observability для Claude Code координирует поиск неисправностей в Azure. Направляет агента при анализе ошибок выполнения, задержек и настройке мониторинга. CX Codex +1 ★ 3 232звёзд репо troubleshoot SuperClaude-Org Команда troubleshoot находит причины ошибок в коде, сборок, просадок скорости и проблем деплоя. По умолчанию только собирает диагностику и не трогает файлы без флага --fix. CX Codex +1 ★ 23 950звёзд репо network-troubleshoot PatrickJS Правило network-troubleshoot ведёт агента через безопасную диагностику сетевых сбоев: от сбора ошибки до проверки результата. CX Codex +1 ★ 41 178звёзд репо webflow-code-component:troubleshoot-deploy webflow Скилл webflow-code-component:troubleshoot-deploy для Claude Code диагностирует ошибки деплоя Webflow Code Components. Запускается после сбоя webflow library share или проблем с опубликованным компонентом. CX Codex +1 1 000установок issue-reporting manojbajaj95 Скилл issue-reporting для Claude Code собирает детали сбоя скилла из claude-gtm-plugin и готовит GitHub issue. Оформляет жалобы на неверный совет или несработавший скилл. CX Codex +1 118установок

Из того же набора312

Весь набор →
runai-diagnose-failed-workload Скилл runai-diagnose-failed-workload разбирает причины падений, нехватки памяти OOMKilled и ошибок запуска контейнеров в Run:ai без вмешательства в кластер. CX Codex +1 ★ 4 106звёзд репо doca-upgrade Скилл doca-upgrade проверяет установленную версию DOCA, показывает разницу с целевой и ждёт явного подтверждения перед обновлением. Он также помогает с откатом и восстановлением после прерванного обновления. CX Codex +1 ★ 4 106звёзд репо doca-telemetry-utils Запускает doca_telemetry_utils на хосте с DOCA и помогает отследить, куда исчезают метрики телеметрии между экспортером и коллектором. CX Codex +1 ★ 4 106звёзд репо doca-socket-relay Скилл помогает пробросить сокеты приложений с хоста на BlueField DPU через DOCA Socket Relay без переписывания кода. CX Codex +1 ★ 4 106звёзд репо doca-dpdk-bridge Скилл doca-dpdk-bridge для Claude Code помогает встроить DOCA Flow и другие возможности DOCA в существующее DPDK-приложение без переписи data-plane. Добавляет аппаратное ускорение на BlueField или ConnectX. CX Codex +1 ★ 4 106звёзд репо doca-dts Скилл doca-dts развёртывает и настраивает службу NVIDIA DOCA Telemetry Service (DTS) на картах BlueField, хостах docker и в кластерах DPF. CX Codex +1 ★ 4 106звёзд репо doca-structured-tools-contract Скилл doca-structured-tools-contract для Claude Code сводит сведения о DOCA через структурированный инструмент или цепочку ручных команд. Проверяет окружение, устройства, возможности и состояние перед коммитом. CX Codex +1 ★ 4 106звёзд репо nemo-fabric-build-adapter Скилл nemo-fabric-build-adapter помогает разрабатывать и проверять адаптеры NVIDIA NeMo Fabric по контракту платформы: от дескрипторов и маппинга конфигураций до реализации жизненного цикла и проверки спецификации. CX Codex +1 ★ 4 106звёзд репо

Спонсоры проекта

Компании, которые помогают держать аналитику открытой