Skip to main content
skill CX Codex

doca-rdma — Claude Code skill: RDMA on BlueField/ConnectX

doca-rdma — a skill for moving data over RDMA on NVIDIA BlueField or ConnectX NICs using RoCE and InfiniBand.

Apache-2.0 ★ 4,106 updated 2026-10-23

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add NVIDIA/skills --skill doca-rdma -a claude-code
Open on GitHub

What doca-rdma does

doca-rdma wraps NVIDIA DOCA's high-level API for RDMA work, avoiding direct calls to libibverbs and librdmacm. The skill covers initializing an RDMA context on a doca_dev, connection approaches (RDMA CM, bridge/OOB, gRPC-based descriptor exchange), eleven task types (Send/Receive/Send-Imm, Read/Write/Write-Imm, Atomic CmpSwap/FetchAdd, Get/Set/Add Remote Sync Event), aligning mmap and RDMA permissions, queue and connection sizing, querying doca_rdma_cap_*, and handling DOCA_ERROR_* results.

Requires DOCA SDK installed on Linux (Ubuntu 22.04/24.04, RHEL, or SLES) with a BlueField DPU or ConnectX NIC. The skill discovers the module via pkg-config --list-all | grep -i doca and resolves its prefix with pkg-config --variable=prefix MODULE, where MODULE is the name from the first query. Validation steps need a working DOCA setup.

For raw QP or verbs control, see the separate doca-verbs skill. General RDMA theory, DOCA installation, and non-RDMA DOCA libraries are out of scope.

What it helps with

  • bring up an RDMA context and connect two hosts or processes
  • pick the right RDMA task type for your data-movement pattern
  • debug errors like "permission denied" or "completions never arrive"
  • align permissions between mmap and RDMA access
  • size queues and query device capabilities

Example prompts for the agent

  • Which RDMA task type should I use for one-sided write with completion via Send-with-Immediate?
  • Set up an RDMA CM connection between two processes on the same host for initial testing
  • Why does Atomic CmpSwap fail with DOCA_ERROR_FULL on the send queue under burst load? How do I increase the queue?

How to install doca-rdma

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill doca-rdma -a claude-code

2Manually: put the files in place

~/.claude/skills/doca-rdma/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/doca-rdma/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about doca-rdma

How do I install doca-rdma in Claude Code?

Run npx skills add NVIDIA/skills --skill doca-rdma -a claude-code, then open ~/.claude/skills/doca-rdma/SKILL.md.

What does doca-rdma do?

It lets you use NVIDIA DOCA's RDMA API to move data between processes or hosts over NVIDIA BlueField or ConnectX NICs, instead of writing code against raw libibverbs and librdmacm.

What do I need to use this skill?

DOCA SDK installed on Linux (Ubuntu 22.04/24.04, RHEL, or SLES) with an active BlueField DPU or ConnectX NIC, plus a working DOCA environment for configuration checks.

Source text

SKILL.md · Apache-2.0
---
license: Apache-2.0
name: doca-rdma
description: >
  RDMA data movement on NVIDIA BlueField/ConnectX NICs (RoCE/InfiniBand)
  with DOCA RDMA, instead of hand-written libibverbs/rdma_cm. Use when the
  user moves data between processes or hosts over the NIC, or hits
  "one-sided read returns permission denied", "completions never arrive
  after submit", "connection callback never fires", atomic
  compare-and-swap over RoCE, or DOCA_ERROR_FULL on the send queue under
  burst, even without naming DOCA RDMA. Covers bringing up an RDMA context
  on a doca_dev, connection methods (RDMA CM, bridge/OOB, or gRPC exchange
  of doca_rdma_export()), the eleven task types (Send/Receive/Send-Imm,
  Read/Write/Write-Imm, Atomic CmpSwap/FetchAdd, Get/Set/Add Remote Sync
  Event), matching mmap + RDMA permissions, queue and connection sizing,
  doca_rdma_cap_* queries, and DOCA_ERROR_* from RDMA calls. Raw QP/verbs
  control belongs to doca-verbs. Refuse and route elsewhere for general
  RDMA / ibverbs theory (queue pairs, MRs, RoCE vs IB), installing DOCA
  itself, or non-RDMA DOCA libraries.
metadata:
  author: NVIDIA <[email protected]>
  kind: library
  doca_version: "3.5"
  tags:
    - doca
    - rdma
    - roce
compatibility: >
  Requires DOCA SDK on Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a
  BlueField DPU or ConnectX NIC. Resolves the local install and module via
  `pkg-config --list-all | grep -i doca` and
  `pkg-config --variable=prefix MODULE`, replacing `MODULE` with the exact
  name returned by the preceding query. The normal module is the umbrella
  `doca`, while split installs may expose a per-library module.
  The validation steps require a live DOCA install.
---

# DOCA RDMA

## Examples

- **"Completions never arrive after submit."** — stay on the DOCA
  RDMA task and progress-engine path. Do not rewrite it as raw verbs.
- **"One-sided read returns permission denied."** — check mmap +
  RDMA permissions and `doca_rdma_cap_*` before changing the task
  type.

## Instructions

Follow Loading order below. Open the tasks companion for configure /
build / run. Open the capabilities companion for the task taxonomy
and safety policy. Do not open every companion file.

## Non-negotiable: the deliverable uses DOCA RDMA, not raw verbs

When this skill is in scope, the user is asking for **DOCA RDMA**. The
program you produce **must link `libdoca_rdma` and call the
`doca_rdma_*` API** (directly in C/C++, or through a thin FFI/cgo shim
from another language). Do **NOT** implement the request with raw
`libibverbs` / `librdmacm` / RDMA-CM and call it done. Those move bytes
but completely bypass DOCA — which defeats the entire purpose of using
this library, loses the DOCA programming model (progress engine, task/
event lifecycle, capability discovery, portability across BlueField/
ConnectX generations), and is the single most common failure mode.

"Raw verbs is fewer lines" / "it avoids building a DOCA binding layer"
is **not** an acceptable reason to bypass DOCA. The correct low-friction
path for a non-C language (Go, Rust, Python, …) is **not** to re-bind
the whole API — it is to start from a **shipped DOCA RDMA sample** under
`$(pkg-config --variable=prefix doca)/samples/doca_rdma/` (substitute the
module resolved on the target) and wrap its entry functions in
a **thin** cgo/FFI shim built with `#cgo pkg-config: doca` (Go) or the
equivalent. That shim is a single small file, not "a large custom
binding layer". See ## build Step 0 and
## modify.

If `pkg-config doca` or the DOCA build fails, **fix the build** (module
name, `PKG_CONFIG_PATH`, sample path) — do not silently fall back to
verbs. A binary whose `ldd` shows no `libdoca_rdma` is a failed
DOCA-RDMA task, regardless of whether bytes moved.

Concrete path (verified on DOCA 3.3, one RoCE port, two processes):

0. **Record the installed version first:** run
   `pkg-config --modversion doca-rdma` and show the result.
1. **Build the shipped pair in place, output outside.** The samples'
   `meson.build` reads `../../../VERSION`, so a copied samples tree does
   not configure. Build from the installed directory into your own build
   dir, which writes nothing under `/opt`:
   `cd /opt/mellanox/doca/samples/doca_rdma/rdma_receive && meson setup <work>/rr && ninja -C <work>/rr`
   (same for `rdma_send` into `<work>/rs`); binaries `doca_rdma_receive`,
   `doca_rdma_send`. Confirm flags with `--help`.
2. **Run both in one command.** Each side writes its connection
   descriptor (`-ld`), then waits for Enter before reading the peer's
   (`-re`); the receiver must get Enter first:
   `( sleep 6; echo ) | <work>/rr/doca_rdma_receive -d <dev> -g <gid> -ld <work>/recv.desc -re <work>/send.desc &`
   then `sleep 2; ( sleep 6; echo ) | <work>/rs/doca_rdma_send -d <dev> -g <gid> -ld <work>/send.desc -re <work>/recv.desc; wait`.
   Green: the receiver logs `Got from sender: "..."` and both print
   `Sample finished successfully`. `Failed to open the file ... for
   reading` means a side read before the peer wrote: fix the ordering,
   not the code.
3. **Extend only after the stock run is green.** For extra output (a
   checksum or proof line), copy the whole samples tree with
   `/opt/mellanox/doca/VERSION` kept three levels above the sample
   directory, change only the payload and the receiver's print, and keep
   the same run sequence and any program names or output format the user
   asked for. Never write a new RDMA program from scratch,
   never edit the installed samples, and never fall back to raw verbs,
   `rdma_cm` code, `ib_write_bw`, `rping` or a loopback.

**Device and GID.** Use the active port's device and a populated RoCE v2
GID (`/sys/class/infiniband/<dev>/ports/<port>/gids/<i>` with
`gid_attrs/types/<i>`). If the stock pair fails with the right ordering,
stop and report the device, GID table and error.

**Where to start:** This skill assumes DOCA is already installed and
the user is doing **hands-on RDMA work** on a BlueField / ConnectX /
host with DOCA. Open the tasks companion if the user wants to *do*
something (configure / build / modify / run / test / debug); open
the capabilities companion when the question is *what can
RDMA express* on this version. If the user has not installed DOCA
yet, route to `doca-setup/SKILL.md` first.

## Example questions this skill answers well

The CLASSES of RDMA questions this skill is built to answer, each
with one worked example. The agent should treat the *class* as the
load-bearing piece — the worked example is a single instance.

- **"How do I bring up an RDMA context and connect two sides?"** —
  worked example: *"set up sender + receiver with RDMA CM on a single
  host for first-run testing"*. Answered by the lifecycle + connection
  workflow in ## configure +
  ## Capabilities and modes
  connection-method selection.
- **"Which RDMA task type fits this data-movement pattern?"** —
  worked example: *"one-sided write + completion via Send-with-Immediate
  for a small control message"*. Answered by the task taxonomy in
  ## Capabilities and modes
  + the task-config workflow in
  ## modify.
- **"What mmap permissions does this task need? Do I have to export
  the mmap?"** — worked example: *"my Read task fails with insufficient
  permissions"*. Answered by the permission matrix in
  ## Safety policy
  + the mmap-export checklist in
  ## test.
- **"Is this RDMA capability supported on my device + transport?"** —
  worked example: *"does this device support Atomic Compare-and-Swap
  over RoCE"*. Answered by the capability-query rule
  (`doca_rdma_cap_task_*_is_supported` against a `doca_devinfo`) in
  ## Capabilities and modes
  + the discovery step in
  ## configure.
- **"Is this RDMA API available on my installed DOCA version?"** —
  worked example: *"is RDMA CM in DOCA 2.6.0"*. Answered by the
  version-compatibility section in
  ## Version compatibility
  + the version-discovery rule (`pkg-config --modversion doca`)
  pinned in ## configure.
- **"What does this `DOCA_ERROR_*` from an RDMA call mean and which
  layer caused it?"** — worked example: *"`DOCA_ERROR_BAD_STATE` from
  `doca_rdma_connection_disconnect`"*. Answered by the RDMA overlay
  on the cross-library taxonomy in
  ## Error taxonomy
  + the layered ladder in
  ## debug that escalates to
  `doca-debug/SKILL.md`.

## Audience

This skill serves **external developers building applications that
consume the DOCA RDMA library** — i.e., users whose code calls
`doca_rdma_*` (directly in C/C++, or through FFI/bindings from
another language) to do RDMA data movement between two sides
(host↔host, host↔BlueField, DPU↔DPU, or SF↔SF on a BlueField). It
is *not* for NVIDIA developers contributing to DOCA RDMA itself.

**Language scope.** DOCA RDMA normally ships as a C library *inside the
umbrella `doca` pkg-config module* (public header `doca_rdma.h`, shared
object `libdoca_rdma.so`); split installs may expose a per-library module.
Always discover the module on the target (`pkg-config --list-all |
grep -i doca`) rather than assuming either layout. The
shipped samples are written in C
(NVIDIA's choice). C and C++ consumers are the canonical case and
the worked examples in the tasks companion assume that path. Other-language
consumers (Rust, Go, Python, …) consume the same `*.so` through FFI
or language-specific bindings; the skill's contribution in that case
is to keep the lifecycle, capability-discovery, permission-matrix,
error-taxonomy, and connection-method guidance language-neutral, and
to route the agent to the public C ABI as the authoritative surface
that any wrapper will eventually call. **The non-C deliverable is still
a DOCA program**: a thin cgo/FFI shim over the shipped `doca_rdma`
sample that links `libdoca_rdma` (`#cgo pkg-config: doca`) — never a
raw-libibverbs reimplementation chosen to avoid wrapping DOCA (see the
mandate at the top of this file).

## When to load this skill

Load this skill when the user is doing hands-on DOCA RDMA work, in
any language. Concretely:

- Initializing an RDMA context on a `doca_dev`. The Core order is
  create → configure (task `set_conf` + mmap/RDMA permissions +
  optional `set_*`) → start → connect → use → stop → destroy.
  Configuration MUST finish before `doca_ctx_start()`.
- Establishing a connection — picking between RDMA CM
  (`doca_rdma_connect_to_addr()` / `doca_rdma_start_listen_to_port()`
  / `doca_rdma_connection_accept()`), bridge / OOB
  (`doca_rdma_bridge_*`), or gRPC (out-of-band exchange
  of `doca_rdma_export()` output).
- Setting permissions on `doca_mmap` correctly for the chosen task
  type (Read needs RDMA-read + local read-write; Write needs
  RDMA-write; Atomic needs RDMA-atomic; Send needs only local
  read-write).
- Reading / setting library properties via `doca_rdma_set_*` and
  `doca_rdma_cap_get_*` to size queues, list lengths, and
  transport-type selection.
- Checking which RDMA task types and transport types are supported
  on the active `doca_devinfo`.
- Debugging a `DOCA_ERROR_*` returned from an RDMA call (lifecycle
  vs. permission vs. capability vs. driver-below) and the connection
  state-machine transitions (`doca_rdma_set_connection_state_callbacks`).
- Designing or extending non-C bindings (Rust, Go, Python, …) that
  wrap the RDMA C ABI — for the lifecycle, permission, and
  capability rules the wrapper must honor.

Do **not** load this skill for general DOCA orientation, install of
DOCA itself, or non-RDMA library questions. For those, use
`doca-public-knowledge-map/SKILL.md`.

## What this skill provides

This is a **thin loader**. The body keeps only the orientation
needed to pick the right next file. The substantive RDMA-specific
material lives in two companion files:

- the capabilities companion — what RDMA can express on this version: the
  eleven task types and their permission matrix, the three
  connection methods, transport types (RC baseline and alpha-level
  DC for the export/connect CPU-datapath flow — there is no UD) — note
  these are the per-QP service
  type controlled by `doca_rdma_set_transport_type()`, NOT the
  link-layer (IB vs RoCE) which is inherited from the device
  port configuration, the
  capability-query surface (`doca_rdma_cap_*`), the RDMA error
  taxonomy (mapped onto the cross-library `DOCA_ERROR_*` set), the
  observability surface (per-task events, connection state callbacks),
  and the safety policy that gates permission and export decisions.
- the tasks companion — step-by-step workflows for the six in-scope RDMA
  verbs: `configure`, `build`, `modify`, `run`, `test`, `debug`.
  Plus a `Deferred task verbs` block that points out-of-scope
  questions at the right next skill.

The skill assumes a host or BlueField where DOCA is already
installed at the standard location and the user has the privileges
their public install profile expects. It does not cover installing
DOCA — that path goes through
`doca-setup/SKILL.md`.

## What this skill deliberately does not ship

This skill is **agent guidance**, not a samples or templates bundle.
To keep the boundary clean, it deliberately does not contain — and
pull requests should not add:

- **Pre-written DOCA RDMA application source code, in any
  language.** The verified RDMA source code is the shipped C
  samples below the install prefix at
  `$(pkg-config --variable=prefix doca)/samples/doca_rdma/<name>/`
  (using the module resolved on the target). The
  agent's job is to route the user to those files and prescribe a
  minimum-diff modification on them via the universal
  modify-a-sample workflow in
  `doca-programming-guide/SKILL.md`,
  layered with the RDMA-specific overrides in
  ## modify.
- **Standalone build manifests** (`meson.build`, `CMakeLists.txt`,
  `Cargo.toml`, …) parked inside the skill. The agent constructs
  the build manifest *in the user's project directory* against the
  user's installed DOCA, where `pkg-config --modversion doca`
  is the source of truth (resolve the module per ## build
  Step 0 — prefer `doca-rdma` when it resolves).
- **A `samples/`, `bindings/`, or `reference/` subtree** of any
  kind. A mock or incomplete artifact in this skill's tree, even
  one labeled "reference", is misleading: users will read it as
  buildable.

## Loading order

1. Read this skill first to confirm the user's question is in
   scope.
2. If the question is *what RDMA can express*—task taxonomy,
   link-layer vs transport-type selection, connection methods,
   permissions, errors, observability, or safety—read
   the capabilities companion.
3. If the question is *how to do it*—configure, build, modify, run,
   test, or debug—read the matching H2 in the tasks companion.
   Read both companions only when the workflow depends on a
   capability or safety decision; do not load both unconditionally.

Both companion files cross-link to each other and to
`doca-public-knowledge-map/SKILL.md`
whenever the right answer is "look it up in the public docs or the
installed package layout" rather than "RDMA-specific guidance".

## Related skills

- `doca-public-knowledge-map/SKILL.md` — the
  routing table for every public DOCA documentation source and the
  on-disk layout of an installed DOCA package. Always available
  alongside this skill.
- `doca-setup/SKILL.md` — env preparation,
  install verification, and the *I have no install yet* path with
  the public NGC DOCA container. This skill assumes its
  preconditions are satisfied.
- `doca-programming-guide/SKILL.md` —
  general DOCA programming patterns shared by every library: the
  canonical `pkg-config` + meson build pattern, the universal
  modify-a-shipped-sample first-app workflow, the universal
  lifecycle, the cross-library `DOCA_ERROR_*` taxonomy, and the
  program-side debug order. This skill layers RDMA specifics on
  top.
- `doca-debug/SKILL.md` — the cross-cutting
  debug ladder (install / version / build / link / runtime /
  program / driver). RDMA-specific debug (state-machine transitions,
  permission failures, connection callbacks) overlays on top of
  that ladder.


© NVIDIA · Apache-2.0 · NVIDIA/skills

Similar skills

All skills for DevOps and infrastructure →

From the same set312

The whole set →
nvflare-autofl nvflare-autofl is a skill that runs candidate-based optimization campaigns for existing NVFLARE jobs and compares results when an optimization objective is defined. CX Codex +1 ★ 4,106repo stars doca-flow Keeps the agent on DOCA Flow when writing packet steering code for NVIDIA NICs. Covers match/action pipes, ports, representors and counters. CX Codex +1 ★ 4,106repo stars doca-dpa doca-dpa is a Claude Code skill for host-side DOCA DPA work on BlueField. It helps set up the context, launch kernels, inspect device capabilities, and troubleshoot errors. CX Codex +1 ★ 4,106repo stars g-assist-mcp-skill This skill connects Claude Code and Codex to NVIDIA G-Assist for reading and changing display and GPU settings. Use it for refresh rate, G-SYNC, brightness, color, and performance controls. CX Codex +1 ★ 4,106repo stars doca-flow-tune Assists in benchmarking and fine-tuning doca-flow pipelines on BlueField DPUs and ConnectX NICs using the doca_flow_tune utility. CX Codex +1 ★ 4,106repo stars doca-sta This skill guides DOCA STA NVMe-oF target work on BlueField, from checking device capabilities to sizing queues and diagnosing I/O errors. CX Codex +1 ★ 4,106repo stars doca-public-knowledge-map doca-public-knowledge-map is a Claude Code skill that locates official NVIDIA DOCA docs, libraries, samples, and paths. It activates when DOCA, BlueField, ConnectX, or /opt/mellanox is mentioned. CX Codex +1 ★ 4,106repo stars doca-telemetry-exporter Guides telemetry publishing, schema setup, and error debugging for applications linking the DOCA Telemetry Exporter library. CX Codex +1 ★ 4,106repo stars

Project sponsors

Companies that keep this analytics open