FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

#17 - Add RTX PRO 6000 benchmark YAMLs and baseline by chloecrozier · Pull Request #162 · NVIDIA/daqiri · GitHub

/ daqiri Public

#17 - Add RTX PRO 6000 benchmark YAMLs and baseline - #162

Open
chloecrozier wants to merge 13 commits into
mainfrom
ccrozier-rtx-pro-6000-bench
Open

#17 - Add RTX PRO 6000 benchmark YAMLs and baseline#162
chloecrozier wants to merge 13 commits into
mainfrom
ccrozier-rtx-pro-6000-bench

Conversation

chloecrozier commented Jun 12, 2026
edited
Loading

Copy link
Copy Markdown
Member

What this does

Makes the RTX PRO 6000 benchmarks measure the cable. The branch described a
machine with no loopback, so the wire configs pointed at the wrong port pair and
the baseline held placeholder numbers. There is a 400 GbE loopback cable now, and
every figure below is a closed loop over it, read from the NIC's own PHY counters.

The results are published as
docs/benchmarks/performance-rtx-pro-6000.md,
alongside the existing DGX Spark page. examples/rtx_pro_6000_baseline.md keeps
the host-specific record — topology, prerequisites, how to re-run — and defers
every number to that page, so there is no second copy to drift.

Results

400 GbE loopback, pause off, MTU 9000, median of 3 repetitions (4 for the UDP
ladder). "Cable" is the rate the link sustained, from the sending port's byte
counter; "App" is what the program consumed.

Benchmark What it does Cable Gb/s App Gb/s
rdma (RoCE RC) 8 MB RDMA writes. The NIC's own engine moves the data — no software sender loop. 396.9 383.5
dpdk GPUDirect 8000 B UDP built in GPU memory, pulled straight from the GPU onto the cable, written straight into another GPU. No CPU touches a payload. 390.5 373.1
ibverbs (MPRQ) The same traffic through the striding-receive-queue ibverbs engine instead of DPDK. 390.4 260.8
dpdk-hds The same journey, but the receiving NIC splits each packet: header to CPU, payload to GPU. 389.0 375.2
socket-tcp Ordinary Linux TCP, 1 MiB writes. 64.3 63.2
socket-udp, flat out Ordinary Linux UDP, 65507 B datagrams, sender unthrottled. 47.1 31.1
socket-udp, paced to 25 The same, sender held to 25 Gb/s. 25.1 25.0

Every run crossed the cable with zero frames lost and zero pause frames — now a
gate rather than a column, so a run failing either is a failure, not a slow result.

RoCE holds 99.2% of the port with the smallest frames of the fast backends, so
frame size is not what separates these results; gaps in the sender are.
socket-udp is the one backend whose sender outruns its own receiver — the port
takes all 46.5 Gb/s off the wire while the application reads 31.1, all of the
difference being datagrams discarded after the frames were safely off the wire.

How much UDP this receiver can take

Sender held to Read Gb/s Clean reps Worst rep
20 Gb/s 20.0 3 of 3 0
25 Gb/s 25.0 4 of 4 0
28 Gb/s 28.0 3 of 4 130,371 lost
30 Gb/s 24.3 0 of 4 193,792 lost
flat out 35.4 0 of 4 364,807 lost

25 Gb/s is the number to quote. 28 is where it stops being dependable: three
repetitions lost nothing and the fourth lost 16% of its datagrams. The median for
that rung is a flat zero and hid it, which is why the ladder counts clean
repetitions instead.

Engine change

socket_config gains rx_buffer_size / tx_buffer_size, applied at socket
creation and exposed to Python. Without them a UDP receiver gets
net.core.rmem_default — 256 KB, under four of the 65507 B datagrams this test
sends — and discards the overflow where no NIC counter and no sender sees it. A
kernel clamp is logged, since being silently capped is how a buffer that looks
configured still drops.

Measurement fixes behind these numbers

Four ways a run could report something other than what it measured, all fixed
here: LD_LIBRARY_PATH searched /opt/daqiri/lib before the build tree, so runs
measured the container image's copy (which is why ibverbs RX looked broken — the
fix was in the source, never the binary); the namespace path dropped
--target-gbps, so every drop-curve rung ran flat out under a different label; a
cell that never reached its target drops nothing and read as the cleanest point on
the curve, so the NIC's transmit rate is now checked against the target; and
dividing the ibverbs cell's bytes by its shorter-lived receiver's window gave
472 Gb/s on a 400 GbE port, so rates are now checked against line rate.
"Sustained" was also the highest single one-second sample; it is now a mean.

Known gaps

  • ibverbs RX consumes 260.8 Gb/s against its own transmit side's 390.4.
  • The UDP receive path allocates and copies per datagram (1.26 us + 0.256 ns/byte),
    holding it near 31 Gb/s. Receiving into pooled buffers instead lifts it to
    42–46 Gb/s, but that is an engine change rather than a benchmark one and is kept
    on a follow-up branch so this PR stays scoped to the harness.
  • No isolcpus here, so spread is a few percent on the fast backends, wider on
    socket-udp.
  • 800 Gb/s is only a bidirectional aggregate; nothing in the harness drives both
    ways.

greptile-apps Bot commented Jun 12, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR delivers the RTX PRO 6000 400 GbE loopback benchmarks: new YAML configs, a discovery-driven runner, NIC-counter validation infrastructure, and a full results page. It also ships two engine fixes (kIbverbsCatchAllPriority truncation fix, apply_socket_int_option return-value semantics) and a new socket_config.rx_buffer_size/tx_buffer_size YAML pair with matching YAML parsing, pybind, and documentation.

  • SocketConfig buffer sizes (include/daqiri/types.h, src/common.cpp, src/engines/socket/daqiri_socket_engine.cpp, python/daqiri_common_pybind.cpp): two new int32_t fields (rx_buffer_size_, tx_buffer_size_) set SO_RCVBUF/SO_SNDBUF at socket creation; the kernel double-stores and clamps, and the engine reads back the granted size and warns when capped. The apply_socket_int_option return value on failure is corrected from !required (which returned true for non-required options, making the new readback guard unreliable) to always false.
  • ibverbs engine fixes (src/engines/ibverbs/daqiri_ibverbs_engine.cpp): kIbverbsCatchAllPriority reduced from 1,000,000 to 0 — mlx5dv_dr_matcher_create takes a uint16_t, so the old value silently truncated and caused the rule to fail with ENOMEM; DrMatchBuf replaced by the full-size DrMatchParam for the catch-all path.
  • Benchmark harness (examples/run_rtx_pro_bench.sh, rtx_pro_nic_counters.sh, rtx_pro_yaml_rewrite.sh, scripts/discover_rtx_pro_topology.sh): runner builds topology from LLDP-confirmed cable pair rather than hardcoded defaults; NIC PHY counter sampler discards ramp intervals and falls back from the app window to the full snapshot when the computed rate exceeds line rate; pacing accuracy check discards cells where the sender never held the requested rate.

Confidence Score: 5/5

  • Safe to merge. The engine changes are bugfixes (ibverbs flow-rule priority truncation, socket buffer-size return-value semantics) and the new benchmark infrastructure is additive.
  • The two engine changes both correct pre-existing misbehavior: the ibverbs catch-all was silently failing to install due to an integer truncation, and apply_socket_int_option was returning true on failure for non-required options. Neither regression was introduced by this PR. The new rx_buffer_size/tx_buffer_size feature is applied at socket-creation time, validated for non-negativity, and documented consistently across all the relevant doc surfaces. The benchmark harness is exercised against real hardware. The one dead-code entry in the CMakeLists.txt static-link loop is harmless: the rdma entry already covers daqiri_rdma.
  • No files require special attention.

Important Files Changed

Filename Overview
src/engines/socket/daqiri_socket_engine.cpp Adds apply_socket_buffer_sizes helper that calls setsockopt/getsockopt for SO_RCVBUF/SO_SNDBUF, applied at UDP socket creation, TCP listen, TCP client, and accepted-socket time. Fixes apply_socket_int_option return value on failure (was !required, now always false), making the new readback guard in apply_socket_buffer_sizes correct. Adds a negative-length guard and max_packet_size bounds check in set_packet_lengths.
include/daqiri/types.h Adds rx_buffer_size_ and tx_buffer_size_ (int32_t, default 0) to SocketConfig. Public API change properly documented in AGENTS.md, docs/api-reference, docs/getting-started.md, docs/tutorials, README, and python pybind.
src/engines/ibverbs/daqiri_ibverbs_engine.cpp Two fixes: kIbverbsCatchAllPriority changed from 1,000,000 to 0 (mlx5dv_dr_matcher_create takes uint16_t, so 1M silently truncated and caused ENOMEM on the rule install); DrMatchBuf replaced by the larger DrMatchParam for the catch-all path, consistent with the full-match buffer already used for flex-parser rules.
examples/CMakeLists.txt Registers new RTX PRO 6000 YAML configs for CMake install. Also adds ibverbs to the static-link engine loop — but the internal CMake target is daqiri_rdma, not daqiri_ibverbs, so this entry never matches anything. The rdma entry already handles linking; the ibverbs entry is dead code.
examples/run_rtx_pro_bench.sh New RTX PRO 6000 benchmark runner that replaces previously hardcoded config paths with LLDP-discovered topology, puts the build tree ahead of /opt/daqiri/lib on LD_LIBRARY_PATH, validates NIC counters, checks pacing accuracy, and routes each backend through the appropriate config-rewrite path.
examples/rtx_pro_nic_counters.sh New shared counter library sourced by the RTX PRO runners. Provides nic_snapshot/nic_report for whole-window rates, and start/stop_wire_sampler for per-second sampled means. Discards ramp intervals, warns on pause frames, falls back from app-window to full snapshot when the computed rate exceeds line rate (the ibverbs two-process case).
src/common.cpp Parses the new rx_buffer_size / tx_buffer_size YAML fields, with a validation that rejects negative values at config-load time.
python/daqiri_common_pybind.cpp Exposes the two new SocketConfig fields (rx_buffer_size, tx_buffer_size) as readable/writable Python attributes via pybind11.
examples/raw_bench_common.cpp Adds a sub-millisecond spin phase to TokenBucketPacer::wait_for_bytes: the last millisecond before the deadline is busy-waited rather than slept, avoiding the sleep-overshoot that previously turned smooth pacing into bursts at rates where the inter-message gap is shorter than a sleep quantum.
docs/benchmarks/performance-rtx-pro-6000.md New benchmark results page for the RTX PRO 6000 400 GbE loopback. Well-documented methodology, honest known-gaps section, and per-section notes about measurement caveats (ibverbs two-process window, GPU-workload 8000 vs 8192 B discrepancy, inconsistent throughput metrics across tables).
scripts/discover_rtx_pro_topology.sh New discovery script that uses LLDP mutual-neighbour confirmation to identify the cabled port pair, reads MACs, resolves PCIe BDFs, and optionally sources rtx_pro_discover.py for GPU ordinals and poll-core affinity. Falls back to carrier-based guessing with an explicit warning when LLDP evidence is unavailable.

Reviews (15): Last reviewed commit: "#17 - Address review feedback on the RTX..." | Re-trigger Greptile

Comment thread examples/CMakeLists.txt Outdated
Partial #17: raw GPUDirect configs for RTX PRO 6000 Blackwell plus
measured baseline on the dev box. No HDS, RoCE, FFT, or GEMM yet.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Adjust sw loopback IP placeholders; expand raw_benchmarking tip block.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
chloecrozier force-pushed the ccrozier-rtx-pro-6000-bench branch from 6b83d46 to 694e527 Compare June 12, 2026 19:46
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Fix three failure classes surfaced by wire closed-loop testing and add a
multi-queue core/payload sweep with plotting.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
…-bench

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

# Conflicts:
#	AGENTS.md
#	examples/CMakeLists.txt
#	examples/bench_pipeline.cu
#	examples/bench_workload.cu
#	examples/bench_workload.h
#	examples/raw_bench_common.cpp
#	examples/raw_bench_common.h
#	examples/raw_gpudirect_bench.cpp
#	examples/raw_hds_bench.cpp
#	examples/rdma_bench.cpp
#	examples/socket_bench.cpp
The merge brought five RTX PRO 6000 example YAMLs (HDS, MQ base, ibverbs RX,
TX-only, RoCE) into examples/ that lacked a leaf in the configuration
walkthrough decision tree, failing scripts/check_doc_refs.py in the docs CI.
Add a decision-tree entry for each so every shipped YAML is covered.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
… fixes for RoCE #223)

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Add rtx_pro_discover.py to pick PIX-linked CUDA ordinals per NIC and
isolcpus-based poll cores from topology. Wire discovery into the MQ and
single-backend runners, split multi-queue TX across two GPUs, default
RoCE workload-max-inflight, and add run_rtx_pro_suite.sh for the standard
RTX PRO sweep.

Signed-off-by: ccrozier <ccrozier@nvidia.com>
Use priority 0 for the mlx5dv_dr catch-all matcher so the uint16_t priority
does not truncate and rule creation no longer fails with ENOMEM during
daqiri_init. Also pin the RTX PRO ibverbs RX hugepage region to NUMA node 0.

Signed-off-by: ccrozier <ccrozier@nvidia.com>
…-bench

# Conflicts:
#	.gitignore
#	AGENTS.md
#	docs/tutorials/configuration-walkthrough.md

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
The RTX PRO configs described a machine with no loopback cable, so the wire
configs pointed at the wrong port pair and the baseline held placeholder
numbers. There is a cable now, and this makes every cell a closed loop over it,
with every rate read from the NIC's own PHY counters.

Harness:

- Resolve the cabled pair from LLDP, or from the wire-loopback namespaces while
  they are up, rather than from enumeration order. This host has a second,
  uncabled ConnectX-7 whose ports also show carrier and look like a pair.
- Fill topology, destination MACs, GPU ordinals and poll cores from discovery,
  and gate every run on pause frames, MTU, the receive-buffer ceiling, and
  frames actually reaching the far port. A run that was flow-controlled or never
  crossed the cable is recorded as a failure, not as a slow result.
- Run RoCE and sockets per role in separate network namespaces, since both ports
  belong to one host and the kernel would otherwise route between them
  internally and never touch the wire.
- Add a container launcher that survives this host's faulted GPU, which makes
  nvidia-container-cli fail NVML enumeration and breaks `docker run --gpus`.

Four ways a run could report something other than what it measured, all fixed:
LD_LIBRARY_PATH searched /opt/daqiri/lib before the build tree, so runs measured
the container image's copy -- which is why ibverbs RX looked broken, the fix
having been in the source and never in the binary under test; the namespace path
dropped --target-gbps, so every drop-curve rung ran flat out under a different
label; a cell that never reached its target drops nothing and so read as the
cleanest point on the curve, hence the NIC's transmit rate is now checked against
the target; and dividing a two-process cell's bytes by its shorter-lived
receiver's window gave 472 Gb/s on a 400 GbE port, hence every rate is now
checked against line rate. "Sustained" was also the highest single one-second
sample, and is now a mean.

Engine: socket_config gains rx_buffer_size / tx_buffer_size, applied at socket
creation and exposed to Python. Without them a UDP receiver gets
net.core.rmem_default -- 256 KB, under four of the 65507 B datagrams this test
sends -- and discards the overflow where no NIC counter and no sender ever sees
it. A kernel clamp is logged, since being silently capped is how a buffer that
looks configured still drops. They have to be configuration rather than a
setsockopt call: a server socket is already receiving during daqiri_init() and a
TCP accepted socket has already inherited the listener's buffers, so both are
sized before an application holds a conn_id. Four docs pages stated the blanket
rule that socket options are never YAML, and now name the exception.

Results are published as docs/benchmarks/performance-rtx-pro-6000.md alongside
the DGX Spark page, and examples/rtx_pro_6000_baseline.md keeps the
host-specific record while deferring every number to that page. RoCE holds
396.9 Gb/s of the 400 GbE port, DPDK GPUDirect 390.5, and socket-udp carries
25 Gb/s without losing a datagram in any repetition.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
chloecrozier force-pushed the ccrozier-rtx-pro-6000-bench branch from 35c0bf4 to 2a55e72 Compare August 26, 2026 15:12
Four files carried no weight in this branch and are removed, along with the
code that existed only to serve them:

- scripts/plot_rtx_pro_bench.py: no published figure comes from it, and both
  callers guarded the invocation with a file test, so nothing depended on it.
- examples/run_rtx_pro_push_400g.sh: the exploratory aggregate-throughput
  harness. No measurement on the results page comes from it; the two runners
  that produce every published number are run_rtx_pro_bench.sh and
  run_rtx_pro_mq_bench.sh. Its removal also retires list_loopback_pairs() and
  RTX_LOOPBACK_PAIRS in discovery, which had no other consumer.
- examples/run_rtx_pro_suite.sh: a wrapper around two commands the docs already
  spell out, referenced by nothing.
- examples/daqiri_bench_rdma_tx_rx_rtx_pro_6000.yaml: unreachable. The runner
  selected it only for a software-loopback RoCE run, which the mode guard
  rejects before the config is ever opened. RoCE on this host has to cross a
  namespace or the two ports resolve each other in-kernel and never reach the
  cable, so the netns base is the only config that can serve it.

No behaviour change to any measured path: 893 fewer lines, mkdocs --strict and
check_doc_refs.py clean, and the rdma sw-smoke guard still reports the same
message it did when the dead branch was present.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

## Results summary

Median of 3 repetitions per cell (4 for the UDP ladder), measured 2026-08-25.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Do we need to include a date here?


| Stream / Protocol | Best case | Sustained on the cable | App consumed | Lost on the cable |
| ----------------- | --------- | ---------------------: | -----------: | ----------------- |
| Socket / RoCE (SEND) | 8 MB message | **396.9 Gb/s** (99.2% of the port) | 383.5 | 0 |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

In the future we should probably include the same size as raw ethernet, otherwise these comparisons are a bit of apples to oranges. RoCE will have the advantage here just due to the message size chosen.

is the subject of [Socket / UDP](#socket-udp) below.

**Frame size is not what separates these results.** RoCE holds the highest rate
with the *smallest* frames of the fast backends (4057 B on the wire, against

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is true, but as mentioned above RoCE also has an 8MB message submitted to it rather than a bunch of 4096B packets, which gives it an advantage. If RoCE were instead given a message size of 8K the performance will drop due to its overhead.

| Figure | What it is |
| ------ | ---------- |
| **Sustained on the cable** | The sending port's byte counter sampled once a second and averaged over the run, discarding the first and last (partial) samples. A mean, not a peak: a maximum is one sample and says nothing about what the link held. It counts the frame, CRC, preamble and interframe gap — everything that occupies the link — so it is directly comparable to the port's 400 Gb/s. |
| **App consumed** | What the receiving program itself reported. Shown so the two can be compared: where they disagree, the difference is being lost inside the host, after the wire already delivered it. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is otherwise known as "goodput" and can be mentioned here. The "Sustained on the cable" is "throughput".

| ------ | ---------- |
| **Sustained on the cable** | The sending port's byte counter sampled once a second and averaged over the run, discarding the first and last (partial) samples. A mean, not a peak: a maximum is one sample and says nothing about what the link held. It counts the frame, CRC, preamble and interframe gap — everything that occupies the link — so it is directly comparable to the port's 400 Gb/s. |
| **App consumed** | What the receiving program itself reported. Shown so the two can be compared: where they disagree, the difference is being lost inside the host, after the wire already delivered it. |
| **Lost on the cable** | Frames the sending port emitted minus frames the receiving port took off. Any non-zero value implicates the link itself. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Maybe just "Lost packets" rather than "Lost on the cable"?

| **Pause frames** | 802.3x flow control seen by either port. Pause throttles a sender without incrementing any drop counter, so a paused run looks clean and is simply slow. One pause frame fails the run. |

A **paced** run gets one more check. If the NIC's transmit rate lands more than 5%
under the rate the sender was told to hold, the cell is discarded rather than

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

or over?

the two ports agreeing to within 0.01 Gb/s in each. The receiver reports ~5,500
dropped packets out of 115 million (0.005%), all of it host-side.

### Payload sweep (best batch per payload, 15 s per cell)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I would add in the title that these are single-core results

# Namespaces down:
./examples/run_rtx_pro_bench.sh dpdk nic-smoke --seconds 30
./examples/run_rtx_pro_bench.sh dpdk sweep --seconds 15
./examples/run_rtx_pro_bench.sh dpdk issue17 --seconds 20

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

could we rename this flag from "issue17" to something more descriptive?

Comment thread examples/rtx_pro_6000_baseline.md Outdated

RamyaGuru Aug 26, 2026
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think this file doesn't need to published, but could be helpful for internal use. So maybe we remove it form the PR. Looks like it has more so the implementation details for the internal cluster. Correct me if I'm wrong, @chloecrozier

Down to 4096 B the two columns agree and the link is the limit. Below that they
part company — and the cable figure falls too, because with one queue per
direction small packets bottleneck the *sender* as well. That is why 64 B reaches
only 33.7 Gb/s on the wire here, while two send queues push 111.8 Gb/s at the same

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

In future benchmarks we should separate out sender from receiver so they can assume the transmitter is always matching what the receiver can do.

### Multi-queue core scaling (20 s per cell)

Each cell is **cable Gb/s / app Gb/s** — what the NIC carried, and what the
receiving program read out of it. `2t1r` is two send queues and one receive queue,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Does this mean one core per queue or is it one core across all queues?

15 Gb/s still reports zero loss under a "paced to 20" label — because the receiver
was never pushed — and reads as the cleanest point on the curve.

## Raw Ethernet / GPUDirect (DPDK)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

DPDK is deprecated at this point with ibverbs being the default backend. We can include it, but ibverbs should be listed first if possible.

running out, and the frames it could not take are dropped inside the host after
the wire had already delivered them.

### GPU work on the received data (8000 B, 3 × 20 s)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

The payload should probably be 8192B here. In the FFT and GEMM cases the non-power-of-two will result in poorer performance when it might be able to keep up.

leaves the GPU. It costs essentially nothing: **389.0 Gb/s** on the cable against
GPUDirect's 390.5, with 375.2 consumed.

## Raw Ethernet / ibverbs (MPRQ)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

You can remove MPRQ since DPDK has this as well

instead of DPDK, as a DPDK sender feeding an MPRQ receiver over the cable.

Sending is not the problem: the transmit port holds **390.4 Gb/s**, matching
DPDK. The MPRQ receive side consumes **260.8**, dropping 65.5 M of 146 M packets.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

If this is saying ibverbs is lower than DPDK we probably should investigate this before publishing since it's the opposite of what I was measuring. Is this with device memory?

teardown and therefore understates. The sampled sustained figure is unaffected
and is the one to use.

## Socket / RoCE

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Maybe say RoCE SEND/RECEIVE here since the terminology is for that


## Socket / TCP

Ordinary Linux TCP, written in 1 MiB chunks, one kernel thread per side doing the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Are we sure a single thread achieved this? We had 30Gbps on the Spark slide

into 8 IP fragments to fit the 9000-byte MTU, and the far side reassembles. UDP
never retransmits, so whatever the receiver cannot keep up with is gone.

This is the one backend whose sender comfortably outruns its own receiver. Flat

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

We should eventually investigate this. UDP should outperform TCP easily.


## Known limitations

- **`ibverbs` RX consumes 260.8 Gb/s** while its own transmit side holds 390.4.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Let's resolve this before merging since this is not what I was getting

- **No physical NIC available**: [`daqiri_bench_raw_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_sw_loopback.yaml). `loopback: "sw"`, no NIC required. Useful for first-time build verification, not representative of production performance.
- **RTX PRO 6000 Blackwell wire loopback** (prefilled): [`daqiri_bench_raw_tx_rx_rtx_pro_6000_nic.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_rtx_pro_6000_nic.yaml). Two 400 GbE ports joined by a loopback cable, `kind: device` on both sides, and each memory region pinned to a GPU that is PIX to its own port. Build with [`CMAKE_CUDA_ARCHITECTURES=120`](../tutorials/bare-metal-cmake-build.md). `examples/run_rtx_pro_bench.sh` fills the PCIe addresses, destination MAC, GPU ordinals, and cores from `scripts/discover_rtx_pro_topology.sh`. See the [RTX PRO 6000 profile callout](../benchmarks/raw_benchmarking.md#update-the-loopback-configuration) for run details and [Performance: RTX PRO 6000](../benchmarks/performance-rtx-pro-6000.md) for what it measures.
- **RTX PRO 6000 Blackwell wire loopback** (template, replace `<placeholders>`): [`daqiri_bench_raw_tx_rx_rtx_pro_6000.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_rtx_pro_6000.yaml). Same shape with PCIe addresses, MAC, and GPU ordinals left blank, for a different RTX PRO host or a cross-card topology.
- **RTX PRO 6000 Blackwell software loopback**: [`daqiri_bench_raw_sw_loopback_rtx_pro_6000.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_sw_loopback_rtx_pro_6000.yaml). `loopback: "sw"` with `kind: device` buffers. Verifies the GPUDirect build on an RTX PRO host without a cable. It never touches the NIC, so its throughput is not a wire rate and can exceed line rate.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

We should drop software loopback in most cases since it should be deprecated/unused in almost all cases.

- **Generic** (template, replace IPs): [`daqiri_bench_rdma_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx.yaml).
- **DGX Spark** (prefilled): [`daqiri_bench_rdma_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark.yaml). See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark) for namespace and wire-counter run details.
- **DGX Spark netns wire loopback** (prefilled, combined base): [`daqiri_bench_rdma_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_netns.yaml). Carries both roles. `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`) so RDMA-CM resolves over the wire. See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark).
- **Netns wire loopback** (prefilled, combined base): [`daqiri_bench_rdma_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_netns.yaml). Carries both roles; `scripts/gen_spark_netns_config.py` splits it per role so each runs in its own network namespace (`--mode server` / `--mode client`) and RDMA-CM resolves over the wire instead of short-circuiting in the kernel. Used by `examples/run_spark_bench.sh` on a Spark and by `examples/run_rtx_pro_bench.sh rdma` on an RTX PRO 6000 — the file is not Spark-specific, and neither is the namespace setup once `CLIENT_IF`/`SERVER_IF` are set. See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark); RTX PRO 6000 rates at [Performance: RTX PRO 6000](../benchmarks/performance-rtx-pro-6000.md#socket-roce).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think this is also what's needed for socket mode (non-roce), otherwise you're just getting loopback results

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This seems generic for all nics. What's the reason it's under an rtx pro name?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Same comment as above

Reviewer terminology and structure fixes, with no change to any published
measurement:

- Throughput/goodput replace "sustained on the cable"/"app consumed", and
  "lost packets" replaces "lost on the cable" (cliffburdick).
- The results date is dropped, RoCE is labelled SEND/RECV, and the ibverbs
  section moves ahead of DPDK with the redundant MPRQ qualifier removed.
- An admonition records that each row is that backend's own best case at its
  own transfer size, so RoCE's 8 MB message flatters it against the 8 KB
  packets the raw backends get, and the rows are not a controlled comparison.
- The payload sweep is titled one core per direction and the multi-queue table
  one core per queue, since "2t1r" did not say whether the queues shared a core.
- TCP is described as a single stream rather than a single core, with the
  per-pair DGX Spark figures linked as the reference the review asked about.
- UDP landing below TCP is explained and filed as an engine open item.
- The methodology section now warns that the summary uses the sampled
  per-second mean while the remaining tables use whole-run averages, so
  throughput must not be read across tables.

Harness changes behind those comments:

- issue17 becomes gpu-workload, with the old spelling still accepted so
  existing result directories keep working (RamyaGuru).
- The pacing gate is symmetric: overshooting the target by more than 5% now
  fails a cell the same way undershooting does, since a sender that overshot
  was not holding the rate the row claims.
- buf_size is derived from payload_size for the raw bases, so a payload larger
  than the 8000 B they were written for still fits, and the GPU-workload mode
  selects 8192 B because cuFFT and cuBLAS both pay for a non-power-of-two
  length. The affected table is flagged as predating that change rather than
  restated.
- The wire sampler moves into rtx_pro_nic_counters.sh and the multi-queue
  runner records the sampled rate too, so both runners emit one metric.
- examples/rtx_pro_6000_baseline.md is removed as internal-only (RamyaGuru).

Unrelated to the page, from the same review: apply_socket_int_option now
reports setsockopt failure even when the option is optional, so the guard in
apply_socket_buffer_sizes stops falling through to getsockopt and printing a
"kernel granted" warning after the call failed.

Refreshing the tables themselves, the 8192 B workload cells and the ibverbs
receive-path result are deliberately not in this commit; they need measurements
that are not yet validated.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

Copy link
Copy Markdown
Member Author

Addressed everything except the comments asking for refreshed numbers. Some of those I had run before but never validated, so rather than publish them I left the existing values in place and flagged the tables that still need re-measuring — including the ibverbs RX result, so that one is still open.

Copy link
Copy Markdown
Member Author

@RamyaGuru @cliffburdick I addressed most of the comments except the ones about the most up-to-date numbers. Some of those I had run in bulk before but never fully verified, so I left current values in there so we can just put the real good numbers when we find them and we can coordinate over slack!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Same comment as before, where this should probably drop the name "spark"

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL