FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

#241 - Document IGX Thor hybrid-GPU and ibdev2netdev gaps by RamyaGuru · Pull Request #246 · NVIDIA/daqiri · GitHub

/ daqiri Public

#241 - Document IGX Thor hybrid-GPU and ibdev2netdev gaps - #246

Open
RamyaGuru wants to merge 1 commit into
mainfrom
docs/241-igx-thor-system-tuning
Open

#241 - Document IGX Thor hybrid-GPU and ibdev2netdev gaps#246
RamyaGuru wants to merge 1 commit into
mainfrom
docs/241-igx-thor-system-tuning

Conversation

Copy link
Copy Markdown
Collaborator

Closes Issue #241

Running the raw GPUDirect sweep on an IGX Thor devkit surfaced three places where the system tuning tutorial is wrong or silently unhelpful on a host that pairs the Thor iGPU with a discrete card.

Generalize the "IGX Orin" tab to "IGX Series" and name both tested configurations: Orin with an RTX 6000 Ada on IGX SW 1.1, and the Thor devkit with an RTX PRO 6000 on Ubuntu 24.04 / driver 580. Everything in the tab applies to both; the only divergence is GPU selection, so it is stated once rather than repeated per step.

That divergence, verified on the Thor rig:

  • CUDA enumerates the two GPUs in the opposite order from nvidia-smi/NVML. CUDA_VISIBLE_DEVICES=0 resolves to the discrete RTX PRO 6000 while nvidia-smi calls it index 1, so numeric selection silently targets the wrong device in either direction. UUID is the only reliable form.
  • The Tegra CUDA driver initializes one GPU class per process. CUDA_VISIBLE_DEVICES=0,1 fails in cuInit rather than returning two devices.
  • With no explicit selection CUDA resolves to the iGPU, so is_any_integrated_gpu() returns true and tune_system.py skips or downgrades four checks -- peermem, gpudirect, topo, bar1-size -- none of which ever look at the discrete card. Prefixing the run with the discrete GPU's UUID restores them.

For containers, --privileged populates /dev with the Tegra iGPU nodes and the runtime's device selection stops working: --gpus '"device=N"' and NVIDIA_VISIBLE_DEVICES=GPU- both leave CUDA seeing only the iGPU. -e CUDA_VISIBLE_DEVICES=GPU- does work, and the selected GPU becomes CUDA ordinal 0, so memory_regions[*].affinity must be 0 regardless of what nvidia-smi reports. Documented against the raw benchmarking container recipe, including the two forms that do not work, so they are not retried.

ibdev2netdev is not in Ubuntu's infiniband-diags; it ships in mlnx-tools from the DOCA-Host repository. On a host without that repository the package installs cleanly and the tool is still missing, which is how the MRRS/MPS/MTU checks came to be bypassed with only a warning that recommended the package that had already been installed. Correct the install list and the three tune_system.py messages, and add a sysfs fallback for the IB-device to netdev to PCIe mapping that needs no Mellanox tooling.

Also rename the platform-tab key igx-orin to igx-series in the TOC javascript and stylesheet. The key is positional, not derived from the label, so this is cosmetic.

No logic changes in tune_system.py; the guidance strings only. The per-GPU gating fix for check_bar1_size and check_gpu_nic_topology, which short-circuit on is_any_integrated_gpu() instead of iterating devices the way check_gpudirect_support already does, is left for a follow-up issue.

Assisted-by: Claude

Running the raw GPUDirect sweep on an IGX Thor devkit surfaced three places
where the system tuning tutorial is wrong or silently unhelpful on a host that
pairs the Thor iGPU with a discrete card.

Generalize the "IGX Orin" tab to "IGX Series" and name both tested
configurations: Orin with an RTX 6000 Ada on IGX SW 1.1, and the Thor devkit
with an RTX PRO 6000 on Ubuntu 24.04 / driver 580. Everything in the tab
applies to both; the only divergence is GPU selection, so it is stated once
rather than repeated per step.

That divergence, verified on the Thor rig:

  - CUDA enumerates the two GPUs in the opposite order from nvidia-smi/NVML.
    CUDA_VISIBLE_DEVICES=0 resolves to the discrete RTX PRO 6000 while
    nvidia-smi calls it index 1, so numeric selection silently targets the
    wrong device in either direction. UUID is the only reliable form.
  - The Tegra CUDA driver initializes one GPU class per process.
    CUDA_VISIBLE_DEVICES=0,1 fails in cuInit rather than returning two devices.
  - With no explicit selection CUDA resolves to the iGPU, so
    is_any_integrated_gpu() returns true and tune_system.py skips or downgrades
    four checks -- peermem, gpudirect, topo, bar1-size -- none of which ever
    look at the discrete card. Prefixing the run with the discrete GPU's UUID
    restores them.

For containers, --privileged populates /dev with the Tegra iGPU nodes and the
runtime's device selection stops working: --gpus '"device=N"' and
NVIDIA_VISIBLE_DEVICES=GPU-<uuid> both leave CUDA seeing only the iGPU.
-e CUDA_VISIBLE_DEVICES=GPU-<uuid> does work, and the selected GPU becomes CUDA
ordinal 0, so memory_regions[*].affinity must be 0 regardless of what
nvidia-smi reports. Documented against the raw benchmarking container recipe,
including the two forms that do not work, so they are not retried.

ibdev2netdev is not in Ubuntu's infiniband-diags; it ships in mlnx-tools from
the DOCA-Host repository. On a host without that repository the package
installs cleanly and the tool is still missing, which is how the MRRS/MPS/MTU
checks came to be bypassed with only a warning that recommended the package
that had already been installed. Correct the install list and the three
tune_system.py messages, and add a sysfs fallback for the IB-device to netdev
to PCIe mapping that needs no Mellanox tooling.

Also rename the platform-tab key igx-orin to igx-series in the TOC javascript
and stylesheet. The key is positional, not derived from the label, so this is
cosmetic.

No logic changes in tune_system.py; the guidance strings only. The per-GPU
gating fix for check_bar1_size and check_gpu_nic_topology, which short-circuit
on is_any_integrated_gpu() instead of iterating devices the way
check_gpudirect_support already does, is left for a follow-up issue.

Assisted-by: Claude
Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR expands the IGX system-configuration guidance to cover hybrid-GPU IGX Thor hosts and corrects the documented source of ibdev2netdev.

  • Documents UUID-based GPU selection and CUDA ordinal handling for host and privileged-container workflows.
  • Adds a sysfs fallback for InfiniBand-device, netdev, and PCIe mapping.
  • Updates tune_system.py diagnostics to identify the correct packages and repositories.
  • Renames the IGX platform-tab key and label consistently from Orin-specific to IGX Series.

Confidence Score: 5/5

The PR appears safe to merge, with the documentation, diagnostic messages, and platform-tab assets remaining internally consistent.

No concrete changed-code-triggered runtime, documentation workflow, security, or repository-rule failure remains after checking the package guidance, GPU affinity semantics, tab-key consumers, commit metadata, and whitespace.

Important Files Changed

Filename Overview
docs/tutorials/system_configuration.md Broadens the IGX tutorial to Thor, adds hybrid-GPU selection guidance, corrects package installation information, and provides a tooling-independent sysfs mapping fallback.
docs/benchmarks/raw_benchmarking.md Documents the verified privileged-container invocation and CUDA ordinal requirements for hybrid IGX Thor systems.
python/tune_system.py Changes only diagnostic strings so missing ibdev2netdev and setpci messages identify their actual package sources.
docs/javascripts/platform-tab-toc.js Renames the positional IGX tab key consistently with the updated tutorial label and stylesheet selectors.
docs/stylesheets/extra.css Updates platform-specific TOC filtering selectors to consume the renamed IGX tab key.

Reviews (1): Last reviewed commit: "#241 - Document IGX Thor hybrid-GPU and ..." | Re-trigger Greptile

RamyaGuru requested a review from dleshchev August 18, 2026 12:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL