| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Running the raw GPUDirect sweep on an IGX Thor devkit surfaced three places
where the system tuning tutorial is wrong or silently unhelpful on a host that
pairs the Thor iGPU with a discrete card.
Generalize the "IGX Orin" tab to "IGX Series" and name both tested
configurations: Orin with an RTX 6000 Ada on IGX SW 1.1, and the Thor devkit
with an RTX PRO 6000 on Ubuntu 24.04 / driver 580. Everything in the tab
applies to both; the only divergence is GPU selection, so it is stated once
rather than repeated per step.
That divergence, verified on the Thor rig:
- CUDA enumerates the two GPUs in the opposite order from nvidia-smi/NVML.
CUDA_VISIBLE_DEVICES=0 resolves to the discrete RTX PRO 6000 while
nvidia-smi calls it index 1, so numeric selection silently targets the
wrong device in either direction. UUID is the only reliable form.
- The Tegra CUDA driver initializes one GPU class per process.
CUDA_VISIBLE_DEVICES=0,1 fails in cuInit rather than returning two devices.
- With no explicit selection CUDA resolves to the iGPU, so
is_any_integrated_gpu() returns true and tune_system.py skips or downgrades
four checks -- peermem, gpudirect, topo, bar1-size -- none of which ever
look at the discrete card. Prefixing the run with the discrete GPU's UUID
restores them.
For containers, --privileged populates /dev with the Tegra iGPU nodes and the
runtime's device selection stops working: --gpus '"device=N"' and
NVIDIA_VISIBLE_DEVICES=GPU-<uuid> both leave CUDA seeing only the iGPU.
-e CUDA_VISIBLE_DEVICES=GPU-<uuid> does work, and the selected GPU becomes CUDA
ordinal 0, so memory_regions[*].affinity must be 0 regardless of what
nvidia-smi reports. Documented against the raw benchmarking container recipe,
including the two forms that do not work, so they are not retried.
ibdev2netdev is not in Ubuntu's infiniband-diags; it ships in mlnx-tools from
the DOCA-Host repository. On a host without that repository the package
installs cleanly and the tool is still missing, which is how the MRRS/MPS/MTU
checks came to be bypassed with only a warning that recommended the package
that had already been installed. Correct the install list and the three
tune_system.py messages, and add a sysfs fallback for the IB-device to netdev
to PCIe mapping that needs no Mellanox tooling.
Also rename the platform-tab key igx-orin to igx-series in the TOC javascript
and stylesheet. The key is positional, not derived from the label, so this is
cosmetic.
No logic changes in tune_system.py; the guidance strings only. The per-GPU
gating fix for check_bar1_size and check_gpu_nic_topology, which short-circuit
on is_any_integrated_gpu() instead of iterating devices the way
check_gpudirect_support already does, is left for a follow-up issue.
Assisted-by: Claude
Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
Greptile SummaryThis PR expands the IGX system-configuration guidance to cover hybrid-GPU IGX Thor hosts and corrects the documented source of ibdev2netdev.
Confidence Score: 5/5The PR appears safe to merge, with the documentation, diagnostic messages, and platform-tab assets remaining internally consistent. No concrete changed-code-triggered runtime, documentation workflow, security, or repository-rule failure remains after checking the package guidance, GPU affinity semantics, tab-key consumers, commit metadata, and whitespace. Important Files Changed
Reviews (1): Last reviewed commit: "#241 - Document IGX Thor hybrid-GPU and ..." | Re-trigger Greptile |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Closes Issue #241
Running the raw GPUDirect sweep on an IGX Thor devkit surfaced three places where the system tuning tutorial is wrong or silently unhelpful on a host that pairs the Thor iGPU with a discrete card.
Generalize the "IGX Orin" tab to "IGX Series" and name both tested configurations: Orin with an RTX 6000 Ada on IGX SW 1.1, and the Thor devkit with an RTX PRO 6000 on Ubuntu 24.04 / driver 580. Everything in the tab applies to both; the only divergence is GPU selection, so it is stated once rather than repeated per step.
That divergence, verified on the Thor rig:
For containers, --privileged populates /dev with the Tegra iGPU nodes and the runtime's device selection stops working: --gpus '"device=N"' and NVIDIA_VISIBLE_DEVICES=GPU- both leave CUDA seeing only the iGPU. -e CUDA_VISIBLE_DEVICES=GPU- does work, and the selected GPU becomes CUDA ordinal 0, so memory_regions[*].affinity must be 0 regardless of what nvidia-smi reports. Documented against the raw benchmarking container recipe, including the two forms that do not work, so they are not retried.
ibdev2netdev is not in Ubuntu's infiniband-diags; it ships in mlnx-tools from the DOCA-Host repository. On a host without that repository the package installs cleanly and the tool is still missing, which is how the MRRS/MPS/MTU checks came to be bypassed with only a warning that recommended the package that had already been installed. Correct the install list and the three tune_system.py messages, and add a sysfs fallback for the IB-device to netdev to PCIe mapping that needs no Mellanox tooling.
Also rename the platform-tab key igx-orin to igx-series in the TOC javascript and stylesheet. The key is positional, not derived from the label, so this is cosmetic.
No logic changes in tune_system.py; the guidance strings only. The per-GPU gating fix for check_bar1_size and check_gpu_nic_topology, which short-circuit on is_any_integrated_gpu() instead of iterating devices the way check_gpudirect_support already does, is left for a follow-up issue.
Assisted-by: Claude