FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[blas][rocblas] Restore the pointer mode when a rocBLAS call throws by zjin-lcf · Pull Request #764 · uxlfoundation/oneMath · GitHub

[blas][rocblas] Restore the pointer mode when a rocBLAS call throws - #764

Merged
ndingle-arm merged 2 commits into
uxlfoundation:developfrom
zjin-lcf:fix/rocblas-pointer-mode-leak-486
Aug 27, 2026
Merged

ndingle-arm merged 2 commits into
uxlfoundation:developfrom
zjin-lcf:fix/rocblas-pointer-mode-leak-486

Conversation

Copy link
Copy Markdown
Contributor

Summary

Fixes the pointer-mode leak that can produce the intermittent RotTests.RealSinglePrecision failure reported in #486.

The reduction routines (asum, nrm2, dot, dotc, dotu, rotg, rotm, rotmg, iamax, iamin) switch the rocBLAS handle to device pointer mode and switch it back after the call:

rocblas_set_pointer_mode(handle, rocblas_pointer_mode_device);
...
rocblas_native_func(func, err, handle, n, x_, std::abs(incx), res_);
rocblas_set_pointer_mode(handle, rocblas_pointer_mode_host);

rocblas_native_func throws a rocblas_error on any status other than rocblas_status_success, so the reset is skipped. Handles are cached in thread-local storage and reused, so the handle stays in device pointer mode for every subsequent call on that thread.

Routines that pass a host address for their scalar arguments then have that address dereferenced on the device. rot is the most exposed one — it forwards (rocDataType2*)&c and (rocDataType3*)&s and never sets the pointer mode itself, relying entirely on the handle already being in host mode. On a discrete GPU, where host memory is not device-addressable, that is an illegal address.

This replaces the manual set/reset pairs at all 13 sites with an RAII guard, so the mode is restored during stack unwinding as well as on the normal path. The guard restores the previous mode rather than unconditionally forcing host mode, so it is safe to nest.

Validation

On an AMD Instinct MI210 (gfx90a, ROCm 7.1.1, DPC++), provoking a failure inside a guarded region and then calling rot:

crashes
without the guard 10 / 10
with the guard 0 / 10

Without the guard the rot call fails with hipErrorIllegalAddress (700) and leaves the queue unrecoverable. With it, rot returns correct results and a following asum is correct. Both arms take the identical first exception on the same queue, so the guard is the only difference.

Reproducer:

// step 1: provoke a failure inside a pointer-mode-guarded region
try {
    oneapi::math::blas::column_major::asum(q, n, x, 1, (float*)nullptr).wait();
} catch (const std::exception& e) { /* expected */ }

// step 2: rot passes &c and &s, which are host addresses
const float c = 1.0f, s = 0.0f;          // identity rotation
oneapi::math::blas::column_major::rot(q, n, x, 1, y, 1, c, s).wait();

Also ran the BLAS test suite on the same MI210 with the fix applied: 740 tests pass with no failures.

Caveats

Test plan

Made with Cursor

The reduction routines switch the rocBLAS handle to device pointer mode and
switch it back once the call returns. rocblas_native_func throws on any status
other than success, which skips the reset. Handles are cached per thread and
reused, so the handle stays in device pointer mode for every later call on that
thread.

Routines that pass a host address for their scalar arguments then have that
address dereferenced on the device. rot is the most exposed: it forwards &c and
&s and relies on the handle already being in host pointer mode. On a discrete
GPU this is an illegal address, which matches the intermittent failure of
RotTests.RealSinglePrecision reported in uxlfoundation#486.

Set the mode through an RAII guard so it is restored while unwinding. The guard
restores the previous mode rather than forcing host mode, so it nests safely.

Verified on an AMD Instinct MI210: after provoking a failure inside a guarded
region, a following rot aborts with hipErrorIllegalAddress in 10 of 10 runs
without the guard and in 0 of 10 with it.

Co-authored-by: Cursor <cursoragent@cursor.com>
zjin-lcf requested a review from a team as a code owner August 15, 2026 16:38

melonakos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is a real bug fix and a genuine net simplification — +37/-53 while making the code more correct is the right direction. Approving.

The problem is well identified. rocBLAS handles are cached and reused across calls, so if rocblas_native_func threw between the set_pointer_mode(device) and the manual set_pointer_mode(host) at the bottom, the handle stayed in device mode for every subsequent caller — and the next routine passing a host address for a scalar argument would have rocBLAS dereference it on the device. That's a nasty, action-at-a-distance failure mode, and the old comment block ("we need to reset this to the default value in order to avoid invalid memory accesses") shows the hazard was known but only handled on the happy path. RAII is exactly the right fix.

I verified the conversion is complete — I checked every remaining rocblas_set_pointer_mode occurrence in rocblas_level1.cpp at your head commit, and all 11 hits are inside comments (plus one long-dead commented-out call in rot). No live call site was left on the old manual pattern, and level2/level3/extensions/batch never used it. Nice job not leaving the cleanup half-done — that's usually where these go wrong.

Details I liked: deleting copy/assign, and seeding previous_ to rocblas_pointer_mode_host before the get so a failed query still degrades to the sane default rather than to garbage.

Two small things, neither blocking:

  1. Unchecked rocBLAS status. The constructor ignores the return of both rocblas_get_pointer_mode and rocblas_set_pointer_mode, while the rest of this backend routes rocBLAS calls through the error-checking helpers. The destructor obviously can't throw, so leave that one alone — but it'd be more consistent to check in the constructor. The old code ignored these too, so this isn't a regression either way.

  2. Behavior change worth naming explicitly. The old code unconditionally forced the mode back to host; the guard restores whatever was there on entry. Since every call site now goes through the guard, the invariant holds and I think your version is the more principled one. But it does remove an accidental self-healing property: if some other path ever leaked device mode, the old code would have normalized it back and the guard won't. I'm fine with this — just flagging it so the change is deliberate rather than incidental.

One thing I noticed that is out of scope for this PR — no action wanted here: the USM overloads of dot, sdsdot, rotg, rotm, and rotmg never set device pointer mode at all, while the USM asum/iamax/iamin/nrm2 do. That asymmetry predates your change. If it's a real gap it deserves its own issue; please don't grow this PR to cover it.

ndingle-arm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Approved: sound, focused fix with strong rationale and validation. The CI testing gap is worth noting but non-blocking.

  • The RAII guard correctly restores pointer mode during exceptions, covers all 13 former manual reset sites, and checks constructor operations. This matches rocBLAS pointer-mode semantics. AMD documentation (https://rocm.docs.amd.com/projects/rocBLAS/en/develop/reference/helper-functions.html)
  • The primary commit message is excellent.
  • No automated rocBLAS regression test was added but the reported MI210 results are persuasive.

ndingle-arm merged commit f8f9a60 into uxlfoundation:develop Aug 27, 2026
11 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL