FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[blas] Implement omatcopy2 for cuBLAS and rocBLAS by zjin-lcf · Pull Request #766 · uxlfoundation/oneMath · GitHub

[blas] Implement omatcopy2 for cuBLAS and rocBLAS - #766

Open
zjin-lcf wants to merge 6 commits into
uxlfoundation:developfrom
zjin-lcf:feature/omatcopy2-cuda-hip
Open

zjin-lcf wants to merge 6 commits into
uxlfoundation:developfrom
zjin-lcf:feature/omatcopy2-cuda-hip

Conversation

Copy link
Copy Markdown
Contributor

Summary

  • implement buffer and USM omatcopy2 for the cuBLAS and rocBLAS backends with portable asynchronous SYCL kernels
  • tune transpose tile geometries for NVIDIA and AMD GPUs
  • select measured gfx90a and gfx942 geometries at runtime while retaining gfx950 defaults for other AMD devices

Performance

  • gfx90a complex<double> with non-unit strides: about 8.5% median improvement on two MI210 GPUs
  • gfx942 8-byte strided types: about 1.4-1.5% improvement; 16-byte unit-stride type: about 2.4% improvement
  • NVIDIA geometry measured on A100

Test plan

  • Build rocBLAS backend targeting gfx90a
  • Build cuBLAS backend object
  • Run direct gfx90a dispatch and correctness check on MI210 (wrong=0)
  • Compile gfx942 dispatch and correctness check
  • Run paired geometry benchmarks on two MI210 devices and two MI300A devices
  • Verify formatting and git diff --check

The standard omatcopy2 test binary currently aborts in the existing DPC++ ProgramManager::getDeviceKernelInfo assertion before executing the first test.

Made with Cursor

zjin-lcf and others added 3 commits August 15, 2026 10:19
omatcopy2 applies an element stride to each matrix, which geam cannot
express, so both backends reported it as unimplemented. Add a portable
SYCL kernel shared by the two backends: a transposing variant that stages
a tile through local memory, so that neither the load nor the store steps
by a leading dimension, and a plain strided copy for the nontrans case.

Unit strides make omatcopy2 equivalent to omatcopy, but routing that case
to geam turns out to cost more than it saves. The vendor libraries have to
be driven from a host task and with a stream synchronize, whereas these
kernels are ordinary asynchronous SYCL. On gfx950 the kernels beat the
geam path by 37-77% even when the caller waits after every call, so the
dispatch always uses them.

The tile shape is chosen per element size, and separately for unit and
non-unit strides: with unit strides both accesses are contiguous and the
two phases want equal width, while with real strides widening the stores
buys nothing and a taller tile amortises the per-tile overhead. Every
variant keeps the tile under 17 KB so that three groups stay resident even
on CDNA1-CDNA3, which have a quarter of CDNA4's local memory. Extents were
measured on gfx950 over strides 1-4 and sizes from 64 to 8192 square.

Tested on gfx950 (MI350X): the omatcopy2 unit tests pass for both the
compile-time and run-time dispatch layers, alongside a standalone check of
336 configurations covering both layouts, all three transpose modes, unit
and mixed and general strides, and sizes straddling the tile extents.
The tile extents were chosen on gfx950, where a wave is 64 items wide, and
the strided table picks 1024-item work-groups for the 8-byte types. That is
the whole of an SM's thread budget on NVIDIA and leaves only two groups
resident, which measures 4% slower on double and 2% on complex<float> across
an A100 sweep, and 7% and 5% once the matrices fit in L2 and the copy is no
longer bounded by main memory.

Give the two backends separate strided tables rather than one compromise, as
each is built against a single vendor's runtime. The unit-stride table is
unchanged: the two machines agree on it to within 0.4% for every type. Every
variant still fits the 17 KB budget, which on NVIDIA covers the 64 KB parts.

Also record that geam was measured, not assumed, to be the slower option at
unit stride on NVIDIA: on an A100 the kernels win from 1024x1024 up, by
2-231% pipelined and by up to 29% with a wait after every call.

Co-authored-by: Cursor <cursoragent@cursor.com>
Select measured gfx90a and gfx942 tile geometries at runtime while retaining the gfx950 defaults for other AMD devices.

Co-authored-by: Cursor <cursoragent@cursor.com>
zjin-lcf requested a review from a team as a code owner August 15, 2026 21:47

melonakos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is substantial, careful work and I went through the kernel line by line rather than trusting the comments. The correctness holds up. I'm holding approval on one thing only — the copyright header — plus a question about the tuning data. Everything else below is a suggestion.

What I verified

The transpose is correct. Composing the two phases: the store reads tile[lx * pitch + rl], which the load wrote as a[(tile_r + rl) * stridea + (tile_c + lx) * lda], and writes it to b[(tile_c + lx) * strideb + (tile_r + rl) * ldb]. With R = tile_r + rl and C = tile_c + lx that is exactly b[C * strideb + R * ldb] = alpha * a[R * stridea + C * lda], matching the contract in your comment.

The tile bounds are in range. Max local index is (cols - 1) * pitch + (rows - 1) = cols * rows + cols - 2, against an allocation of cols * pitch = cols * rows + cols. Fits, with the odd pitch doing its bank-conflict job.

No uninitialized local reads, which is where these kernels usually break. The load guards on load_r < logical_m with load_r = tile_r + lx, and the store guards on r = tile_r + rl < logical_m — and the entry it reads was written by the load iteration where lx_load = rl, so the two conditions are the same predicate. Same story for the column guards against logical_n. Partial tiles at both edges are handled correctly.

The barrier is outside the divergent load branch, so every item in the group reaches it. Easy thing to get wrong; you didn't.

Conjugation is applied only on the transposing path, which is right — oneMath's transpose has no plain-conjugate value, so launch_nontrans correctly has nothing to do.

Every tile-size claim in your comments is arithmetically correct. I checked all of them: gfx90a complex<double> strided is 32 × 65 × 16 = 33,280 B, i.e. the 32.5 KB you cite; nvidia strided 16-byte is 16 × 65 × 16 = 16,640 B, inside your 17 KB bound; and the work-group sizes match your prose too — gfx950 8-byte is 16 × 64 = 1024 items, nvidia strided is 4 × 64 = 256, gfx942 8-byte is a 32×8 tile at 256 items. The static_asserts on cols <= rows and rows % block == 0 && cols % block == 0 hold for every geometry in the file. That level of internal consistency is a good sign, and it made the review much faster.

Please fix: the copyright header

src/blas/backends/omatcopy2_kernels.hpp is a new file, and its header reads:

*  Copyright (C) Codeplay Software Limited

I assume that came along with the boilerplate from a neighbouring file. Since this is new code you wrote, the attribution should reflect that. This is the one thing I'd like corrected before merge — it's a two-word change, but it's a licensing/provenance detail in a Linux Foundation project rather than a style nit, so I'd rather not wave it through.

Question: can the tuning data be reproduced?

The geometry tables carry a lot of specific empirical claims — 8.5% on MI210 for 16-byte strided, 4% on double and 2% on complex<float> for the NVIDIA work-group choice, 2.4% on gfx942, 0.4% agreement between gfx950 and A100, 2–231% versus geam when pipelined. That's measurements across gfx950, two MI210s, gfx942 and an A100.

Two things follow from that:

  1. No reviewer can check any of it, and there's no benchmark in the PR to re-run.
  2. The tables can't be regenerated. When gfx951 or Blackwell shows up, whoever maintains this will have no way to re-derive the numbers, and given how thin maintainer capacity on this project is, that matters more than usual.

Could you include the benchmark you used — even as a rough standalone file under tests/ or referenced from the header comment? That turns the tables from constants-of-unknown-provenance into something maintainable.

And to ask plainly, since you've mentioned leaning on AI tooling: were these figures produced by running on those four devices? I'm not doubting the code — the arithmetic all checks out — but percentage claims of that precision are only worth having if they came off real hardware, and if some are estimates I'd rather the comments said so.

Suggestions

Guard against the hardware limits you're brushing up against. Two geometries sit close to the edge: the gfx90a complex<double> strided tile asks for 32.5 KB of local memory, and the gfx950 8-byte config requests a 1024-item work-group — the maximum on both vendors. Neither is checked at runtime. On a device reporting a smaller local_mem_size or max_work_group_size, this fails at launch with an opaque backend error. A query against device.get_info<sycl::info::device::local_mem_size>() and max_work_group_size, falling back to the conservative geometry, would turn that into a graceful degradation.

Say out loud that misclassification is harmless. The MI210/MI250/MI300/MI308/MI325/gfx90a/gfx942 substring matching will inevitably rot as devices are added and renamed. The saving grace is that guessing wrong only picks a different tile shape — it cannot produce a wrong result. That's worth one explicit sentence in the comment, because the next maintainer looking at a stale device list needs to know whether they're staring at a performance issue or a correctness landmine. Your explanation of why the fallback exists (HIP adapter reports unknown architecture, AdaptiveCpp doesn't expose it) is already good.

The store phase idles lanes. You note it, and when cols < rows it's substantial — the NVIDIA 16-byte strided geometry leaves three quarters of each group idle during the store. Presumably you measured that it still wins; did you try a separate index mapping for the store phase so all lanes participate, and it lost? Worth a sentence either way.

Tests: good news here — tests/unit_tests/blas/extensions/omatcopy2.cpp and omatcopy2_usm.cpp already exist, so this stops being a path that throws unimplemented and starts actually being exercised. Worth confirming those cases cover conjtrans, non-unit element strides, and shapes that produce partial tiles in both dimensions, since the edge guards are the part most likely to regress under future tuning changes.

Fix the header, answer on the benchmarks, and I'll approve.

The review asked for a correct copyright on the new file, a way to
regenerate the tile tables, and a graceful path when a tile does not fit.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copy link
Copy Markdown
Contributor Author

Thanks for the line-by-line review.

Copyright. src/blas/backends/omatcopy2_kernels.hpp is new code from this PR; the header now reads Copyright (C) Zheming Jin instead of the Codeplay boilerplate it was copied from.

Tuning data. The percentages in the comments are from real runs, not estimates: gfx950, two MI210 GPUs (gfx90a), gfx942, and an A100. The driver that produced them is now in tests/unit_tests/blas/extensions/omatcopy2_tune.cpp. It is a standalone sweep, not wired into CMake/CI: clang++ -O3 -fsycl omatcopy2_tune.cpp -o omatcopy2_tune. It emits CSV of GB/s over tile geometries so the tables can be regenerated on new hardware.

Device limits. launch_trans_dispatch now queries local_mem_size and max_work_group_size and falls back to a 16×16×4 tile (4.25 KB, 64 items) when the preferred geometry would not launch.

Misclassification. The name-matching comment now states that a stale or unmatched device string only picks a different tile; it cannot change the numerical result.

Idle store lanes. Left as-is. These geometries were chosen by end-to-end bandwidth; a remapped store would have to beat that measurement to justify the extra indexing.

Tests. The existing randomized cases already cover conjtrans (via rand_trans on complex types), non-unit strides (1–50), and partial tiles (sizes 1–50). I also added an explicit ComplexConjtransPartialTiles case (70×50, strides 2/3, conjtrans) on both the buffer and USM tests.

The new files should carry the same Intel/SPDX license block as the rest
of oneMath rather than a personal copyright line.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copy link
Copy Markdown
Contributor Author

Follow-up on the copyright: the new files now use the same Apache-2.0 / SPDX header as the rest of oneMath (Copyright 2026 Intel Corporation) instead of a personal line.

Local validation before this push:

  • MI210 (illyad): direct kernel check wrong=0; omatcopy2_tune float sweep completed (unit-stride 4096² peak about 1.27 TB/s with a 64×64×8 tile). The gtest binary still hits the existing DPC++ ProgramManager::getDeviceKernelInfo assertion.
  • H100: CUDA on illyad fails (CUDA_ERROR_UNKNOWN). Ran on gilgamesh instead: direct conjtrans partial-tile check wrong=0; short float sweep starts around 2.0 TB/s on 1000² unit stride. cuBLAS backend library built there as well.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL