rdma: single PUT for oversize buffers + fix >2 GiB upload truncation - #241
Conversation
Sorry, something went wrong.
|
Warning Review limit reached@harshavardhana, you've reached your PR review limit, so we couldn't start this review. Next review available in: 27 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR. To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 58915b6e-2a51-45b7-abd3-c608503ef6bd 📥 CommitsReviewing files that changed from the base of the PR and between 67a65ad and 8206f96. 📒 Files selected for processing (4)
WalkthroughThe client adds a 4 GiB cuObject registration limit, ranged RDMA GET support, HTTP fallback for oversized buffers, 64-bit upload sizing, and updated CI tooling. ChangesLarge-buffer RDMA fallback
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant RDMA
participant HTTP
participant libcurl
Client->>Client: Compare buffer size with kCuObjMaxMemoryRegSize
alt Buffer is at most 4 GiB
Client->>RDMA: Register and transfer
RDMA-->>Client: Return transfer result
else Buffer exceeds 4 GiB
Client->>HTTP: Send complete buffer in one request
HTTP->>libcurl: Set InfileSizeLarge with curl_off_t
libcurl-->>HTTP: Upload request
end
Possibly related PRs
Poem 🚥 Pre-merge checks | ✅ 5 ✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. ❤️ ShareComment @coderabbitai help to get the list of available commands. |
Sorry, something went wrong.
A buffer larger than a single cuObject registration (cuMemObjGetDescriptor, ~4 GiB) cannot be pinned for RDMA. PutObject/GetObject already fell back to HTTP in that case, but PutObject re-chunked the buffer into a 16 MiB-part multipart upload. Since the buffer is already fully resident, that buys nothing: send it as a single PUT via the request body instead. AIStor accepts a single PUT up to 5 TiB — far beyond the registration ceiling and beyond anything a client can pin or allocate. - Add kCuObjMaxMemoryRegSize (4 GiB) and gate the RDMA attempt on it, so an oversized buffer skips a registration that is guaranteed to fail. - PutObject HTTP fallback now issues one BaseClient::PutObject from the buffer rather than a multipart stream upload. - Document the limit and the single-PUT behavior in the README; the SDK does not chunk registrations — sizing the buffer is the caller's responsibility.
Execute() set the upload size via curlpp::Options::InfileSize, i.e. CURLOPT_INFILESIZE (long), which curl documents as capped at 2 GiB — a larger single-request body was silently truncated. Any single PUT above 2 GiB was affected; it became reachable in practice once an oversized RDMA buffer (> 4 GiB) falls back to a single PUT. Switch to CURLOPT_INFILESIZE_LARGE (curl_off_t), which is correct for all sizes; Content-Length is already stamped from the full 64-bit body.size().
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agentsVerify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@src/client.cc`: - Around line 700-706: Preserve all caller request options in both HTTP fallback paths: in the GET fallback, copy the original GET arguments before clearing buf and size so version, range, conditionals, SSE-C, extra headers/query parameters, and progress callbacks remain intact; in the PUT fallback, reuse the multipart path’s header-building logic and copy extra_query_params, progressfunc, and progress_userdata while retaining metadata, SSE, tags, retention, and legal hold. Update the fallback logic near the RDMA selection and the corresponding PUT path, and add regression coverage for version/range/SSE-C GETs and metadata/SSE/progress PUTs. - Around line 1221-1226: Apply the kCuObjMaxMemoryRegSize guard to the RDMA registration checks in the multipart upload paths at both cuMemObjGetDescriptor call sites using args.part_size. Ensure registration is attempted only when args.part_size is within the limit; otherwise leave RDMA disabled so the HTTP multipart path is selected, preserving the existing direct-buffer PutObject guard.
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info ⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 53444dd9-873e-4094-b0fe-2a92ef6f184f
📥 CommitsReviewing files that changed from the base of the PR and between 313486d and 668888c.
📒 Files selected for processing (4)- README.md
- include/miniocpp/rdma.h
- src/client.cc
- src/http.cc
Sorry, something went wrong.
The buffer/RDMA GetObject ignored GetObjectArgs::offset and always fetched the whole object into the buffer, so an object larger than one cuObject registration (> 4 GiB) had no ranged path — unlike PUT, which can be chunked via multipart. Honor the offset so a caller can stream a large object as a sequence of <= 4 GiB ranged GETs, each of which registers cleanly and rides the RDMA fast path. Per the AIStor server contract (rdmaTransferBounds): the object offset travels in a signed Range header (server derives rangeBase from it) and is independent of the buffer address carried in the RDMA token's StartAddr; the server replies 206 for the range, which the client already accepts. rdmaGet/rdmaGetWithRetry take an object range_offset (-1 = whole object); the HTTP fallback sets the same offset/length so it reads identical bytes.
The workflows floated on vcpkg's default branch and installed CMake 3.28 via apt. On arm64 Linux vcpkg has no prebuilt CMake and falls back to the system one; 3.28 is too old for vcpkg's SPDX generation (`string(JSON ... STRING_ENCODE)`), so any port built from source (e.g. openssl on a cache miss) fails with "invalid mode 'STRING_ENCODE'". amd64 uses vcpkg's own recent CMake and is unaffected. - Pin vcpkg to release 2026.07.29 for reproducible builds. - Use lukka/get-cmake to put a recent cmake+ninja on PATH (fixes arm64), and drop the apt cmake install.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agentsVerify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@include/miniocpp/rdma.h`: - Around line 257-265: Update the ranged-read logic around the range header construction to reject or return early when range_offset is nonnegative and size is zero, before formatting bytes=... Prevent generating a descending range such as offset--1 while preserving valid nonzero range requests and the existing GetObjectArgs::Validate() contract. - Around line 226-232: Update rdmaGet and the related RDMA helper declarations to use std::optional<int64_t> range_offset with a std::nullopt default instead of the -1 sentinel. Adjust the internal RDMA caller to pass the optional value and preserve whole-object behavior when it is unset, including the existing offset behavior when a value is present.
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info ⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: dd29f257-fd72-497b-b342-5f7e87b0addc
📥 CommitsReviewing files that changed from the base of the PR and between 668888c and 2a34de4.
📒 Files selected for processing (4)- .github/workflows/ci-rdma.yml
- .github/workflows/ci.yml
- include/miniocpp/rdma.h
- src/client.cc
Sorry, something went wrong.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agentsVerify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In @.github/workflows/ci-rdma.yml: - Line 55: Replace the mutable lukka/get-cmake@latest reference at .github/workflows/ci-rdma.yml:55 and .github/workflows/ci.yml:113 with the same reviewed immutable commit SHA, preserving the existing action usage and adding no other workflow changes.
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info ⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ac6ca90e-14df-40a7-a95c-387fd32aead4
📥 CommitsReviewing files that changed from the base of the PR and between 2a34de4 and 67a65ad.
📒 Files selected for processing (2)- .github/workflows/ci-rdma.yml
- .github/workflows/ci.yml
Sorry, something went wrong.
… action) - Apply the kCuObjMaxMemoryRegSize (4 GiB) guard to the two multipart RDMA registration sites (parallel pool + serial), matching the direct-buffer guard and the documented contract; a > 4 GiB part now skips the doomed registration and uses the HTTP multipart path. - rdmaGet: reject a zero-length ranged read before formatting the header, which would otherwise emit an invalid descending range (bytes=X-(X-1)). - ci: pin lukka/get-cmake to an immutable commit (v4.4.2) in both workflows instead of the mutable @latest.
CodeRabbit Autofix — AppliedAddressed 3 of 5 review items in 8206f96. Applied:
Deferred (with reason):
Files modified: src/client.cc, include/miniocpp/rdma.h, .github/workflows/ci.yml, .github/workflows/ci-rdma.yml |
Sorry, something went wrong.
What
Two related fixes for the RDMA direct-buffer path.
1. Oversize buffers → single PUT, not multipart
A buffer larger than a single cuObject registration (cuMemObjGetDescriptor, ~4 GiB) cannot be pinned for RDMA. PutObject already fell back to HTTP, but re-chunked the buffer into a 16 MiB-part multipart upload. The buffer is already fully resident, so that buys nothing — send it as a single PUT via the request body instead. AIStor accepts a single PUT up to 5 TiB, far beyond the registration ceiling and beyond anything a client can pin or allocate.
2. Fix >2 GiB upload truncation (CURLOPT_INFILESIZE_LARGE)
Execute() set the upload size with curlpp::Options::InfileSize → CURLOPT_INFILESIZE (long), which curl documents as capped at 2 GiB — larger single-request bodies were silently truncated. This became reachable once an oversized RDMA buffer falls back to a single PUT. Switch to CURLOPT_INFILESIZE_LARGE (curl_off_t), correct for all sizes. Content-Length was already stamped from the full 64-bit body.size().
Testing
Relies on ci-rdma.yml (builds -DMINIO_CPP_ENABLE_RDMA=ON) for compile coverage — I could not build locally (vcpkg deps absent). The single-PUT construction mirrors the existing part_count == 1 path; the InfileSizeLarge change is the 64-bit counterpart of the existing option.
🤖 Generated with Claude Code
Summary by CodeRabbit