FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

chore: remove old appendble upload implementation by BenWhitehead · Pull Request #3227 · googleapis/java-storage · GitHub

chore: remove old appendble upload implementation - #3227

Closed
BenWhitehead wants to merge 1 commit into
nonblocking-appendable/10/channel-bootablefrom
nonblocking-appendable/11remove-old-impl
Closed

chore: remove old appendble upload implementation#3227
BenWhitehead wants to merge 1 commit into
nonblocking-appendable/10/channel-bootablefrom
nonblocking-appendable/11remove-old-impl

Conversation

BenWhitehead commented Aug 5, 2025
edited
Loading

Copy link
Copy Markdown
Collaborator

Failing compilation is expected here as the test classes have not yet been updated to use the new implementation, and we're deleting the old one.

BenWhitehead requested a review from a team August 5, 2025 19:31
product-auto-label Bot added size: xl Pull request size is extra large. api: storage Issues related to the googleapis/java-storage API. labels Aug 5, 2025
BenWhitehead force-pushed the nonblocking-appendable/10/channel-bootable branch from c0febee to bff6b91 Compare August 5, 2025 19:54
BenWhitehead requested a review from a team August 5, 2025 19:54
BenWhitehead force-pushed the nonblocking-appendable/11remove-old-impl branch from 174cd8f to 71e4e98 Compare August 5, 2025 19:55
BenWhitehead force-pushed the nonblocking-appendable/10/channel-bootable branch from bff6b91 to 287df3a Compare August 5, 2025 20:42
BenWhitehead force-pushed the nonblocking-appendable/11remove-old-impl branch from 71e4e98 to 60a6b1f Compare August 5, 2025 20:45
BenWhitehead force-pushed the nonblocking-appendable/10/channel-bootable branch from 287df3a to bff9131 Compare August 8, 2025 20:55
BenWhitehead force-pushed the nonblocking-appendable/11remove-old-impl branch from 60a6b1f to 1e3b2ad Compare August 8, 2025 20:55
BenWhitehead force-pushed the nonblocking-appendable/10/channel-bootable branch from bff9131 to fb34cb2 Compare August 11, 2025 19:39
BenWhitehead force-pushed the nonblocking-appendable/11remove-old-impl branch from 1e3b2ad to acff0cd Compare August 11, 2025 19:39

BrandonY left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This one's much easier to review 👍

BenWhitehead added a commit that referenced this pull request Aug 20, 2025
## Description
feat: *breaking behavior* rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)

Rewrite internals of BlobAppendableUpload to provide non-blocking write calls, and it take advantage of grpc async message handling.

When `AppendableUploadWriteableByteChannel#write(ByteBuffer)` is called, an attempt will be made to enqueue the bytes in the outbound queue to GCS.
If there is only enough room to partially consume the bytes provided in the `ByteBuffer` the write call will return early specifying the number of bytes actually consumed.

As acknowledgements come in from gcs, enqueued messages will be evicted freeing space in the outbound queue. Thereby allowing more bytes to be consumed and enqueued.

Given appendable objects are still in private preview I can't quote any metrics here, however preliminary benchmarking of several million objects across a range of sizes show across the board throughput improvments.

Because the channel's write call is now non-blocking, if you want to block your application until the full buffer is consumed some new helper methods have been added in StorageChannelUtils to provide blocking behavior.

A new method `MinFlushSizeFlushPolicy#withMaxPendingBytes(long)` has been added to allow limiting the number of pending outbound bytes. The default values is 16MiB, but can be configured lower if necessary.

## Release Notes

BEGIN_COMMIT_OVERRIDE

BEGIN_NESTED_COMMIT
feat: *breaking behavior* rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
feat: add StorageChannelUtils to provide helper methods to perform blocking read/write to/from non-blocking channels (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
feat: add MinFlushSizeFlushPolicy#withMaxPendingBytes(long) (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: update BlobAppendableUploadConfig and FlushPolicy.MinFlushSizeFlushPolicy to default to 4MiB minFlushSize and 16MiB maxPendingBytes (#3249)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: make FlushPolicy${Min,Max}FlushSizeFlushPolicy constructors private (#3217)
END_NESTED_COMMIT

END_COMMIMT_OVERRIDE

## Sub PRs
This PR is made of up the following PRs, in sequence
1. #3217
2. #3218 
3. #3219
4. #3220
5. #3221
6. #3222
7. #3223
8. #3224 
9. #3225 
10. #3226 
11. #3227 
12. #3228 
13. #3229 
14. #3230 
15. #3235 
16. #3236 
17. #3241
18. #3242
19. #3246
20. #3248
21. #3249
22. #3252

Copy link
Copy Markdown
Collaborator Author

Merged in #3231

BenWhitehead deleted the nonblocking-appendable/11remove-old-impl branch August 20, 2025 21:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/java-storage API. size: xl Pull request size is extra large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL