| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
They still retain their public factory methods, just the constructors are now private -- they should have been to begin with.
Update GrpcPlainRequestLoggingInterceptor to access StorageV2ProtoUtils.fmtProto via PackagePrivateMethodWorkarounds
The failure scenarios outlined are not resumable specific, and will be used by single-shot and appendable uploads as well.
This will be used by an upcoming PR that will integrate it with a BidiWriteObject stream. At a high level this class hierarchy models the state and lifecycle events which happen when performing an upload -- in this change append specific, but many generalize to all uploads.
This class is a higher level abstraction over bidi stream, retries and state tracking. Taking care of most concerns once data is in a ChunkSegment.
This will have tests added in a followup commit
…blocking Default is still blocking, but non-blocking can be chosen now.
* Add FlushPolicy.MinFlushSizeFlushPolicy.maxPendingBytes * Add new contructor to ChannelSession to allow the resultFuture to be passed in. * Create temporary BlobAppendableUploadImpl.AppendableUnbufferedWritableByteChannel to allow both old and new implementations to be passed to BlobAppendableUploadImpl -- this interface will be removed in cleanup * Add new implementation using streaming buffer management as a sibling to the existing implementation. The existing implementation will be removed entirely in cleanup. * Refactored tests to follow
…ementation * Add incremental checksums to all messages * Add final object_checksums for all finalization * Remove tests that were testing the same thing -- especially when the setup was fairly involved * Remove some Integration tests that are now covered by unit tests of the bidi stream / bidi upload state (redirect limits, etc) * Refactor lots of common messages and test run flows to reduce duplication and make debugging easier * Ignore some tests that previously would partially split data rather than messages -- the current implementation only evicts when a whole message is consumed * In general, signal `onComplete()` after `onNext` in response to a finalize message * Update ITAppendableUploadTest to run for multiple permutations of BlobAppendableUploadConfigs
All appendable uploads will crc32c checksum their segments always.
Further testing is needed to determine if MaxFlushSizeFlushPolicy can/should be removed all together.
…l to indicate it is non-blocking
Move existing Buffers.{emptyTo,fillFrom} over to StorageNonBlockingChannelUtils, prefix their method names with `blocking`. Add new tests for the methods, and move the existing methods from BuffersTest that make sense.
…#write non-blocking even when accessed from multiple threads Rather than blocking to acquire the lock, we try to acquire the lock, if acquired proceed otherwise return 0 immediately.
…of 2 Some cursory tracing and log inspection showed that initially requesting 2 instead of 1 didn't improve performance at all, but did make downstream tracers harder to reason about.
…tehAndClose to explain why we are blocking
| Back | FazBrowse Home | New Git URL |
Description
feat: breaking behavior rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)
Rewrite internals of BlobAppendableUpload to provide non-blocking write calls, and it take advantage of grpc async message handling.
When AppendableUploadWriteableByteChannel#write(ByteBuffer) is called, an attempt will be made to enqueue the bytes in the outbound queue to GCS.
If there is only enough room to partially consume the bytes provided in the ByteBuffer the write call will return early specifying the number of bytes actually consumed.
As acknowledgements come in from gcs, enqueued messages will be evicted freeing space in the outbound queue. Thereby allowing more bytes to be consumed and enqueued.
Given appendable objects are still in private preview I can't quote any metrics here, however preliminary benchmarking of several million objects across a range of sizes show across the board throughput improvments.
Because the channel's write call is now non-blocking, if you want to block your application until the full buffer is consumed some new helper methods have been added in StorageChannelUtils to provide blocking behavior.
A new method MinFlushSizeFlushPolicy#withMaxPendingBytes(long) has been added to allow limiting the number of pending outbound bytes. The default values is 16MiB, but can be configured lower if necessary.
Release Notes
BEGIN_COMMIT_OVERRIDE
BEGIN_NESTED_COMMIT
feat: breaking behavior rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat: add StorageChannelUtils to provide helper methods to perform blocking read/write to/from non-blocking channels (#3231)
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat: add MinFlushSizeFlushPolicy#withMaxPendingBytes(long) (#3231)
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
fix: update BlobAppendableUploadConfig and FlushPolicy.MinFlushSizeFlushPolicy to default to 4MiB minFlushSize and 16MiB maxPendingBytes (#3249)
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
fix: make FlushPolicy${Min,Max}FlushSizeFlushPolicy constructors private (#3217)
END_NESTED_COMMIT
END_COMMIMT_OVERRIDE
Sub PRs
This PR is made of up the following PRs, in sequence