| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Sorry, something went wrong.
There was a problem hiding this comment.
This pull request modifies the channel affinity logic in GapicSpannerRpc.java. Specifically, it disables the setting of AFFINITY_KEY when dynamic channel pooling is disabled to prevent sticky binding issues under concurrent load, allowing the system to perform fresh least-busy channel picks instead. I have no feedback to provide.
Sorry, something went wrong.
Under grpc-gcp (default since 6.105.0, googleapis#4239), newCallContext set GcpManagedChannel.AFFINITY_KEY on every data RPC. GcpManagedChannel binds each new key via pickLeastBusyChannel, which reads activeStreamsCount before any concurrent caller's start() has incremented it (tiebreak channelRefs[0]). A high-concurrency cold start therefore binds keys to channel 0 and the bindings are sticky, so RPCs funnel through one HTTP/2 connection and queue at MAX_CONCURRENT_STREAMS. The static-numChannels path bounded the key to ~2*numChannels-1 distinct values, making the collapse permanent (~6x throughput regression at 400 concurrent). The dynamic-channel-pool path used per-transaction random keys and largely self-corrected, but a few BitSet-recycled hints still sticky-bound, leaving a p99 tail. Multiplexed sessions get no backend-locality benefit from sticky per-transaction channel affinity, so don't set the key under grpc-gcp at all. getChannelRef(null) does a fresh per-call least-busy pick with no sticky binding and no affinity-map growth. Drops the now-unobservable distinct-AFFINITY_KEY assertions from RetryOnDifferentGrpcChannelMockServerTest; the request-count and session assertions still cover the retry loop.
|
@akash329d Can you please check if this #12726 fixes your issue? |
Sorry, something went wrong.
|
@rahul2393 IMO its a worse solution because its keeping a bunch of unnecessary complexity. But I think it decreases the severity of the issue significantly. However does not fully fix it [tbf neither does my PR], as the initial burst race still exists. (before any statements have been run, you still have the pickLeastBusyChannel race condition). I think we need to land the GoogleCloudPlatform/grpc-gcp-java#234 to really fix this. |
Sorry, something went wrong.
|
Closing this since we already released another fix https://repo1.maven.org/maven2/com/google/cloud/google-cloud-spanner/6.115.0/ |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Problem
Since 6.105.0 (#4239 enabled grpc-gcp by default), a Spanner client whose first traffic is a high-concurrency burst sees throughput collapse and p50 latency climb. disableGrpcGcpExtension() restores the expected scaling.
(single multiplexed-session client, SELECT 1; each cell is a fresh client whose first traffic is at the target concurrency. Dynamic-pool row from a separate probe with enableDynamicChannelPool() set on the builder.)
Root cause
newCallContext sets GcpManagedChannel.AFFINITY_KEY on every data RPC under grpc-gcp. GcpManagedChannel.getChannelRef(key) binds each new key via pickLeastBusyChannel() (1754-1790), which reads activeStreamsCount and ties to channelRefs.get(0). The count isn't incremented until later in GcpClientCall.start() (:284), so a concurrent first burst all binds to channel 0 and the bindings are sticky. Subsequent RPCs funnel through one HTTP/2 connection and queue at MAX_CONCURRENT_STREAMS.
The static-numChannels path used affinity.intValue() % numChannels (≈63 distinct keys), so the collapse is permanent. The dynamic-pool path mostly self-corrects (p50 matches OFF) because most hints are per-transaction random longs — but MultiplexedSessionDatabaseClient.getSingleUseChannelHint allocates the first numChannels concurrent hints from a recycled BitSet (values 0..N-1), and those few sticky-bind during the warmup race and keep getting reused, leaving a p99 tail. (Separately: in our testing, setting enableDynamicChannelPool=true via JDBC/connection properties did not fully propagate to GcpManagedChannel, so DCP is not currently a workaround for connection-API users.)
Fix
Don't set AFFINITY_KEY when grpc-gcp is on. Multiplexed sessions are a single session, so sticky per-transaction channel affinity provides no backend-locality benefit. With no key, getChannelRef(null) does a fresh per-call least-busy pick with no sticky binding and no affinity-map growth — matches the OFF curve. (Math.floorMod alone doesn't help; the race still ties bounded keys to channel 0.)
RetryOnDifferentGrpcChannelMockServerTest previously asserted distinct AFFINITY_KEY values via an interceptor; with no key set those assertions are unobservable, so they're removed (the request-count and session assertions still cover the retry loop). Note this means the opt-in spanner.retry_deadline_exceeded_on_different_channel feature now relies on grpc-gcp's per-call least-busy pick rather than a forced distinct channel under grpc-gcp (the wedged channel will normally have a higher active-stream count, so least-busy usually picks a different one, but it's no longer guaranteed); the GAX withChannelAffinity path (used when grpc-gcp is off) is unchanged.
Repro
The trigger is the client's first traffic burst being high-concurrency (e.g., a connection pool warming many connections at once); a gentle low-C warmup spreads the keys and masks the bug. Standalone single-file reproducer (only dep google-cloud-spanner):
SpannerAffinityRepro.java