feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy by chandra-siri · Pull Request #17262 · googleapis/google-cloud-python · GitHub

1. Overview of the Solution

This solution implements end-to-end full-object checksum validation in AsyncMultiRangeDownloader for the asynchronous Google Cloud Storage Python client library. As asynchronous multiplexed downloads of non-contiguous ranges are performed concurrently over a single bidirectional gRPC connection, this feature automatically and incrementally calculates a rolling checksum as bytes arrive and validates it against the server's authoritative object checksum once the download completes.

The technical approach consists of three coordinated layers:

_AsyncReadObjectStream (Stream Ingestion): Safely extracts the authoritative server checksum (full_obj_server_crc32c) and finalization status (is_finalized) from the object metadata received in the first data payload response of the stream.
_ReadResumptionStrategy & _DownloadState (Verification Logic): Computes an isolated, persistent rolling checksum in the individual _DownloadState object to ensure calculations do not bleed across concurrent multiplexed ranges. Crucially, the rolling hash updates only after buffer writes succeed to prevent state corruption during retry re-connects, raising a DataCorruption exception on completion if a mismatch occurs.
AsyncMultiRangeDownloader (Orchestration & Cleanup): Detects candidate full-object ranges (e.g., (0, 0) or (0, persisted_size)), propagates checksum settings to the resumption strategy, and guarantees robust cleanup (closing the stream immediately and unregistering IDs) if data corruption or write errors occur.

2. What This PR Specifically Does

This PR implements Step 2: Full-Object Rolling Checksum & Resumption Verification Logic of the solution:

Upgrades _DownloadState to track is_full_object_read and initialize an isolated google_crc32c.Checksum() rolling instance.
Updates _ReadResumptionStrategy.update_state_from_response() to run buffer writes before updating the rolling checksum, ensuring transactional safety during connection failures and retry reconnects.
Optimizes performance by bypassing rolling checksum calculations entirely if enable_checksum is False.
Performs the final validation match at range_end against the server's authoritative checksum, raising a DataCorruption exception if a mismatch is found.
Adds comprehensive unit tests in test_reads_resumption_strategy.py to verify successful validation, failure exceptions, and bypassed checks when validation is disabled.

…stream

…esumption strategy

Code Review

This pull request introduces support for full-object rolling checksum verification during resumed reads, allowing verification once the stream completes. It also adds a flag to conditionally disable checksum validation. The review feedback suggests optimizing performance by bypassing rolling checksum updates when checksums are disabled, and recommends adding a unit test to verify that full-object checksum mismatches are ignored when checksum validation is disabled.

…mport lint error

…lures

…yncio fix from main

…est-asyncio issues

…hNanoseconds

…on is disabled

PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v0.19.0 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:234b9d1f2ddb057ed7ac6a38db0bf8163d839c65c6cf88ade52530cddebce59e <details><summary>gapic-generator: v1.35.0</summary> ## [v1.35.0](gapic-generator-v1.34.1...gapic-generator-v1.35.0) (2026-06-11) ### Features * setup.py matches prerelease versions (#17370) ([25b857e](25b857e1)) ### Bug Fixes * require protobuf 6.33.5 to address CVE-2026-0994 (#17349) ([6642263](66422636)) </details> <details><summary>google-auth: v2.54.0</summary> ## [v2.54.0](google-auth-v2.53.0...google-auth-v2.54.0) (2026-06-11) ### Features * implement regional access boundary support for standalone JWT and async service accounts (#17025) ([35af616](35af6168)) ### Bug Fixes * configure mTLS for impersonated credentials (#17404) ([57269d5](57269d56)) * fail-fast on missing ECP config file to avoid 30s hang (#17377) ([e096127](e0961270)) * Rename the 'seed' argument for setting an initial regional access boundary for clarity (#17186) ([e5c8cf9](e5c8cf92)) * update incorrect urls in setup.py to point at monorepo vs splitrepo (#17237) ([eaed04b](eaed04ba)) </details> <details><summary>google-cloud-alloydb: v0.11.0</summary> ## [v0.11.0](google-cloud-alloydb-v0.10.0...google-cloud-alloydb-v0.11.0) (2026-06-11) ### Features * update API sources and regenerate (#17413) ([59fe7cf](59fe7cf8)) </details> <details><summary>google-cloud-biglake: v0.5.0</summary> ## [v0.5.0](google-cloud-biglake-v0.4.0...google-cloud-biglake-v0.5.0) (2026-06-11) ### Features * update API sources and regenerate (#17431) ([2e75c78](2e75c78c)) </details> <details><summary>google-cloud-ces: v0.7.0</summary> ## [v0.7.0](google-cloud-ces-v0.6.0...google-cloud-ces-v0.7.0) (2026-06-11) ### Features * update API sources and regenerate (#17413) ([59fe7cf](59fe7cf8)) </details> <details><summary>google-cloud-confidentialcomputing: v0.11.0</summary> ## [v0.11.0](google-cloud-confidentialcomputing-v0.10.0...google-cloud-confidentialcomputing-v0.11.0) (2026-06-11) ### Features * update API sources and regenerate (#17413) ([59fe7cf](59fe7cf8)) </details> <details><summary>google-cloud-modelarmor: v0.7.0</summary> ## [v0.7.0](google-cloud-modelarmor-v0.6.0...google-cloud-modelarmor-v0.7.0) (2026-06-11) ### Features * update API sources and regenerate (#17413) ([59fe7cf](59fe7cf8)) </details> <details><summary>google-cloud-network-services: v0.10.0</summary> ## [v0.10.0](google-cloud-network-services-v0.9.0...google-cloud-network-services-v0.10.0) (2026-06-11) ### Features * update API sources and regenerate (#17431) ([2e75c78](2e75c78c)) </details> <details><summary>google-cloud-oracledatabase: v0.6.0</summary> ## [v0.6.0](google-cloud-oracledatabase-v0.5.0...google-cloud-oracledatabase-v0.6.0) (2026-06-11) ### Features * update API sources and regenerate (#17413) ([59fe7cf](59fe7cf8)) </details> <details><summary>google-cloud-spanner: v3.68.0</summary> ## [v3.68.0](google-cloud-spanner-v3.67.0...google-cloud-spanner-v3.68.0) (2026-06-11) ### Features * add asynchronous code snippets and minor cleanup changes (#17337) ([d6aaf61](d6aaf610)) ### Performance Improvements * optimize query result decoding (#17375) ([3f70b2f](3f70b2ff)) </details> <details><summary>google-cloud-storage: v3.12.0</summary> ## [v3.12.0](google-cloud-storage-v3.11.0...google-cloud-storage-v3.12.0) (2026-06-11) ### Features * full object checksum: implement rolling checksum and verification in reads resumption strategy (#17262) ([2361ba6](2361ba6e)) * Enable full object checksum PR 1/3 : parse finalize_time and server crc32c in async object stream (#17261) ([72c7a27](72c7a272)) * full object checksum: integrate full-object checksum in AsyncMultiRangeDownloader (#17263) ([b6a85e4](b6a85e49)) </details> <details><summary>google-developer-knowledge: v0.1.0</summary> ## [v0.1.0](google-developer-knowledge-v0.0.0...google-developer-knowledge-v0.1.0) (2026-06-11) ### Features * add google-developer-knowledge (#17417) ([ca02afc](ca02afce)) </details>

chandra-siri added 2 commits May 26, 2026 20:53

feat(storage): parse finalize_time and server crc32c in async object …

42ab953

…stream

feat(storage): implement rolling checksum and verification in reads r…

d972c2c

…esumption strategy

chandra-siri requested a review from a team as a code owner May 26, 2026 20:54

gemini-code-assist Bot reviewed May 26, 2026

View reviewed changes

Comment thread packages/google-cloud-storage/google/cloud/storage/asyncio/retry/reads_resumption_strategy.py Outdated

Comment thread packages/google-cloud-storage/tests/unit/asyncio/retry/test_reads_resumption_strategy.py

chandra-siri changed the title feat(storage): implement rolling checksum and verification in reads resumption strategy feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy May 26, 2026

chandra-siri added 10 commits May 26, 2026 21:02

style(storage): apply user feedback on finalize_time and fix unused i…

7110744

…mport lint error

merge: integrate updated finalize_time checks and lint fixes from PR 1

a437b66

merge: integrate latest main to resolve pytest-asyncio event loop fai…

9704da7

…lures

style(storage): run ruff format to resolve CI lint failures

31ef2e3

merge: integrate PR 1 updates including ruff formatting and pytest-as…

25c39e2

…yncio fix from main

test(storage): add conftest autouse event loop fixture to resolve pyt…

c371f59

…est-asyncio issues

merge: integrate PR 1 updates including conftest.py event loop fixture

21144f9

fix(storage): use second attribute instead of seconds for DatetimeWit…

c470641

…hNanoseconds

merge: integrate PR 1 second attribute fixes

3820424

perf(storage): bypass rolling checksum updates when checksum validati…

2f49dcb

…on is disabled

chandra-siri requested review from kalragauri and v-pratap May 27, 2026 09:20

chandra-siri assigned kalragauri and v-pratap May 27, 2026

kalragauri approved these changes May 29, 2026

View reviewed changes

Comment thread packages/google-cloud-storage/google/cloud/storage/asyncio/retry/reads_resumption_strategy.py

Base automatically changed from feat/stream-checksum-metadata to main June 4, 2026 09:42

chandra-siri added 2 commits June 4, 2026 10:10

chore: trigger workflows

8fdb571

fix(storage): skip rolling checksum creation when validation is disabled

465d210

chandra-siri enabled auto-merge (squash) June 4, 2026 11:08

chandra-siri merged commit 2361ba6 into main Jun 4, 2026
32 checks passed

chandra-siri deleted the feat/full-object-rolling-checksum branch June 4, 2026 11:22

sofisl mentioned this pull request Jun 11, 2026

chore: librarian release pull request: 20260611T192009Z #17432

Merged

feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy by chandra-siri · Pull Request #17262 · googleapis/google-cloud-python · GitHub

Uh oh!

feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy - #17262

feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy#17262
chandra-siri merged 14 commits into
mainfrom
feat/full-object-rolling-checksum

chandra-siri commented May 26, 2026 •
edited
Loading

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy by chandra-siri · Pull Request #17262 · googleapis/google-cloud-python · GitHub

Uh oh!

feat(storage): full object checksum: implement rolling checksum and verification in reads resumption strategy - #17262

Conversation

chandra-siri commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Overview of the Solution

2. What This PR Specifically Does

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chandra-siri commented May 26, 2026 •
edited
Loading