#15: Add `storage.batch.Batch` by tseaver · Pull Request #654 · googleapis/google-cloud-python · GitHub

Changes Unknown when pulling 5fed4d2 on tseaver:15-add_storage_batch into * on GoogleCloudPlatform:master*.

@craigcitro, @dhermes I'm willing to do the work to vendor in and use the apitools batch module, if Craig doesn't think it will have a PyPI release soon. I can also work on getting it up-to-speed in terms of test coverage and Py3k straddling in the canonical repository, if that would help.

Should I review this or rely on the vendored in code?

@dhermes we don't have a vendored-in version of that module yet. My earlier question was whether we should go ahead and vendor that code in, given that it has no coverage and does not straddle Py3k yet.

The amount of code in the new storage/batch.py module which would be replaced if we chose to vendor the apitools module is relatively small (135 out of 544 lines):

The MIMEApplicationHTTP class and its tests
The bits of `Batch.finish()' which build, send, and process the response for the multipart request and corresponding tests.

Coverage remained the same at 100.0% when pulling ae0e9cf on tseaver:15-add_storage_batch into 8e68a20 on GoogleCloudPlatform:master.

Coverage remained the same at 100.0% when pulling 6261272 on tseaver:15-add_storage_batch into 8e68a20 on GoogleCloudPlatform:master.

Coverage remained the same at 100.0% when pulling af2a382 on tseaver:15-add_storage_batch into 8e68a20 on GoogleCloudPlatform:master.

A batch proxies a connection, deferring write requests.

In preparation for making 'storage.batch.Batch' derive from Connection, allowing it to override only the actual transmission of the HTTP request.

Drop patching the connection's 'http', as well as proxying its attributes (we get that via subclassing).

@dhermes PTAL.

Rebased on top of master, squashing all previous commits; then implemented subclassing per our discussion today.

@dhermes yes! so here's the current apitools plan:

i'm working on merging the gsutil fork today
once i do that, i'm going to submit to pypi
after that, help getting it py3 ready would be AWESOME
even right now, py3 support for protorpc would be super

The batching interface is not specific to the storage API.

It does not case-normalize header key lookup, but stores headers only as lowercase.

Addresses: #654 (comment)

For posterity (RE: Craig's comment):

Note: Currently, Google Cloud Storage does not support batch operations for
media, either for upload or download.

Coverage remained the same at 100.0% when pulling de6fd48 on tseaver:15-add_storage_batch into 808c38b on GoogleCloudPlatform:master.

I think everything looks good except removing that unused sys import.

LGTM

@craigcitro Can we start a new issue to discuss apitools / protorpc? Should the issue be here or in one (or both) of those repos?

@dhermes yeah, new issue wherever sgtm. maybe a "py3 support" issue in each of protorpc/apitools, and an issue here for "better apitools dependency" blocked by those two?

Coverage remained the same at 100.0% when pulling c828bda on tseaver:15-add_storage_batch into d43c4a0 on GoogleCloudPlatform:master.

#15: Add `storage.batch.Batch`

@tseaver We discussed using the apitools batch support when this went in. We should discuss if / how that's possible (in light of #811).

…#654) This was noticed when attempting to generate Bigtable Admin in a message definition: an imported module is given an alias to prevent collision with a field name. When the module is referenced to describe the type of a singleton field it is properly disambiguated. When used to describe the type of a MapField it is _not_ disambiguated. Fix for that. Closes #618

It turns out this was unnecessary and can cause problems of its own. Fixes #649 #634

Closes #654.

* chore: use gapic-generator-python 0.65.1 PiperOrigin-RevId: 441524537 Source-Link: googleapis/googleapis@2a27391 Source-Link: googleapis/googleapis-gen@ab6756a Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYWI2NzU2YTQ4Yzg5YjViY2I5ZmI3MzQ0M2NiOGU1NWQ1NzRmNDY0MyJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

Adds a SpannerPickleType that can be used as the implementation for the standard SQLAlchemy PickleType. The SpannerPickleType ensures that the binary values are encoded/decoded to/from base64 strings, which is how Spanner stores binary values. Fixes #654

* fix: Django and SQLAlchemy APIs are failing to use rowcount * lint fix Co-authored-by: Vikash Singh <3116482+vi3k6i5@users.noreply.github.com>

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Feb 16, 2015

tseaver mentioned this pull request Feb 16, 2015

Add support for batch requests in gcloud.storage #15

Closed

dhermes reviewed Feb 18, 2015
View reviewed changes

Comment thread gcloud/storage/batch.py Outdated

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

dhermes added the api: storage Issues related to the Cloud Storage API. label Feb 18, 2015

dhermes reviewed Feb 18, 2015
View reviewed changes

Comment thread gcloud/storage/batch.py

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tseaver added 3 commits February 25, 2015 13:28

Add 'storage.batch.Batch'

653fa14

A batch proxies a connection, deferring write requests.

Factor out 'storage.connection.Connection._do_request()'.

63ea7ba

In preparation for making 'storage.batch.Batch' derive from Connection, allowing it to override only the actual transmission of the HTTP request.

Make 'Batch' subclass 'Connection', and override '_do_request'.

dd2046a

Drop patching the connection's 'http', as well as proxying its attributes (we get that via subclassing).

dhermes reviewed Feb 26, 2015
View reviewed changes

Comment thread gcloud/storage/batch.py Outdated

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tseaver added 4 commits February 26, 2015 13:30

Fix URL for batch requests.

b5a8124

The batching interface is not specific to the storage API.

Conform to httplib2.response.

96807c8

It does not case-normalize header key lookup, but stores headers only as lowercase.

Exercise 'storage.batch.Batch' during bucket cleanup.

edeb950

Use six.PY2, rather than sys.version_info comparison.

c083ceb

Addresses: #654 (comment)

Unused import.

c828bda

tseaver added a commit that referenced this pull request Feb 26, 2015

Merge pull request #654 from tseaver/15-add_storage_batch

f0a1157

#15: Add `storage.batch.Batch`

tseaver merged commit f0a1157 into googleapis:master Feb 26, 2015

tseaver deleted the 15-add_storage_batch branch February 26, 2015 19:10

This was referenced Feb 26, 2015

Bring apitools Python3 support. google/apitools#1

Closed

Bring protorpc Python3 support. google/protorpc#3

Closed

dhermes mentioned this pull request Mar 12, 2015

Upgrading version to 0.4.3. #723

Merged

craigcitro mentioned this pull request Apr 7, 2015

DISCUSSION: Switch to apitools as a dependency #755

Closed

parthea pushed a commit that referenced this pull request Nov 24, 2025

fix: json fields dictionary has modification side effect (#654 )

7d94be9

parthea pushed a commit that referenced this pull request Nov 24, 2025

feat: don't flush entire global cache on transient errors (#654 )

afa4fdb

It turns out this was unnecessary and can cause problems of its own. Fixes #649 #634

parthea pushed a commit that referenced this pull request Nov 25, 2025

fix: json fields dictionary has modification side effect (#654 )

24689d8

parthea pushed a commit that referenced this pull request Nov 26, 2025

feat: add Python 3.9 support, drop Python 3.5 support (#655 )

65b79d4

Closes #654.

parthea pushed a commit that referenced this pull request Mar 9, 2026

chore(deps): update dependency google-cloud-storage to v1.43.0 (#654 )

63aa519

parthea pushed a commit that referenced this pull request Mar 9, 2026

fix: Django and SQLAlchemy APIs are failing to use rowcount (#654 )

0251a18

* fix: Django and SQLAlchemy APIs are failing to use rowcount * lint fix Co-authored-by: Vikash Singh <3116482+vi3k6i5@users.noreply.github.com>

#15: Add `storage.batch.Batch` by tseaver · Pull Request #654 · googleapis/google-cloud-python · GitHub

Uh oh!

#15: Add storage.batch.Batch - #654

#15: Add storage.batch.Batch#654tseaver merged 9 commits intogoogleapis:mastergoogleapis/google-cloud-python:masterfrom tseaver:15-add_storage_batchCopy head branch name to clipboard

Conversation

tseaver commented Feb 16, 2015

Uh oh!

coveralls commented Feb 16, 2015

Uh oh!

tseaver commented Feb 16, 2015

Uh oh!

dhermes commented Feb 17, 2015

Uh oh!

tseaver commented Feb 17, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

coveralls commented Feb 18, 2015

Uh oh!

coveralls commented Feb 18, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

coveralls commented Feb 19, 2015

Uh oh!

tseaver commented Feb 25, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

craigcitro commented Feb 26, 2015

Uh oh!

dhermes commented Feb 26, 2015

Uh oh!

coveralls commented Feb 26, 2015

Uh oh!

dhermes commented Feb 26, 2015

Uh oh!

dhermes commented Feb 26, 2015

Uh oh!

craigcitro commented Feb 26, 2015

Uh oh!

coveralls commented Feb 26, 2015

Uh oh!

dhermes commented Apr 9, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

#15: Add storage.batch.Batch#654
tseaver merged 9 commits into
googleapis:masterfrom
tseaver:15-add_storage_batch