Jobs: load from storage by tseaver · Pull Request #1056 · googleapis/google-cloud-python · GitHub

Add support for jobs loading table data from CloudStorage files.

Does this have a diffbase?

Does this have a diffbase?

Nope -- I rebased it before pushing. Too big?

I just wasn't sure since there were 8 commits. Also, you're failing pep8 (that one runs fast, pylint is the slow one).

Also ISTM a rebase won't feel very good due to the merging of #1051.

I will fix it up (both the pep8 and the date stuff) and re-push.

Thanks. Ping me when you push.

Ugh! The pep8 fix was trivial, but working around the pylint 'too-many-branches' failures made me grind my teeth (4c14890)

This thing is all ready to review?

Hold off a bit -- I'm adding a system test for the "load from cloud storage" case, and it is failing.

@dhermes OK, got the system test added and passing.

Found while attempting to add a system test for the 'load from storage' case.

Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37027413

Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37028059

Addresses: #1056 (comment)

Users who want to avoid tripping over string literal mismatches can use class constants instead. Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37029533

- 'Client.load_from_storage' -> 'Client.load_table_from_storage'. - 'job.LoadFromStorageJob' -> 'job.LoadTableFromStorageJob'. Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37120652

Why not just take the jump and use enum34?

I didn't want to eat the extra dependency. We have #1031 to track the idea, though.

Why not? It's lightweight and hopefully most users already have it

>>> import enum
>>> 
>>> class RedBlue(enum.Enum):
...     RED = 'RED'
...     BLUE = 'BLUE'
... 
>>> 
>>> to_validate = 'RED'
>>> rb = RedBlue(to_validate)
>>> print(rb)
RedBlue.RED
>>> print(rb.value)
RED
>>> 
>>> to_validate = 'GREEN'
>>> try:
...     RedBlue(to_validate)
... except ValueError as exc:
...     print(repr(exc))
...     print('%s was not validated' % (to_validate,))
... 
ValueError('GREEN is not a valid RedBlue',)
GREEN was not validated

Most users are on Python2, and likely won't have it unless some other library has pushed them to add it. Again, we can clean up here (and in the other APIs) when we address #1031.

LGTM. I really hope the follow-up commits can spread out some of the pain here.

I was saying that most Python 2 only users would have lots of libraries installed that straddled Py2 and Py3, and hence would have a good chance that one of them used enum34. (I could've been clearer there.)

@tseaver Please ping me on anything that will get rebased after this goes in.

Jobs: load from storage

Fragment tests are defined by a small proto file describing an API surface with characteristics such that it is desirable to test the generated surface for correctness or to prevent regressions. As part of a fragment test, the generator is run on a fragment to create a GAPIC library for the fragment. The generated unit tests for the fragment are then executed to test the surface.

🤖 I have created a release \*beep\* \*boop\* --- ## [0.55.0](https://www.github.com/googleapis/gapic-generator-python/compare/v0.54.0...v0.55.0) (2021-11-01) ### Features * add fragment tests ([#1056](https://www.github.com/googleapis/gapic-generator-python/issues/1056)) ([9d9b33d](https://www.github.com/googleapis/gapic-generator-python/commit/9d9b33dadf587a6d0b09031edeea597d6d2eae62)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Fixes #1056. Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

* chore: update Java and Python dependencies PiperOrigin-RevId: 408420890 Source-Link: googleapis/googleapis@2921f9f Source-Link: googleapis/googleapis-gen@6598ca8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNjU5OGNhOGNiYmY1MjI2NzMzYTA5OWM0NTA2NTE4YTVhZjZmZjc0YyJ9 * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…dding interceptors support for testing (#1056) * feat: Fixing and refactoring transaction retry logic in dbapi. Also adding interceptors support for testing * Comments incorporated and changes for also storing Cursor object with the statements details added for retry * Some refactoring of transaction_helper.py and maintaining state of rows update count for batch dml in cursor * Small fix * Maintaining a map from cursor to last statement added in transaction_helper.py * Rolling back the transaction when Aborted exception is thrown from interceptor * Small change * Disabling a test for emulator run * Reformatting

tseaver added the api: bigquery Issues related to the BigQuery API. label Aug 12, 2015

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 12, 2015

tseaver mentioned this pull request Aug 12, 2015

Initial BQ system tests #1047

Merged

tseaver added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 13, 2015

tseaver removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 13, 2015

dhermes reviewed Aug 13, 2015
View reviewed changes

tseaver added 11 commits August 17, 2015 16:24

Add 'LoadFromStorageJob.begin' API method.

cd1fdba

Add 'LoadFromStorageJob.exists' API method.

66dd746

Add 'LoadFromStorageJob.reload' API method.

4da9bb5

Accomodate datetime centralization from #1051 .

15d8663

Include 'sourceUris', 'destination' in load configuration.

e37ab0b

Found while attempting to add a system test for the 'load from storage' case.

Add system test for 'load from storage' case.

f88dcc3

Add docstring.

31026aa

Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37027413

Use a longer var name for temporary file.

31c4b1b

Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37028059

Implement 'cancel' API request.

6549219

Addresses: #1056 (comment)

Expose quasi-enum types for string values w/ choices.

9260ae9

Users who want to avoid tripping over string literal mismatches can use class constants instead. Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37029533

Make load job name / API more explicit:

3e10e52

- 'Client.load_from_storage' -> 'Client.load_table_from_storage'. - 'job.LoadFromStorageJob' -> 'job.LoadTableFromStorageJob'. Addresses: https://github.com/GoogleCloudPlatform/gcloud-python/pull/1056/files#r37120652

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Aug 17, 2015

tseaver added a commit that referenced this pull request Aug 17, 2015

Merge pull request #1056 from tseaver/bigquery-jobs_load_from_storage

cf93465

Jobs: load from storage

tseaver merged commit cf93465 into googleapis:master Aug 17, 2015

tseaver deleted the bigquery-jobs_load_from_storage branch August 17, 2015 22:34

tseaver mentioned this pull request Aug 18, 2015

Fix broken system test #1070

Merged

dhermes mentioned this pull request Sep 30, 2015

Upgrading version to 0.8.0. #1158

Merged

parthea pushed a commit that referenced this pull request Nov 24, 2025

fix: Fix partitioning by DATE column (#1074 )

b2502a4

Fixes #1056. Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

parthea pushed a commit that referenced this pull request Nov 24, 2025

chore: add java 21 to fix emulator tests (#1056 )

ebe7bad

parthea pushed a commit that referenced this pull request Mar 9, 2026

chore(main): release 2.10.0 (#1056 )

975eb4d

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

Jobs: load from storage by tseaver · Pull Request #1056 · googleapis/google-cloud-python · GitHub

Uh oh!

Jobs: load from storage - #1056

Jobs: load from storage#1056tseaver merged 16 commits intogoogleapis:mastergoogleapis/google-cloud-python:masterfrom tseaver:bigquery-jobs_load_from_storageCopy head branch name to clipboard

Conversation

tseaver commented Aug 12, 2015

Uh oh!

dhermes commented Aug 12, 2015

Uh oh!

tseaver commented Aug 12, 2015

Uh oh!

dhermes commented Aug 12, 2015

Uh oh!

dhermes commented Aug 12, 2015

Uh oh!

tseaver commented Aug 12, 2015

Uh oh!

dhermes commented Aug 12, 2015

Uh oh!

tseaver commented Aug 12, 2015

Uh oh!

dhermes commented Aug 12, 2015

Uh oh!

tseaver commented Aug 13, 2015

Uh oh!

tseaver commented Aug 13, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

tseaver commented Aug 17, 2015

Uh oh!

dhermes commented Aug 17, 2015

Uh oh!

tseaver commented Aug 17, 2015

Uh oh!

dhermes commented Aug 17, 2015

Uh oh!

dhermes commented Aug 17, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jobs: load from storage#1056
tseaver merged 16 commits into
googleapis:masterfrom
tseaver:bigquery-jobs_load_from_storage