refactor: update insert_rows.py for more efficient streaming by jmgreger · Pull Request #212 · googleapis/python-bigquery · GitHub

Added in the use of selected_fields=schema to explicitly define the schema. This prevents excessive calling of table metadata when streaming large volumes of data.

There may be a better way to showcase this in the code, but I wanted to put this out there for review.

Fixes #211 🦕

Please rebase to remove the release-related changes from this PR (the first three commits). Oops, my bad: the issue is that your PR targets the release-v1.27.0 branch, instead of master.

Please rebase to remove the release-related changes from this PR (the first three commits). Oops, my bad: the issue is that your PR targets the release-v1.27.0 branch, instead of master.

Ahh, sorry about that. I don't submit code very often :/. Should I make a new PR against master instead?

I re-based this off of the master branch, now it just needs the files added by the chore bot to be removed. Doesn't look like I have rights to do that though.

@jmgreger If you use git rebase -i in your local clone, you should be able to delete the bot-generated commits. E.g.:

$ git checkout master
$ git fetch --all upstream  #  assuming this repo's remote is named 'upstream'.
$ git pull upstream/master
$ git checkout patch-1
$ git rebase -i master  # delete first three commits and the last merge commit
$ git push -f origin patch-1

As an alternative, you could just cherry-pick the patch from this commit into a new branch against current master, and then submit a new PR from that branch.

Please keep the sample simple. This is unnecessary branching. If anything, we can call out that the schema can be fetched from a table in a comment.

No need to catch this ValueError. Just let the exception bubble up.

LGTM once Tim's comments are addressed and as long as tests pass

Added in the use of selected_fields=schema to explicitly define the schema. This prevents excessive calling of table metadata when streaming large volumes of data. There may be a better way to showcase this in the code, but I wanted to put this out there for review.

…eapis#208) Co-authored-by: Tres Seaver <tseaver@palladion.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com>

Closes googleapis#217.

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

Co-authored-by: Tim Swast <swast@google.com>

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

Closing this in favor of #253 as per comment in that PR.

jmgreger requested a review from shollyman as a code owner August 3, 2020 16:39

google-cla Bot added the cla: yes This human has signed the Contributor License Agreement. label Aug 3, 2020

HemangChothani changed the title Update insert_rows.py for more efficient streaming refactor: update insert_rows.py for more efficient streaming Aug 4, 2020

plamut added the type: docs Improvement to the documentation for an API. label Aug 4, 2020

release-please Bot force-pushed the release-v1.27.0 branch from dd0f632 to b842fb5 Compare August 4, 2020 16:39

release-please Bot requested a review from a team as a code owner August 4, 2020 16:39

release-please Bot requested review from leahecole and removed request for a team August 4, 2020 16:39

tseaver suggested changes Aug 5, 2020

View reviewed changes

tseaver added do not merge Indicates a pull request not ready for merge, due to either quality or timing. and removed type: docs Improvement to the documentation for an API. labels Aug 5, 2020

jmgreger changed the base branch from release-v1.27.0 to master August 11, 2020 02:33

jmgreger requested a review from tseaver August 11, 2020 02:36

product-auto-label Bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Aug 21, 2020

tswast suggested changes Aug 24, 2020

View reviewed changes

jmgreger and others added 4 commits August 31, 2020 15:52

testing: split system tests into separate builds (googleapis#207 )

9ceeafc

chore(deps): pin ipython to v7.17.0 for samples, python >= 3.7 (googl…

023b513

…eapis#208) Co-authored-by: Tres Seaver <tseaver@palladion.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com>

chore: fix docs build broken by Sphinx 3.2.0 (googleapis#218 )

970d839

Closes googleapis#217.

jmgreger force-pushed the patch-1 branch from c856136 to 970d839 Compare August 31, 2020 16:03

google-cla Bot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Aug 31, 2020

Update samples/table_insert_rows.py

3ee4572

Co-authored-by: Tim Swast <swast@google.com>

jmgreger mentioned this pull request Aug 31, 2020

refactor: Update insert_rows.py for more efficient streaming jmgreger/python-bigquery#1

Closed

jmgreger mentioned this pull request Aug 31, 2020

Update table_insert_rows.py #253

Closed

plamut closed this Sep 1, 2020

+                  client.insert_rows(table_id, selected_fields=schema, rows_to_insert)  # Stream data to BQ
+                  print("New rows have been added.")
+              except ValueError:
+                  print("Table’s schema is not set or rows is not a Sequence.")

refactor: update insert_rows.py for more efficient streaming by jmgreger · Pull Request #212 · googleapis/python-bigquery · GitHub

Uh oh!

refactor: update insert_rows.py for more efficient streaming - #212

refactor: update insert_rows.py for more efficient streaming#212
jmgreger wants to merge 5 commits into
googleapis:masterfrom
jmgreger:patch-1

jmgreger commented Aug 3, 2020

Uh oh!

tseaver left a comment •
edited
Loading

Uh oh!

Uh oh!

jmgreger commented Aug 11, 2020

Uh oh!

jmgreger commented Aug 11, 2020

Uh oh!

tseaver commented Aug 11, 2020

Uh oh!

tswast Aug 24, 2020

Uh oh!

Uh oh!

tswast Aug 24, 2020

Uh oh!

leahecole commented Aug 24, 2020

Uh oh!

google-cla Bot commented Aug 31, 2020

Uh oh!

google-cla Bot commented Aug 31, 2020

Uh oh!

plamut commented Sep 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

+              rows_to_insert = [(u"Phred Phlyntstone", 32), (u"Wylma Phlyntstone", 29)] #populate data for entry
+              try:
+                  if 'schema' not in globals():

refactor: update insert_rows.py for more efficient streaming by jmgreger · Pull Request #212 · googleapis/python-bigquery · GitHub

Uh oh!

refactor: update insert_rows.py for more efficient streaming - #212

refactor: update insert_rows.py for more efficient streaming#212jmgreger wants to merge 5 commits intogoogleapis:mastergoogleapis/python-bigquery:masterfrom jmgreger:patch-1jmgreger/python-bigquery:patch-1Copy head branch name to clipboard

Conversation

jmgreger commented Aug 3, 2020

Uh oh!

tseaver left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmgreger commented Aug 11, 2020

Uh oh!

jmgreger commented Aug 11, 2020

Uh oh!

tseaver commented Aug 11, 2020

Uh oh!

tswast Aug 24, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tswast Aug 24, 2020

Choose a reason for hiding this comment

Uh oh!

leahecole commented Aug 24, 2020

Uh oh!

google-cla Bot commented Aug 31, 2020

Uh oh!

google-cla Bot commented Aug 31, 2020

Uh oh!

plamut commented Sep 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

refactor: update insert_rows.py for more efficient streaming#212
jmgreger wants to merge 5 commits into
googleapis:masterfrom
jmgreger:patch-1

tseaver left a comment •
edited
Loading