fix: address issues with concurrent BigQuery tests - #3426
Conversation
Sorry, something went wrong.
Gotchas: * Keeping track of which sample tags are in scope at a given variable is a pain. * variables aren't all instantiated at the same spot, so you need to leverage the END/START tag blocks for each variable that gets overriden.
|
I'm here for it, but I tend to have less strong opinions about this than some of the other folks. @tmatsuo has done a LOT of refactoring tests recently, so I want his eyes too to double check |
Sorry, something went wrong.
There was a problem hiding this comment.
Looks good overall.
If the client library doesn't automatically retry transient failures, I'd retry few times upon such exceptions.
I found that backoff is a decent library for this purpose.
The code looks like:
@backoff.on_exception(backoff.expo, max_time=60)
def delete_dataset():
try:
client.delete_dataset(item, delete_contents=True)
except NotFound as e:
# We ignore this case.
print("The dataset doesn't exist: detail: {}".format(str(e)))
delete_dataset()If the client library already retries transient errors, just ignore my comment. Also, I'm open to do that research after merging this PR.
Sorry, something went wrong.
|
|
||
| from google.cloud import bigquery | ||
| import pytest | ||
| import uuid |
There was a problem hiding this comment.
Can you do the following?
import uuid
from google.cloud import bigquery
import pytest
import authorized_view_tutorialUsually imports should be 1) standard libs 2) 3rd party libs and 3) package local libs, separated by an empty line.
Sorry, something went wrong.
Yeah, BQ uses a default retry policy. |
Sorry, something went wrong.
…-docs-samples into refactor_terrible_tests
tmatsuo
left a comment
There was a problem hiding this comment.
Thanks
Sorry, something went wrong.
…oudPlatform/python-docs-samples#3426) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
…oudPlatform/python-docs-samples#3426) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
…oudPlatform/python-docs-samples#3426) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
This PR changes how testing is done for the BQ tests which now run multiple instances concurrently.
Basic premise is we inject new values into the code under test, and use the template syntax to elide the replacement from the embedded representations. the test harness takes care of deleting resources.