FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Loading AVRO files from local filesystem. by tswast · Pull Request #3427 · googleapis/google-cloud-python · GitHub

/ google-cloud-python Public

Loading AVRO files from local filesystem. - #3427

Merged
tseaver merged 4 commits into
googleapis:masterfrom
tswast:bq-3416
May 16, 2017
ConversationCommits (4)ChecksFiles changed
Merged

Loading AVRO files from local filesystem.#3427
tseaver merged 4 commits into
googleapis:masterfrom
tswast:bq-3416

Conversation

tswast commented May 16, 2017

Copy link
Copy Markdown
Contributor

Adds an integration test for loading AVRO files.

Don't set the schema if none is provided (patch).

Closes #3416.

tswast added 3 commits May 16, 2017 13:01
Add test avro data.
c27100c 
Generated with `avro-tools`

    $ avro-tools fromjson \
          --schema-file tests/data/colors.avroschema \
          tests/data/colors.json > tests/data/colors.avro

See:
http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
Add test to reproduce googleapis#3416.
dcb05b3
Don't set schema in load job if none is provided.
2b1f149
tswast requested a review from tseaver May 16, 2017 21:00
googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 16, 2017

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast Can you push your branch to this repository, so that the system test actually gets run by the CI before we merge?

dhermes commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast If you don't have push privileges one of us can do it

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

Also, @tweeter0830 reported that the patch did not fix the issue with uploading AVRO data.

tswast commented May 16, 2017

Copy link
Copy Markdown
Contributor Author

I saw that, but I was not able to reproduce the issue after the patch.

tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. labels May 16, 2017

tswast commented May 16, 2017

Copy link
Copy Markdown
Contributor Author

Hmm, I see that this change made the Avro load test pass, but now the CSV load test fails.
https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1920

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast That is weird, as the testcase clearly sets the schema on the table: it should be passing the schema along.

tswast commented May 16, 2017

Copy link
Copy Markdown
Contributor Author

We missed the "fields" property of the schema object in the patch. Seems to be running fine now. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1922

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

I've pushed the branch here (as tswast-bq-3146) to get the system tests running under CI before merging.

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

BQ system tests are green on Circle CI.

tseaver merged commit 4877315 into googleapis:master May 16, 2017
tswast deleted the bq-3416 branch May 16, 2017 23:14
tseaver mentioned this pull request Jun 26, 2017
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

tseaver Awaiting requested review from tseaver

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL