Add partition support to BigQuery - #2058
Conversation
Sorry, something went wrong.
| """ | ||
| partitioned = None | ||
| if "timePartitioning" in self._properties: | ||
| partitioned = self._properties.get('timePartitioning').get('type') |
This comment was marked as spam.
|
The docs for creating partitioned tables point to a query for listing partitions, Could we use that to implement `table.list_partitions'? |
Sorry, something went wrong.
| with self.assertRaises(ValueError): | ||
| table.partitioning_type = "HASH" | ||
| with self.assertRaises(ValueError): | ||
| table.partition_expiration = "NEVER" |
This comment was marked as spam.
|
Thank you for the patch! |
Sorry, something went wrong.
|
I've got a method I didn't include for list_partitions that does generate and run that query. What I'm not sure about is the best way to fake a partitioned table for testing said method. |
Sorry, something went wrong.
Assuming the method looks something like: def list_partitions(self):
query = self._client.query(
'SELECT partition_id from [%s.%s$__PARTITIONS_SUMMARY__]' %
(self.dataset_name, self.name))
query.run()
return [row[0] for row in query.rows]I think I would just mock up the query method on the _Client mock to return an instance of a mocked _Query class: it would then have a run method which would populate its rows attribute. |
Sorry, something went wrong.
Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
#2058) * perf: avoid re-authenticating if credentials have already been fetched * Update bigframes/_config/bigquery_options.py * move lock to module
* fix: allow bigframes.options.bigquery.credentials to be `None` This is a partial revert of "perf: avoid re-authenticating if credentials have already been fetched (#2058)", commit 913de1b31f3bb0b306846fddae5dcaff6be3cec4. * add unit test
This adds the ability to create a partitioned table. It would be nice to expose a table.list_partitions() method, but currently there is not an API-based way of returning partitions. So, for now, it simply allows users to interact with the current partitioning API.