This repository was archived by the owner on Apr 1, 2026. It is now read-only.
feat: mutate rows batching - #770
Merged
daniel-sanche merged 159 commits intoJun 26, 2023
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
I added timeout configuration to the batcher init signature. Previously, it would always use the table defaults. This is part of the public API, so just wanted to make sure you saw the change in case you have thoughts |
Sorry, something went wrong.
daniel-sanche
commented
Jun 23, 2023
Comment thread
tests/unit/test_mutations_batcher.py
Comment thread
google/cloud/bigtable/mutations_batcher.py
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
| 20 exceptions have been encountered since the last flush, only the | ||
| first and last 10 exceptions will be stored. | ||
| """ | ||
| return self._oldest_exceptions + self._newest_exceptions |
Copy link
Copy Markdown
Contributor
Author
There was a problem hiding this comment.
Do we want to make the staged exception list public? Or maybe just make a has_errors function?
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
not sure what you mean
Sorry, something went wrong.
Comment thread
google/cloud/bigtable/client.py
Outdated
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
Comment thread
google/cloud/bigtable/mutations_batcher.py
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
| 20 exceptions have been encountered since the last flush, only the | ||
| first and last 10 exceptions will be stored. | ||
| """ | ||
| return self._oldest_exceptions + self._newest_exceptions |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
not sure what you mean
Sorry, something went wrong.
Comment thread
google/cloud/bigtable/mutations_batcher.py
| timer_task = asyncio.create_task(timer_routine(self, interval)) | ||
| return timer_task | ||
|
|
||
| async def append(self, mutation_entry: RowMutationEntry): |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Please add a TODO to return a future with the result of this mutation
Sorry, something went wrong.
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
| in_process_requests.append(batch_task) | ||
| # wait for all inflight requests to complete | ||
| found_exceptions = await self._wait_for_batch_results(*in_process_requests) | ||
| # allow previous flush tasks to finalize before adding new exceptions to list |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
is this comment stale?
Sorry, something went wrong.
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
Comment thread
google/cloud/bigtable/mutations_batcher.py
Outdated
mutianf
reviewed
Jun 26, 2023
Comment thread
google/cloud/bigtable/mutations_batcher.py
igorbernstein2
approved these changes
Jun 26, 2023
daniel-sanche
added a commit
that referenced
this pull request
Feb 5, 2024
* feat: add new v3.0.0 API skeleton (#745) * feat: improve rows filters (#751) * feat: read rows query model class (#752) * feat: implement row and cell model classes (#753) * feat: add pooled grpc transport (#748) * feat: implement read_rows (#762) * feat: implement mutate rows (#769) * feat: literal value filter (#767) * feat: row_exists and read_row (#778) * feat: read_modify_write and check_and_mutate_row (#780) * feat: sharded read rows (#766) * feat: ping and warm with metadata (#810) * feat: mutate rows batching (#770) * chore: restructure module paths (#816) * feat: improve timeout structure (#819) * fix: api errors apply to all bulk mutations * chore: reduce public api surface (#820) * feat: improve error group tracebacks on < py11 (#825) * feat: optimize read_rows (#852) * chore: add user agent suffix (#842) * feat: optimize retries (#854) * feat: add test proxy (#836) * chore(tests): add conformance tests to CI for v3 (#870) * chore(tests): turn off fast fail for conformance tets (#882) * feat: add TABLE_DEFAULTS enum for table method arguments (#880) * fix: pass None for retry in gapic calls (#881) * feat: replace internal dictionaries with protos in gapic calls (#875) * chore: optimize gapic calls (#863) * feat: expose retryable error codes to users (#879) * chore: update api_core submodule (#897) * chore: merge main into experimental_v3 (#900) * chore: pin conformance tests to v0.0.2 (#903) * fix: bulk mutation eventual success (#909) --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Reviewers
2 more reviewers3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocked on #769. This PR builds off of changes from this branch
Implements MutationsBatcher to submit mutations in the background.