FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: implement native asyncio support via Cross-Sync (#1509) · googleapis/google-cloud-python@f822fd7 · GitHub

feat: implement native asyncio support via Cross-Sync (#1509) · googleapis/google-cloud-python@f822fd7 · GitHub
Skip to content

Navigation Menu

Sign in
Sign in
Sign up

Commit f822fd7

Browse files
sinhasubham
authored
feat: implement native asyncio support via Cross-Sync (#1509)
****Description: Native Asyncio Support**** This PR introduces comprehensive, native asyncio support to the google-cloud-spanner library. It transitions the library into a "Cross-Sync" architecture, where the asynchronous implementation serves as the source of truth, and the synchronous implementation is automatically kept in parity. ****Key Technical Changes**** Core Library Porting * Asynchronous API: Introduced AsyncClient, AsyncInstance, and AsyncDatabase classes. * Session Management: Completely refactored pool.py to support asyncio. Replaced threading.Lock and queue.Queue with their asyncio counterparts via the CrossSync abstraction. * Transactions & Snapshots: Native async implementation of run_in_transaction, including robust retry logic for Aborted exceptions and proper lock management. ****Verification & Testing**** 9 New System Tests: Created a dedicated async system test suite (tests/system/_async/) covering: Rich data types (Timestamp, JSON, Protobuf, etc.) Transaction retry loops Partitioned DML operations Session pool lifecycle 100% Pass Rate: All new async tests and existing sync tests pass reliably against the Spanner Emulator. Mock Server Updates: Added 40+ mock server tests to verify specific async behaviors like result-set iteration and error handling.
1 parent 00d5f8e commit f822fd7

208 files changed

Lines changed: 30345 additions & 4243 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/google-cloud-spanner/.agents/workflows/verify-asyncio.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
description: How to verify a Spanner Asyncio launch is ready.
3+
---
4+
# Spanner Asyncio Launch Verification Workflow
5+
6+
This workflow provides the necessary steps to verify that the Spanner Asyncio implementation is correct, stable, and ready for launch.
7+
8+
## 1. Run Async Unit Tests
9+
Run the complete suite of asynchronous unit tests across all supported Python versions.
10+
```bash
11+
nox -s unit
12+
```
13+
Ensure that all tests in `tests/unit/_async/` pass.
14+
15+
## 2. Run Async System Tests
16+
Verify the asynchronous behavior against the Spanner Emulator.
17+
// turbo
18+
```bash
19+
export SPANNER_EMULATOR_HOST="localhost:9010"
20+
export GCLOUD_PROJECT="emulator-test-project"
21+
export GOOGLE_CLOUD_TESTS_CREATE_SPANNER_INSTANCE="true"
22+
nox -s system -- tests/system/_async
23+
```
24+
**Note**: Ensure `pytest-asyncio` is installed in the system test environment.
25+
26+
## 3. Verify Sync/Async Parity
27+
Run the cross-sync generation tool and ensure no regressions in the synchronous codebase.
28+
```bash
29+
python3 .cross_sync/generate.py
30+
nox -s unit-3.14
31+
nox -s system-3.14
32+
```
33+
34+
## 4. Check for Coroutine Leaks
35+
Ensure all asynchronous GAPIC calls are properly awaited. Search for any unawaited coroutines in the `_async` directory.
36+
```bash
37+
grep -r "await " google/cloud/spanner_v1/_async | grep -v "async def"
38+
```
39+
40+
## 5. Verify Sample Code
41+
Verify that the provided samples work correctly.
42+
```bash
43+
python3 samples/async_samples.py
44+
```

‎packages/google-cloud-spanner/.coveragerc‎

Lines changed: 0 additions & 40 deletions
This file was deleted.

‎packages/google-cloud-spanner/.cross_sync/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# CrossSync
2+
3+
CrossSync provides a simple way to share logic between async and sync code.
4+
It is made up of a small library that provides:
5+
1. a set of shims that provide a shared sync/async API surface
6+
2. annotations that are used to guide generation of a sync version from an async class
7+
8+
Using CrossSync, the async code is treated as the source of truth, and sync code is generated from it.
9+
10+
## Usage
11+
12+
### CrossSync Shims
13+
14+
Many Asyncio components have direct, 1:1 threaded counterparts for use in non-asyncio code. CrossSync
15+
provides a compatibility layer that works with both
16+
17+
| CrossSync | Asyncio Version | Sync Version |
18+
| --- | --- | --- |
19+
| CrossSync.Queue | asyncio.Queue | queue.Queue |
20+
| CrossSync.Condition | asyncio.Condition | threading.Condition |
21+
| CrossSync.Future | asyncio.Future | Concurrent.futures.Future |
22+
| CrossSync.Task | asyncio.Task | Concurrent.futures.Future |
23+
| CrossSync.Event | asyncio.Event | threading.Event |
24+
| CrossSync.Semaphore | asyncio.Semaphore | threading.Semaphore |
25+
| CrossSync.Awaitable | typing.Awaitable | typing.Union (no-op type) |
26+
| CrossSync.Iterable | typing.AsyncIterable | typing.Iterable |
27+
| CrossSync.Iterator | typing.AsyncIterator | typing.Iterator |
28+
| CrossSync.Generator | typing.AsyncGenerator | typing.Generator |
29+
| CrossSync.Retry | google.api_core.retry.AsyncRetry | google.api_core.retry.Retry |
30+
| CrossSync.StopIteration | StopAsyncIteration | StopIteration |
31+
| CrossSync.Mock | unittest.mock.AsyncMock | unittest.mock.Mock |
32+
33+
Custom aliases can be added using `CrossSync.add_mapping(class, name)`
34+
35+
Additionally, CrossSync provides method implementations that work equivalently in async and sync code:
36+
- `CrossSync.sleep()`
37+
- `CrossSync.gather_partials()`
38+
- `CrossSync.wait()`
39+
- `CrossSync.condition_wait()`
40+
- `CrossSync.event_wait()`
41+
- `CrossSync.create_task()`
42+
- `CrossSync.retry_target()`
43+
- `CrossSync.retry_target_stream()`
44+
45+
### Annotations
46+
47+
CrossSync provides a set of annotations to mark up async classes, to guide the generation of sync code.
48+
49+
- `@CrossSync.convert_sync`
50+
- marks classes for conversion. Unmarked classes will be copied as-is
51+
- if add_mapping is included, the async and sync classes can be accessed using a shared CrossSync.X alias
52+
- `@CrossSync.convert`
53+
- marks async functions for conversion. Unmarked methods will be copied as-is
54+
- `@CrossSync.drop`
55+
- marks functions or classes that should not be included in sync output
56+
- `@CrossSync.pytest`
57+
- marks test functions. Test functions automatically have all async keywords stripped (i.e., rm_aio is unneeded)
58+
- `CrossSync.add_mapping`
59+
- manually registers a new CrossSync.X alias, for custom types
60+
- `CrossSync.rm_aio`
61+
- Marks regions of the code that include asyncio keywords that should be stripped during generation
62+
63+
### Code Generation
64+
65+
Generation can be initiated using `nox -s generate_sync`
66+
from the root of the project. This will find all classes with the `__CROSS_SYNC_OUTPUT__ = "path/to/output"`
67+
annotation, and generate a sync version of classes marked with `@CrossSync.convert_sync` at the output path.
68+
69+
There is a unit test at `tests/unit/data/test_sync_up_to_date.py` that verifies that the generated code is up to date
70+
71+
## Architecture
72+
73+
CrossSync is made up of two parts:
74+
- the runtime shims and annotations live in `/google/cloud/aio/_cross_sync`
75+
- the code generation logic lives in `/.cross_sync/` in the repo root

‎packages/google-cloud-spanner/.cross_sync/generate.py‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
from __future__ import annotations
15+
from typing import Sequence
16+
import ast
17+
"""
18+
Entrypoint for initiating an async -> sync conversion using CrossSync
19+
20+
Finds all python files rooted in a given directory, and uses
21+
transformers.CrossSyncFileProcessor to handle any files marked with
22+
__CROSS_SYNC_OUTPUT__
23+
"""
24+
25+
26+
def extract_header_comments(file_path) -> str:
27+
"""
28+
Extract the file header. Header is defined as the top-level
29+
comments before any code or imports
30+
"""
31+
header = []
32+
with open(file_path, "r", encoding="utf-8-sig") as f:
33+
for line in f:
34+
if line.startswith("#") or line.strip() == "":
35+
header.append(line)
36+
else:
37+
break
38+
header.append("\n# This file is automatically generated by CrossSync. Do not edit manually.\n\n")
39+
return "".join(header)
40+
41+
42+
class CrossSyncOutputFile:
43+
44+
def __init__(self, output_path: str, ast_tree, header: str | None = None):
45+
self.output_path = output_path
46+
self.tree = ast_tree
47+
self.header = header or ""
48+
49+
def render(self, with_formatter=True, save_to_disk: bool = True) -> str:
50+
"""
51+
Render the file to a string, and optionally save to disk
52+
53+
Args:
54+
with_formatter: whether to run the output through black before returning
55+
save_to_disk: whether to write the output to the file path
56+
"""
57+
full_str = self.header + ast.unparse(self.tree)
58+
if with_formatter:
59+
import black # type: ignore
60+
import autoflake # type: ignore
61+
62+
full_str = black.format_str(
63+
autoflake.fix_code(full_str, remove_all_unused_imports=True),
64+
mode=black.FileMode(),
65+
)
66+
if save_to_disk:
67+
import os
68+
os.makedirs(os.path.dirname(self.output_path), exist_ok=True)
69+
with open(self.output_path, "w") as f:
70+
f.write(full_str)
71+
return full_str
72+
73+
74+
def convert_files_in_dir(directory: str) -> set[CrossSyncOutputFile]:
75+
import glob
76+
from transformers import CrossSyncFileProcessor
77+
78+
# find all python files in the directory
79+
files = glob.glob(directory + "/**/*.py", recursive=True)
80+
# keep track of the output files pointed to by the annotated classes
81+
artifacts: set[CrossSyncOutputFile] = set()
82+
file_transformer = CrossSyncFileProcessor()
83+
# run each file through ast transformation to find all annotated classes
84+
for file_path in files:
85+
with open(file_path, encoding="utf-8-sig") as f:
86+
ast_tree = ast.parse(f.read())
87+
output_path = file_transformer.get_output_path(ast_tree)
88+
if output_path is not None:
89+
# contains __CROSS_SYNC_OUTPUT__ annotation
90+
converted_tree = file_transformer.visit(ast_tree)
91+
header = extract_header_comments(file_path)
92+
artifacts.add(CrossSyncOutputFile(output_path, converted_tree, header))
93+
# return set of output artifacts
94+
return artifacts
95+
96+
97+
def save_artifacts(artifacts: Sequence[CrossSyncOutputFile]):
98+
for a in artifacts:
99+
a.render(save_to_disk=True)
100+
101+
102+
if __name__ == "__main__":
103+
import sys
104+
105+
if len(sys.argv) < 2:
106+
print("Usage: python .cross_sync/generate.py <directory>")
107+
sys.exit(1)
108+
109+
search_root = sys.argv[1]
110+
outputs = convert_files_in_dir(search_root)
111+
print(f"Generated {len(outputs)} artifacts: {[a.output_path for a in outputs]}")
112+
save_artifacts(outputs)

0 commit comments

Comments
 (0)

Footer

© 2026 GitHub, Inc.
Back | FazBrowse Home | New Git URL