FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

chore(bigtable): improve system test instance usage by daniel-sanche · Pull Request #18149 · googleapis/google-cloud-python · GitHub

/ google-cloud-python Public

chore(bigtable): improve system test instance usage #18149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
daniel-sanche wants to merge 6 commits into
base: main
Choose a base branch
Could not load branches
Branch not found: {{ refName }}

Are you sure you want to change the base?

Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
from
Conversation 7 Commits 6 Checks 43 Files changed
Open

chore(bigtable): improve system test instance usage #18149

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
47e4e4f
don't clear old instances for emulator tests
daniel-sanche Aug 18, 2026
620b484
added logging and cap to stale instance deletion
daniel-sanche Aug 18, 2026
5498bd3
added project override to system_tests_emulated
daniel-sanche Aug 18, 2026
78d20fa
re-use existing instances where possible
daniel-sanche Aug 18, 2026
79c69e3
Update packages/google-cloud-bigtable/tests/system/utils.py
daniel-sanche Aug 18, 2026
4af7095
improved logs
daniel-sanche Aug 19, 2026
File filter

Filter by extension

Filter by extension .py  (6) All 1 file type selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
1 change: 1 addition & 0 deletions packages/google-cloud-bigtable/noxfile.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,7 @@ def system_emulated(session):

hostport = "localhost:8789"
session.env["BIGTABLE_EMULATOR_HOST"] = hostport
session.env["GOOGLE_CLOUD_PROJECT"] = "emulated-test-project"

p = subprocess.Popen(
["gcloud", "beta", "emulators", "bigtable", "start", "--host-port", hostport]
Expand Down
19 changes: 11 additions & 8 deletions packages/google-cloud-bigtable/tests/system/admin_overlay/test_system_async.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -347,14 +347,17 @@ async def test_optimize_restored_table(
admin_v2.StorageType.HDD,
)

instance_to_restore, _ = await create_instance(
instance_admin_client,
table_admin_client,
data_client,
admin_overlay_project_id,
instances_to_delete,
second_instance_storage_type,
)
if second_instance_storage_type == admin_v2.StorageType.HDD:
instance_to_restore = instance_with_backup
else:
instance_to_restore, _ = await create_instance(
instance_admin_client,
table_admin_client,
data_client,
admin_overlay_project_id,
instances_to_delete,
second_instance_storage_type,
)

backup = await create_backup(
instance_admin_client,
Expand Down
19 changes: 11 additions & 8 deletions packages/google-cloud-bigtable/tests/system/admin_overlay/test_system_autogen.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -259,14 +259,17 @@ def test_optimize_restored_table(
instances_to_delete,
admin_v2.StorageType.HDD,
)
instance_to_restore, _ = create_instance(
instance_admin_client,
table_admin_client,
data_client,
admin_overlay_project_id,
instances_to_delete,
second_instance_storage_type,
)
if second_instance_storage_type == admin_v2.StorageType.HDD:
instance_to_restore = instance_with_backup
else:
instance_to_restore, _ = create_instance(
instance_admin_client,
table_admin_client,
data_client,
admin_overlay_project_id,
instances_to_delete,
second_instance_storage_type,
)
backup = create_backup(
instance_admin_client,
table_admin_client,
Expand Down
4 changes: 4 additions & 0 deletions packages/google-cloud-bigtable/tests/system/data/__init__.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,12 @@ def cleanup_old_instances(self, project_id):
"""
Automatically deletes any test instances older than 1 day.
"""
from google.cloud.environment_vars import BIGTABLE_EMULATOR
from tests.system.utils import clear_stale_instances

if os.getenv(BIGTABLE_EMULATOR):
return

clear_stale_instances(project_id, "python-bigtable-tests", older_than_days=1)

@pytest.fixture(scope="session")
Expand Down
50 changes: 43 additions & 7 deletions packages/google-cloud-bigtable/tests/system/utils.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,40 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import os
from datetime import datetime, timedelta, timezone

from typing import Union, Tuple

from google.api_core.exceptions import NotFound
from google.cloud.environment_vars import BIGTABLE_EMULATOR

from google.cloud import bigtable_admin_v2 as admin_v2


def clear_stale_instances(project_id: str, prefix: str, older_than_days: int = 1):
def clear_stale_instances(
project_id: str,
prefix: Union[str, Tuple[str, ...]] = (
"python-bigtable-tests",
"g-c-p",
"admin-overlay-instance",
),
older_than_days: int = 1,
max_deletions: int = 5,
):
"""
Synchronously deletes any instances in the given project that are older
than older_than_days and whose name or display name matches the given prefix.
"""
if os.getenv(BIGTABLE_EMULATOR):
return
print(f"Clearing stale instances in project {project_id}...")
client = admin_v2.BigtableInstanceAdminClient(
client_options={"quota_project_id": project_id}
)
parent = client.common_project_path(project_id)
next_page_token = ""
deleted_count = 0

while True:
try:
Expand All @@ -46,12 +63,31 @@ def clear_stale_instances(project_id: str, prefix: str, older_than_days: int = 1

if display_name_matches or name_matches:
if instance.create_time:
now = datetime.now(timezone.utc)
if now - instance.create_time > timedelta(days=older_than_days):
try:
client.delete_instance(name=instance.name)
except NotFound:
pass
try:

mutianf Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Are instances cleaned up properly in the tests? Over time, would this cause instances to slowly increase on the test project? and eventually we could run into SSD node or HDD node quota and will need to clean up the test project manually

daniel-sanche Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Instances are expected to clean themselves up in a post-test step when tests are finished. But there previously were (and maybe still are, and maybe will be in the future) resource leaks in certain cases, causing instances to build up slowly in the test project over time over time.

This function is trying to identify > day old test instances, and clear them, as an extra defense against this build-up. But ideally, it shouldn't be needed
create_time = instance.create_time
if hasattr(create_time, "to_datetime"):
create_time = create_time.to_datetime()
if create_time.tzinfo is None:
create_time = create_time.replace(tzinfo=timezone.utc)
now = datetime.now(timezone.utc)
if now - create_time > timedelta(days=older_than_days):
try:
print(f"Deleting stale instance: {instance.name}")
client.delete_instance(name=instance.name)
deleted_count += 1
if deleted_count >= max_deletions:
print(
f"Reached cap of {max_deletions} stale instance deletions"
)
return
except NotFound:
pass
except Exception as e:
print(
f"Failed to delete stale instance {instance.name}: {e}"
)
except Exception as e:
print(f"Failed to check age for instance {instance.name}: {e}")

next_page_token = response.next_page_token
if not next_page_token:
Expand Down
25 changes: 7 additions & 18 deletions packages/google-cloud-bigtable/tests/system/v2_client/test_table_admin.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -334,23 +334,12 @@ def test_table_backup(
assert restored_table in tables
restored_table.delete()

# Testing `Backup.restore()` into a different instance:
# Setting up another instance...
alt_instance_id = f"gcp-alt-{unique_suffix}"
alt_cluster_id = f"{alt_instance_id}-cluster"
alt_instance = admin_client.instance(alt_instance_id, labels=instance_labels)
alt_cluster = alt_instance.cluster(
cluster_id=alt_cluster_id,
location_id=location_id,
serve_nodes=1,
# Testing `Backup.restore()`:

mutianf Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

did this change the behavior? it was testing restoring to a different instance before. Now it's just backing up to a different table? on the same instance?

daniel-sanche Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Yes, this is a slightly different test case now.

Looking at the code, the client isn't really aware of whether the passed in is is new or old when building a request. So we're not really exercising any different client-side logic here, and it didn't feel worth consuming an extra Instance create/delete for this.

But if you think this is important to cover, we can leave it as-is
restored_table_id_2 = "test-backup-table-restored-2"
restore_op = temp_backup.restore(
restored_table_id_2, data_instance_populated.instance_id
)
create_op = alt_instance.create(clusters=[alt_cluster])
instances_to_delete.append(alt_instance)
create_op.result(timeout=240)

# Testing `restore()`...
restore_op = temp_backup.restore(restored_table_id, alt_instance_id)
restore_op.result(timeout=240)
restored_table = alt_instance.table(restored_table_id)
assert restored_table in alt_instance.list_tables()
restored_table.delete()
restored_table_2 = data_instance_populated.table(restored_table_id_2)
assert restored_table_2 in data_instance_populated.list_tables()
restored_table_2.delete()
Loading
Back | FazBrowse Home | New Git URL