FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: removed pkg_resources from all test files and moved importlib i… · googleapis/google-cloud-python@b1f6c9e · GitHub

/ google-cloud-python Public

Commit b1f6c9e

Browse files
authored
feat: removed pkg_resources from all test files and moved importlib into pandas extra (#1726)
* feat: Introduce compatibility with native namespace packages * Update copyright year * removed pkg_resources from all test files and moved importlib into pandas extra * feat: removed pkg_resources from all test files and moved importlib into pandas extra * Adding no cover tag to test code * reformatted with black * undo revert * perf: use the first page a results when `query(api_method="QUERY")` (#1723) * perf: use the first page a results when `query(api_method="QUERY")` * add tests * respect max_results with cached page * respect page_size, also avoid bqstorage if almost fully downloaded * skip true test if bqstorage not installed * coverage * fix: ensure query job retry has longer deadline than API request deadline (#1734) In cases where we can't disambiguate API failure from job failure, this ensures we can still retry the job at least once. * fix: `load_table_from_dataframe` now assumes there may be local null values (#1735) Even if the remote schema is REQUIRED Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/python-bigquery/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Fixes #1692 🦕 * chore: standardize samples directory - delete unneeded dependencies (#1732) * chore: standardize samples directory = delete unneeded dependencies * Removed unused import for linter * fix: move grpc, proto-plus and protobuf packages to extras (#1721) * chore: move grpc, proto-plus and protobuff packages to extras * formatted with black * feat: add `job_timeout_ms` to job configuration classes (#1675) * fix: adds new property and tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * updates docs to correct a sphinx failure * Updates formatting * Update tests/system/test_query.py * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update google/cloud/bigquery/job/base.py * updates one test and uses int_or_none * Update tests/system/test_query.py testing something. * Update tests/system/test_query.py * testing coverage feature * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * minor edits * tweaks to noxfile for testing purposes * add new test to base as experiment * adds a test, updates import statements * add another test * edit to tests * formatting fixes * update noxfile to correct debug code * removes unneeded comments. --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> --------- Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Swast <swast@google.com> * remove unnecessary version checks * undo bad commit, remove unneeded version checks * Revert "undo bad commit, remove unneeded version checks" This reverts commit 5c82dcf85f5fd73ad37ab7805b85b88a65351801. * Revert "remove unnecessary version checks" This reverts commit 9331a7e034de1c651c5ebc454f38f602d612a03d. * revert bad changes, remove pkg_resources from file * after clarification, reimplement changes and ignore 3.12 tests * reformatted with black * removed minimum check * updated pandas installed version check --------- Co-authored-by: Anthonios Partheniou <partheniou@google.com> Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent 13439b3 commit b1f6c9e

12 files changed

Lines changed: 96 additions & 133 deletions

File tree

‎packages/google-cloud-bigquery/.coveragerc‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,3 @@ exclude_lines =
1212
pragma: NO COVER
1313
# Ignore debug-only repr
1414
def __repr__
15-
# Ignore pkg_resources exceptions.
16-
# This is added at the module level as a safeguard for if someone
17-
# generates the code and tries to run it without pip installing. This
18-
# makes it virtually impossible to test properly.
19-
except pkg_resources.DistributionNotFound

‎packages/google-cloud-bigquery/google/__init__.py‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎packages/google-cloud-bigquery/google/cloud/__init__.py‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎packages/google-cloud-bigquery/noxfile.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ def mypy(session):
137137
"types-requests",
138138
"types-setuptools",
139139
)
140-
session.run("mypy", "google/cloud", "--show-traceback")
140+
session.run("mypy", "-p", "google", "--show-traceback")
141141

142142

143143
@nox.session(python=DEFAULT_PYTHON_VERSION)
@@ -149,7 +149,8 @@ def pytype(session):
149149
session.install("attrs==20.3.0")
150150
session.install("-e", ".[all]")
151151
session.install(PYTYPE_VERSION)
152-
session.run("pytype")
152+
# See https://github.com/google/pytype/issues/464
153+
session.run("pytype", "-P", ".", "google/cloud/bigquery")
153154

154155

155156
@nox.session(python=SYSTEM_TEST_PYTHON_VERSIONS)

‎packages/google-cloud-bigquery/setup.py‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@
6262
"pandas>=1.1.0",
6363
pyarrow_dependency,
6464
"db-dtypes>=0.3.0,<2.0.0dev",
65+
"importlib_metadata>=1.0.0; python_version<'3.8'",
6566
],
6667
"ipywidgets": [
6768
"ipywidgets>=7.7.0",
@@ -108,16 +109,10 @@
108109
# benchmarks, etc.
109110
packages = [
110111
package
111-
for package in setuptools.PEP420PackageFinder.find()
112+
for package in setuptools.find_namespace_packages()
112113
if package.startswith("google")
113114
]
114115

115-
# Determine which namespaces are needed.
116-
namespaces = ["google"]
117-
if "google.cloud" in packages:
118-
namespaces.append("google.cloud")
119-
120-
121116
setuptools.setup(
122117
name=name,
123118
version=version,
@@ -143,7 +138,6 @@
143138
],
144139
platforms="Posix; MacOS X; Windows",
145140
packages=packages,
146-
namespace_packages=namespaces,
147141
install_requires=dependencies,
148142
extras_require=extras,
149143
python_requires=">=3.7",

‎packages/google-cloud-bigquery/tests/system/test_pandas.py‎

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,13 @@
2323
import warnings
2424

2525
import google.api_core.retry
26-
import pkg_resources
2726
import pytest
2827

28+
try:
29+
import importlib.metadata as metadata
30+
except ImportError:
31+
import importlib_metadata as metadata
32+
2933
from google.cloud import bigquery
3034

3135
from google.cloud.bigquery import enums
@@ -42,11 +46,9 @@
4246
)
4347

4448
if pandas is not None:
45-
PANDAS_INSTALLED_VERSION = pkg_resources.get_distribution("pandas").parsed_version
49+
PANDAS_INSTALLED_VERSION = metadata.version("pandas")
4650
else:
47-
PANDAS_INSTALLED_VERSION = pkg_resources.parse_version("0.0.0")
48-
49-
PANDAS_INT64_VERSION = pkg_resources.parse_version("1.0.0")
51+
PANDAS_INSTALLED_VERSION = "0.0.0"
5052

5153

5254
class MissingDataError(Exception):
@@ -310,10 +312,7 @@ def test_load_table_from_dataframe_w_automatic_schema(bigquery_client, dataset_i
310312
]
311313

312314

313-
@pytest.mark.skipif(
314-
PANDAS_INSTALLED_VERSION < PANDAS_INT64_VERSION,
315-
reason="Only `pandas version >=1.0.0` is supported",
316-
)
315+
@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
317316
def test_load_table_from_dataframe_w_nullable_int64_datatype(
318317
bigquery_client, dataset_id
319318
):
@@ -342,7 +341,7 @@ def test_load_table_from_dataframe_w_nullable_int64_datatype(
342341

343342

344343
@pytest.mark.skipif(
345-
PANDAS_INSTALLED_VERSION < PANDAS_INT64_VERSION,
344+
PANDAS_INSTALLED_VERSION[0:2].startswith("0."),
346345
reason="Only `pandas version >=1.0.0` is supported",
347346
)
348347
def test_load_table_from_dataframe_w_nullable_int64_datatype_automatic_schema(
@@ -1043,9 +1042,7 @@ def test_list_rows_max_results_w_bqstorage(bigquery_client):
10431042
assert len(dataframe.index) == 100
10441043

10451044

1046-
@pytest.mark.skipif(
1047-
PANDAS_INSTALLED_VERSION >= pkg_resources.parse_version("2.0.0"), reason=""
1048-
)
1045+
@pytest.mark.skipif(PANDAS_INSTALLED_VERSION[0:2] not in ["0.", "1."], reason="")
10491046
@pytest.mark.parametrize(
10501047
("max_results",),
10511048
(

‎packages/google-cloud-bigquery/tests/unit/job/test_query_pandas.py‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717
import json
1818

1919
import mock
20-
import pkg_resources
2120
import pytest
2221

2322

@@ -45,14 +44,19 @@
4544
except (ImportError, AttributeError): # pragma: NO COVER
4645
tqdm = None
4746

47+
try:
48+
import importlib.metadata as metadata
49+
except ImportError:
50+
import importlib_metadata as metadata
51+
4852
from ..helpers import make_connection
4953
from .helpers import _make_client
5054
from .helpers import _make_job_resource
5155

5256
if pandas is not None:
53-
PANDAS_INSTALLED_VERSION = pkg_resources.get_distribution("pandas").parsed_version
57+
PANDAS_INSTALLED_VERSION = metadata.version("pandas")
5458
else:
55-
PANDAS_INSTALLED_VERSION = pkg_resources.parse_version("0.0.0")
59+
PANDAS_INSTALLED_VERSION = "0.0.0"
5660

5761
pandas = pytest.importorskip("pandas")
5862

@@ -656,9 +660,7 @@ def test_to_dataframe_bqstorage_no_pyarrow_compression():
656660
)
657661

658662

659-
@pytest.mark.skipif(
660-
PANDAS_INSTALLED_VERSION >= pkg_resources.parse_version("2.0.0"), reason=""
661-
)
663+
@pytest.mark.skipif(PANDAS_INSTALLED_VERSION[0:2] not in ["0.", "1."], reason="")
662664
@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
663665
def test_to_dataframe_column_dtypes():
664666
from google.cloud.bigquery.job import QueryJob as target_class

‎packages/google-cloud-bigquery/tests/unit/test__pandas_helpers.py‎

Lines changed: 10 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,11 @@
1919
import operator
2020
import queue
2121
import warnings
22-
import pkg_resources
22+
23+
try:
24+
import importlib.metadata as metadata
25+
except ImportError:
26+
import importlib_metadata as metadata
2327

2428
import mock
2529

@@ -57,13 +61,10 @@
5761

5862
bigquery_storage = _versions_helpers.BQ_STORAGE_VERSIONS.try_import()
5963

60-
PANDAS_MINIUM_VERSION = pkg_resources.parse_version("1.0.0")
61-
6264
if pandas is not None:
63-
PANDAS_INSTALLED_VERSION = pkg_resources.get_distribution("pandas").parsed_version
65+
PANDAS_INSTALLED_VERSION = metadata.version("pandas")
6466
else:
65-
# Set to less than MIN version.
66-
PANDAS_INSTALLED_VERSION = pkg_resources.parse_version("0.0.0")
67+
PANDAS_INSTALLED_VERSION = "0.0.0"
6768

6869

6970
skip_if_no_bignumeric = pytest.mark.skipif(
@@ -542,9 +543,7 @@ def test_bq_to_arrow_array_w_nullable_scalars(module_under_test, bq_type, rows):
542543
],
543544
)
544545
@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
545-
@pytest.mark.skipif(
546-
PANDAS_INSTALLED_VERSION >= pkg_resources.parse_version("2.0.0"), reason=""
547-
)
546+
@pytest.mark.skipif(PANDAS_INSTALLED_VERSION[0:2] not in ["0.", "1."], reason="")
548547
@pytest.mark.skipif(isinstance(pyarrow, mock.Mock), reason="Requires `pyarrow`")
549548
def test_bq_to_arrow_array_w_pandas_timestamp(module_under_test, bq_type, rows):
550549
rows = [pandas.Timestamp(row) for row in rows]
@@ -806,10 +805,7 @@ def test_list_columns_and_indexes_with_named_index_same_as_column_name(
806805
assert columns_and_indexes == expected
807806

808807

809-
@pytest.mark.skipif(
810-
pandas is None or PANDAS_INSTALLED_VERSION < PANDAS_MINIUM_VERSION,
811-
reason="Requires `pandas version >= 1.0.0` which introduces pandas.NA",
812-
)
808+
@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
813809
def test_dataframe_to_json_generator(module_under_test):
814810
utcnow = datetime.datetime.utcnow()
815811
df_data = collections.OrderedDict(
@@ -837,16 +833,8 @@ def test_dataframe_to_json_generator(module_under_test):
837833
assert list(rows) == expected
838834

839835

836+
@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
840837
def test_dataframe_to_json_generator_repeated_field(module_under_test):
841-
pytest.importorskip(
842-
"pandas",
843-
minversion=str(PANDAS_MINIUM_VERSION),
844-
reason=(
845-
f"Requires `pandas version >= {PANDAS_MINIUM_VERSION}` "
846-
"which introduces pandas.NA"
847-
),
848-
)
849-
850838
df_data = [
851839
collections.OrderedDict(
852840
[("repeated_col", [pandas.NA, 2, None, 4]), ("not_repeated_col", "first")]

‎packages/google-cloud-bigquery/tests/unit/test_client.py‎

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,11 @@
3030
import requests
3131
import packaging
3232
import pytest
33-
import pkg_resources
33+
34+
try:
35+
import importlib.metadata as metadata
36+
except ImportError:
37+
import importlib_metadata as metadata
3438

3539
try:
3640
import pandas
@@ -76,13 +80,10 @@
7680
from test_utils.imports import maybe_fail_import
7781
from tests.unit.helpers import make_connection
7882

79-
PANDAS_MINIUM_VERSION = pkg_resources.parse_version("1.0.0")
80-
8183
if pandas is not None:
82-
PANDAS_INSTALLED_VERSION = pkg_resources.get_distribution("pandas").parsed_version
84+
PANDAS_INSTALLED_VERSION = metadata.version("pandas")
8385
else:
84-
# Set to less than MIN version.
85-
PANDAS_INSTALLED_VERSION = pkg_resources.parse_version("0.0.0")
86+
PANDAS_INSTALLED_VERSION = "0.0.0"
8687

8788

8889
def _make_credentials():
@@ -8145,10 +8146,7 @@ def test_load_table_from_dataframe_unknown_table(self):
81458146
timeout=DEFAULT_TIMEOUT,
81468147
)
81478148

8148-
@unittest.skipIf(
8149-
pandas is None or PANDAS_INSTALLED_VERSION < PANDAS_MINIUM_VERSION,
8150-
"Only `pandas version >=1.0.0` supported",
8151-
)
8149+
@unittest.skipIf(pandas is None, "Requires `pandas`")
81528150
@unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
81538151
def test_load_table_from_dataframe_w_nullable_int64_datatype(self):
81548152
from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
@@ -8193,10 +8191,7 @@ def test_load_table_from_dataframe_w_nullable_int64_datatype(self):
81938191
SchemaField("x", "INT64", "NULLABLE", None),
81948192
)
81958193

8196-
@unittest.skipIf(
8197-
pandas is None or PANDAS_INSTALLED_VERSION < PANDAS_MINIUM_VERSION,
8198-
"Only `pandas version >=1.0.0` supported",
8199-
)
8194+
@unittest.skipIf(pandas is None, "Requires `pandas`")
82008195
# @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
82018196
def test_load_table_from_dataframe_w_nullable_int64_datatype_automatic_schema(self):
82028197
from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES

‎packages/google-cloud-bigquery/tests/unit/test_packaging.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import os
16+
import subprocess
17+
import sys
18+
19+
20+
def test_namespace_package_compat(tmp_path):
21+
# The ``google`` namespace package should not be masked
22+
# by the presence of ``google-cloud-bigquery``.
23+
google = tmp_path / "google"
24+
google.mkdir()
25+
google.joinpath("othermod.py").write_text("")
26+
env = dict(os.environ, PYTHONPATH=str(tmp_path))
27+
cmd = [sys.executable, "-m", "google.othermod"]
28+
subprocess.check_call(cmd, env=env)
29+
30+
# The ``google.cloud`` namespace package should not be masked
31+
# by the presence of ``google-cloud-bigquery``.
32+
google_cloud = tmp_path / "google" / "cloud"
33+
google_cloud.mkdir()
34+
google_cloud.joinpath("othermod.py").write_text("")
35+
env = dict(os.environ, PYTHONPATH=str(tmp_path))
36+
cmd = [sys.executable, "-m", "google.cloud.othermod"]
37+
subprocess.check_call(cmd, env=env)

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL