Follow-up to #822, which was insufficient.
The retry tests in PySQLRetryTestsMixin patch urllib3 globally
(HTTPSConnectionPool._get_conn / _validate_conn) and assert the
connection's own request was retried exactly N times. #822 disabled
telemetry on the retry tests' OWN connections, but the flake persisted:
under pytest-xdist, MANY other e2e tests run in the SAME worker process
before the retry tests, and each opens a connection with telemetry
enabled by default. Telemetry is a process-global singleton
(TelemetryClientFactory) with a shared executor and a daemon flush
thread, so those prior tests leave real TelemetryClient instances and
background threads/futures registered in the worker. When a retry test
installs its global urllib3 mock, an in-flight or subsequently-fired
telemetry POST to /telemetry-ext from a *previous* test's client goes
through the same mocked pool and inflates the mock call_count past the
expected value -> flaky AssertionError. CI logs on #825 show
/telemetry-ext retries firing inside the failing retry test even though
that test's own connection had telemetry disabled.
Fix: add an autouse, function-scoped fixture in a new tests/e2e/conftest.py
that force-installs NoopTelemetryClient for every e2e connection (and
drains the factory before/after each test). With no real TelemetryClient
ever created in the worker, there are no background telemetry threads or
futures to leak into any test's urllib3 mock. This removes the flake at
its source deterministically rather than draining racy background threads.
The telemetry suite (tests/e2e/test_telemetry_e2e.py) asserts real
telemetry behavior and is marked @pytest.mark.serial while managing its
own factory state, so the fixture explicitly skips serial-marked tests.
Verified against dogfood:
- A normal connection opened with telemetry defaulting ON gets a
NoopTelemetryClient (no real executor/flush thread).
- serial-marked tests keep the real initialize_telemetry_client;
non-serial tests get the no-op installer.
- test_telemetry_e2e.py still passes (12 tests, real telemetry).
- Full PySQLRetryTestsMixin passes (36 tests).
Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
Follow-up to #822, which was insufficient — the retry tests still flaked on PR #825 run 26943350470 (test_oserror_retries, test_retry_max_count_not_exceeded).
Real root cause
The retry tests in PySQLRetryTestsMixin patch urllib3 globally (HTTPSConnectionPool._get_conn / _validate_conn) and assert the connection's own request was retried exactly N times. #822 disabled telemetry on the retry tests' own connections — but that wasn't enough.
Under pytest-xdist, many other e2e tests run in the same worker process before the retry tests, and each opens a connection with telemetry enabled by default. Telemetry is a process-global singleton (TelemetryClientFactory) with a shared ThreadPoolExecutor and a daemon periodic-flush thread, so those prior tests leave real TelemetryClient instances and background threads/futures registered in the worker. When a retry test installs its global urllib3 mock, an in-flight or subsequently-fired telemetry POST to /telemetry-ext from a previous test's client goes through the same mocked pool and inflates the mock call_count past the expected value → flaky AssertionError.
CI logs on #825 confirm this: /telemetry-ext retries fire inside the failing retry test, even though that test's own connection had enable_telemetry=False. The count came out at 7 instead of 6 — one stray foreign telemetry call.
Fix
Add an autouse, function-scoped fixture in a new tests/e2e/conftest.py that force-installs NoopTelemetryClient for every e2e connection (and drains the factory before/after each test). With no real TelemetryClient ever created in the worker, there are no background telemetry threads or futures to leak into any test's urllib3 mock. This removes the flake at its source deterministically, rather than trying to drain racy background threads (the approach the old _isolated_from_telemetry helper took, which restores foreign state in finally and races).
The telemetry suite (tests/e2e/test_telemetry_e2e.py) asserts real telemetry behavior and is marked @pytest.mark.serial while managing its own factory state, so the fixture explicitly skips serial-marked tests.
Verification (against dogfood)
The retry behavior under test is unchanged; only the telemetry side-channel traffic (from this and every other e2e test) is removed.
This pull request and its description were written by Isaac.