FEAT: Support for Apache Arrow in Bulk Copy API by gargsaumya · Pull Request #665 · microsoft/mssql-python · GitHub

Work Item / Issue Reference

AB#46268

GitHub Issue: #<ISSUE_NUMBER>

Summary

This pull request introduces a new high-performance bulkcopy_arrow method to the Cursor class for bulk loading data directly from Apache Arrow sources, along with several related improvements and refactorings. It also updates type stubs and documentation to reflect the new API, and improves bulk copy authentication handling by refactoring shared logic.

New feature: Arrow-based bulk copy

Added Cursor.bulkcopy_arrow(table_name, source) for efficient bulk loading from Arrow sources (e.g., pyarrow.Table, RecordBatch, objects exposing the Arrow C Data Interface). This avoids unnecessary Python row materialization and is significantly faster for Arrow-native data. [1] [2]
The classic bulkcopy() method now raises a TypeError if given Arrow-shaped data, steering users to the new method.

Codebase refactoring and improvements

Refactored connection context and Azure AD authentication logic into a shared _build_pycore_context() helper, used by both bulkcopy and bulkcopy_arrow. This ensures consistent authentication handling and reduces code duplication. [1] [2]
Updated type stubs (mssql_python.pyi) to include the new bulkcopy_arrow method and improved the typing for bulkcopy. [1] [2]

Documentation

Updated the CHANGELOG.md to document the new Arrow bulk copy feature and its requirements.

Minor improvements

Minor code formatting improvement in a test file (test_bulkcopy_udt_geometry).

Adds cursor.bulkcopy_arrow(table_name, source) to bulk-load directly from Apache Arrow data (pyarrow Table/RecordBatch/RecordBatchReader, __arrow_c_stream__/__arrow_c_array__ producers, or an iterable of batches) via the mssql_py_core Rust core, streaming through the Arrow C Data Interface with the GIL released during transfer. bulkcopy() now raises TypeError to steer Arrow inputs here. Auth setup is refactored into a shared _build_pycore_context() helper (preserves SQL, pre-acquired-token, and ServicePrincipal factory paths). Adds tests/test_024_bulkcopy_arrow.py (36 unit tests at 100% coverage of the new code + 16 live round-trip/type-matrix tests) and benchmarks/bench_bulkcopy_arrow.py. Requires mssql-py-core 0.1.5+.

Pull request overview

Adds Apache Arrow support to the Bulk Copy API by introducing a dedicated Cursor.bulkcopy_arrow() path and refactoring shared connection/auth context construction so both tuple-based and Arrow-based bulk copy use consistent handling.

Changes:

Added Cursor.bulkcopy_arrow(table_name, source) and Arrow-source detection/steering to prevent Arrow inputs from going through tuple bulkcopy.
Refactored bulk-copy connection/auth parsing into Cursor._build_pycore_context() for reuse across bulk copy entry points.
Added comprehensive tests and a benchmark comparing tuple vs Arrow bulk copy throughput.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_024_bulkcopy_arrow.py	Adds unit + live-DB integration coverage for Arrow bulk copy, auth context building, and Arrow-source detection.
mssql_python/mssql_python.pyi	Updates public type stubs to include bulkcopy and new bulkcopy_arrow API.
mssql_python/cursor.py	Implements _build_pycore_context, Arrow source detection/steering, and the new bulkcopy_arrow method.
CHANGELOG.md	Documents the new Arrow bulk copy feature and related requirements/behavior changes.
benchmarks/bench_bulkcopy_arrow.py	Adds a benchmark script comparing tuple-based vs Arrow-based bulk copy performance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…sign D13) Remove the keyword-only '*' from Cursor.bulkcopy_arrow (cursor.py + stub) so batch_size/timeout match bulkcopy's signature per the finalized design (D13/section 2). Also set cursor._timeout in the test mock-cursor helper (main's #650 added self._timeout to the shared _build_pycore_context) and add a positional-args regression test.

…al (List[str]) semantics

…w_c_array__); use non-localhost host in unit-test conn strings

…s, memory, MB/s, CSV)

…oney/xml/legacy-datetime/date64/fixed-binary

# Conflicts: # tests/test_019_bulkcopy.py

…nd args

…tion

📊 Code Coverage Report

🔥 Diff Coverage 100%	🎯 Overall Coverage 81%	📈 Total Lines Covered: 7272 out of 8885 📁 Project: mssql-python

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

mssql_python/cursor.py (100%)

Summary

Total: 88 lines
Missing: 0 lines
Coverage: 100%

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.2%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 82.7%
mssql_python.pybind.connection.connection.cpp: 83.7%
mssql_python.connection.py: 84.7%

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

no major problems or blockers found - just some improvements to code structure and a nit miss on error message, will approve once done

…uard str/bytes source, consistent bare re-raise

After merging the Arrow bulk copy feature (#665), _build_pycore_context now reads self.connection._token_provider before the _auth_type dispatch. The test_024 _cursor_with_conn helper builds a bare MagicMock connection, which auto-vivifies a truthy _token_provider and routed every case through the custom-credential token path -> InterfaceError (12 failures in CI). Real Connection objects always initialize _token_provider to None; set it explicitly on the mock so it is faithful.

gargsaumya Please be aware that the release on top of a dev/nightly build of py-core is going to fail. There are checks in the pipeline to prevent such a thing from happening :)
Please create a release of mssql-py-core (now that you have tested with a dev build), and change the version in this file to 0.1.8

### Work Item / Issue Reference > [AB#47087](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47087) ------------------------------------------------------------------- ### Summary Release mssql-python v1.13.0. Version bump to 1.13.0. Updates `mssql_python/__init__.py`, `setup.py`, `PyPI_Description.md`, and the README "Important Note" section. Bundled `mssql_py_core` bumped from 0.1.7 to 0.1.8 (no source/API changes — dev-nightly to stable pin). #### Enhancements - **ODBC driver ships exclusively via `mssql-python-odbc` (Phase 2)** — The `libs/` fallback introduced in v1.12.0 has been removed. `mssql-python` now hard-depends on `mssql-python-odbc==18.6.2.1`; `pip install mssql-python` still pulls the driver package transparently. Smaller wheels; driver binaries managed independently (#693). - **Apache Arrow bulk copy** — New `Cursor.bulkcopy_arrow(table_name, source)` method for high-performance bulk loading from `pyarrow.Table` / `RecordBatch` / Arrow C Data Interface sources; classic `bulkcopy()` now raises `TypeError` for Arrow inputs and steers users to the new method (#665). - **`token_provider=` parameter for Azure Identity credentials** — `connect()` accepts any credential with a `.get_token(scope)` method (`DefaultAzureCredential`, `AzureCliCredential`, `ManagedIdentityCredential`, …). Mutually exclusive with `Authentication=` in the connection string (#603, issue #577). - **Identity-aware connection pooling with token-expiry refresh** — Pool now keys on security context, preventing cross-identity connection leaks; token acquisition deferred to pool-misses; connections with near-expiry tokens refreshed automatically (#660, issues #651, #659). #### Bug Fixes - **Silent zero-row `executemany` batches on late NULLs** — Fixed numeric array parameter binding paths (`TINYINT`/`SMALLINT`/`INT`/`FLOAT`) that left indicator slots uninitialized when a NULL appeared partway through the batch (#702, issue #670). - **`SQL_WVARCHAR` output converter applied as catch-all to non-string columns** — Fallback now gated on `str`/`bytes` mapped types (#692, issue #691). - **Integer-keyed output converters silently never fired** — `add_output_converter(SQL_DECIMAL, ...)` and other integer SQL type code keys now dispatch correctly (pyodbc parity) (#690, issue #684). - **`RecordBatchReader.Close()` for Arrow result sets** — `Cursor.arrow_reader()` now returns a wrapper whose `.close()` releases server-side resources and leaves the parent cursor usable (#644, issue #643). - **`AttributeError` in `Cursor.__del__` on partially-initialized cursor** — `__init__` sets `closed`/`hstmt` before any raise; `__del__` uses correct `sys.is_finalizing()` guard (#646, issue #642). #### Version Bump - `mssql_python/__init__.py`: `__version__ = "1.13.0"` - `setup.py`: `version="1.13.0"` - `PyPI_Description.md`: `## What's new in v1.13.0` section refreshed - `README.md`: "Important Note" updated for Phase 2 (no more `libs/` fallback, `mssql-python-odbc==18.6.2.1`)

resolves the cursor.py conflict from #665, which moved the shared bulkcopy validation into _bulkcopy_core_and_validate. the timeout fix now lives in that single helper, so it covers bulkcopy and bulkcopy_arrow at once. also updates test_024's timeout_non_positive, which asserted the old behaviour, and adds _token_provider to the new mock cursor for the #603 token_provider path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 8, 2026 07:38

github-actions Bot added the pr-size: large Substantial code update label Jul 8, 2026

Copilot started reviewing on behalf of gargsaumya July 8, 2026 07:38 View session

github-advanced-security Bot found potential problems Jul 8, 2026

View reviewed changes

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread mssql_python/cursor.py

Comment thread mssql_python/cursor.py Outdated

gargsaumya marked this pull request as draft July 8, 2026 07:47

gargsaumya added 2 commits July 14, 2026 15:14

Merge remote-tracking branch 'origin/main' into saumya/gh-570

886a30c

github-advanced-security Bot found potential problems Jul 14, 2026

View reviewed changes

Comment thread tests/test_024_bulkcopy_arrow.py Fixed

gargsaumya added 9 commits July 17, 2026 09:43

test: rename mislabeled Arrow column-mapping test to reflect position…

ebf5a5c

…al (List[str]) semantics

fix(bulkcopy): correct Arrow-steering message (tuples/Row, add __arro…

a0bf20f

…w_c_array__); use non-localhost host in unit-test conn strings

bench: extensive bulkcopy vs bulkcopy_arrow benchmark (profiles, stat…

d20c1bb

…s, memory, MB/s, CSV)

Add Arrow bulkcopy tests: GUID, by-name mapping, non-finite floats, m…

3b8ab88

…oney/xml/legacy-datetime/date64/fixed-binary

Update bulkcopy benchmark and test_019 bulkcopy tests

b4d4775

Merge remote-tracking branch 'origin/main' into saumya/gh-570

d733cc5

# Conflicts: # tests/test_019_bulkcopy.py

Re-raise original exception in bulkcopy_arrow to preserve traceback a…

0598678

…nd args

Apply black formatting (line-length 100)

a156ae4

Pin mssql-py-core to 0.1.8-dev nightly for Arrow end-to-end CI valida…

44ef88b

…tion

Merge branch 'main' into saumya/gh-570

a883562

Sumit Sarabhai (sumitmsft) previously approved these changes Jul 30, 2026

View reviewed changes

Gaurav Sharma (bewithgaurav) requested changes Jul 30, 2026

View reviewed changes

Comment thread mssql_python/cursor.py Outdated

Comment thread mssql_python/cursor.py Outdated

Address review: dedupe bulkcopy/bulkcopy_arrow into shared helpers, g…

e6574d6

…uard str/bytes source, consistent bare re-raise

gargsaumya dismissed Sumit Sarabhai (sumitmsft)’s stale review via e6574d6 August 3, 2026 10:04

Merge branch 'main' into saumya/gh-570

24f32f5

gargsaumya requested a review from Gaurav Sharma (bewithgaurav) August 4, 2026 03:25

Gaurav Sharma (bewithgaurav) mentioned this pull request Aug 4, 2026

bulkcopy API rejects timeout=0 despite spec and py-core both defining it as "no timeout" #697

Open

Gaurav Sharma (bewithgaurav) approved these changes Aug 4, 2026

View reviewed changes

gargsaumya marked this pull request as ready for review August 4, 2026 08:45

Sumit Sarabhai (sumitmsft) approved these changes Aug 4, 2026

View reviewed changes

Merge branch 'main' into saumya/gh-570

c7151b8

gargsaumya merged commit bd9ec45 into main Aug 4, 2026
37 checks passed

Saurabh Singh (saurabh500) deleted the saumya/gh-570 branch August 4, 2026 23:13

Saurabh Singh (saurabh500) reviewed Aug 4, 2026

View reviewed changes

Subrata (subrata-ms) mentioned this pull request Aug 6, 2026

RELEASE:1.13.0 #707

Merged

Sam Debruyn (sdebruyn) mentioned this pull request Aug 7, 2026

mssql: use native Arrow bulk copy for Parquet load jobs sdebruyn/dlt#13

Closed

Sam Debruyn (sdebruyn) mentioned this pull request Aug 13, 2026

mssql: load parquet with mssql-python's native Arrow bulk copy instead of ADBC dlt-hub/dlt#4356

Open

FEAT: Support for Apache Arrow in Bulk Copy API by gargsaumya · Pull Request #665 · microsoft/mssql-python · GitHub

Uh oh!

FEAT: Support for Apache Arrow in Bulk Copy API - #665

FEAT: Support for Apache Arrow in Bulk Copy API#665
gargsaumya merged 16 commits into
mainfrom
saumya/gh-570

gargsaumya commented Jul 8, 2026 •
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026 •
edited
Loading

Uh oh!

🔥 Diff Coverage

100%

🎯 Overall Coverage

81%

Uh oh!

Gaurav Sharma (bewithgaurav) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Saurabh Singh (saurabh500) Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		@@ -1 +1 @@
		0.1.7 No newline at end of file
		0.1.8-dev.20260729.163613 No newline at end of file

FEAT: Support for Apache Arrow in Bulk Copy API by gargsaumya · Pull Request #665 · microsoft/mssql-python · GitHub

Uh oh!

FEAT: Support for Apache Arrow in Bulk Copy API - #665

FEAT: Support for Apache Arrow in Bulk Copy API#665gargsaumya merged 16 commits intomainmicrosoft/mssql-python:mainfrom saumya/gh-570microsoft/mssql-python:saumya/gh-570Copy head branch name to clipboard

Conversation

gargsaumya commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Work Item / Issue Reference

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Code Coverage Report

🔥 Diff Coverage

100%

🎯 Overall Coverage

81%

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

Summary

📋 Files Needing Attention

🔗 Quick Links

Uh oh!

Gaurav Sharma (bewithgaurav) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Saurabh Singh (saurabh500) Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

FEAT: Support for Apache Arrow in Bulk Copy API#665
gargsaumya merged 16 commits into
mainfrom
saumya/gh-570

gargsaumya commented Jul 8, 2026 •
edited
Loading

github-actions Bot commented Jul 29, 2026 •
edited
Loading