FIX: Request SQL_CHAR as SQL_C_WCHAR in arrow fetch path - #575
Merged
Subrata (subrata-ms) merged 12 commits intoJun 23, 2026
Merged
FIX: Request SQL_CHAR as SQL_C_WCHAR in arrow fetch path#575Subrata (subrata-ms) merged 12 commits into
Subrata (subrata-ms) merged 12 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Arrow fetch path in the C++ pybind layer to always request SQL_CHAR/SQL_VARCHAR data as SQL_C_WCHAR (UTF-16) so Arrow results are correct regardless of server/client codepage, locale, or platform—addressing the VARCHAR non-ASCII decoding issues reported in #553.
Changes:
- Switch Arrow batch binding/fetching for SQL_CHAR/SQL_VARCHAR from SQL_C_CHAR to SQL_C_WCHAR to avoid codepage-dependent decoding.
- Remove the narrow-char copy path for SQL_CHAR/SQL_VARCHAR in Arrow batch production and route through the existing wide-char → UTF-8 conversion logic.
- Add an Arrow regression test covering Unicode round-tripping through a UTF-8-collated VARCHAR column.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| mssql_python/pybind/ddbc_bindings.cpp | Changes Arrow batch binding and fetch handling so VARCHAR is requested as SQL_C_WCHAR, ensuring consistent Unicode correctness. |
| tests/test_004_cursor_arrow.py | Adds a regression test to validate Arrow output for Unicode data stored in VARCHAR with UTF-8 collation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sorry, something went wrong.
Comment thread
tests/test_004_cursor_arrow.py
Outdated
Copy link
Copy Markdown
Contributor
|
/azp run |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sorry, something went wrong.
gargsaumya
reviewed
May 27, 2026
Comment thread
mssql_python/pybind/ddbc_bindings.cpp
Outdated
Comment thread
mssql_python/pybind/ddbc_bindings.cpp
Outdated
Comment thread
tests/test_004_cursor_arrow.py
Comment thread
tests/test_004_cursor_arrow.py
Outdated
Comment thread
mssql_python/pybind/ddbc_bindings.cpp
Outdated
Subrata (subrata-ms)
requested changes
Jun 10, 2026
Subrata (subrata-ms)
approved these changes
Jun 23, 2026
Comment thread
mssql_python/pybind/ddbc_bindings.cpp
Dismissed
Copy link
Copy Markdown
Contributor
|
/azp run |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sorry, something went wrong.
gargsaumya
approved these changes
Jun 23, 2026
Copy link
Copy Markdown
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changes
Summary
mssql_python/pybind/ddbc_bindings.cppLines 4629-4639 4629
4630 SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules,
4631 int arrowBatchSize,
4632 int charCtype) {
! 4633 // Fetch narrow char data as SQL_C_CHAR if on Linux/macOS and configured by the user
! 4634 charCtype = EffectiveCharCtypeForFetch(charCtype, "utf-8");
! 4635
4636 // An overly large fetch size doesn't seem to help performance
4637 int fetchSize = 64;
4638
4639 SQLRETURN ret;📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.pybind.ddbc_bindings.cpp: 76.4%
mssql_python.row.py: 76.9%
mssql_python.__init__.py: 77.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.logging.py: 85.5%
mssql_python.connection.py: 85.6%🔗 Quick Links
|
Sorry, something went wrong.
Merged
gargsaumya
added a commit
that referenced
this pull request
Jun 26, 2026
[AB#45983](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/45983) Release mssql-python v1.10.0. This PR bumps the driver version from 1.9.0 to 1.10.0 across `mssql_python/__init__.py`, `setup.py`, and updates `PyPI_Description.md` with this release's customer-facing changes. ### Summary **Enhancements** - **Active Directory Service Principal support for Bulk Copy** ([#576](#576)) - `bulkcopy` now supports `Authentication=ActiveDirectoryServicePrincipal`, registering a token-provider callback that resolves the tenant from the STS URL mid-handshake and returns a service-principal JWT. **Bug Fixes** - **Non-ASCII VARCHAR data in the Arrow fetch path** ([#575](#575)) - The Arrow fetch path now requests `SQL_CHAR` data as `SQL_C_WCHAR` (UTF-16LE), ensuring correct decoding regardless of encoding settings, locale, or OS. Thanks @ffelixg for the contribution! - **Bulk load connection timeouts** ([#641](#641)) - Fixed connection timeouts during bulk load operations. Fix lands in the Rust core via bumping `mssql_py_core` from 0.1.4 to 0.1.5. *(via `mssql_py_core`)* **Version bump** - `mssql_python/__init__.py`: `__version__ = "1.10.0"` - `setup.py`: `version="1.10.0"` - `PyPI_Description.md`: updated `## What's new in v1.10.0` section
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Due to #495, we can now request SQL_CHAR data as SQL_C_WCHAR, i.e. utf16le strings. Doing this for the arrow path ensures that arrow methods always return correct data no matter the encoding settings / locale / operating system. There does not seem to be any significant negative performance impact.