PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics by bewithgaurav · Pull Request #528 · microsoft/mssql-python · GitHub

Work Item / Issue Reference

AB#44166

Related GitHub Issue: #500

Summary

This pull request introduces performance benchmarking infrastructure improvements and optimizations to the mssql_python driver, as well as documentation and workflow updates to support a new PERF: pull request prefix. The most significant changes are grouped below.

Driver Performance Optimizations:

Implemented a lightweight _soft_reset_cursor in cursor.py to reuse prepared statement handles, avoiding unnecessary SQLPrepare calls when executing the same SQL repeatedly. This improves performance for repeated statement execution. [1] [2] [3] [4]
Optimized parameter conversion logic to skip redundant conversions when re-executing the same SQL with the same parameter style, further reducing overhead.
Added a new DDBCSQLResetStmt binding in the C++ layer to support the lightweight reset operation, and exposed it to Python. [1] [2]

📊 Code Coverage Report

🔥 Diff Coverage 78%	🎯 Overall Coverage 79%	📈 Total Lines Covered: 6793 out of 8552 📁 Project: mssql-python

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

mssql_python/cursor.py (80.0%): Missing lines 763,765-767,1388
mssql_python/pybind/ddbc_bindings.cpp (76.0%): Missing lines 1383-1384,1386-1387,1389-1390
mssql_python/pybind/ddbc_bindings.h (100%)

Summary

Total: 51 lines
Missing: 11 lines
Coverage: 78%

mssql_python/cursor.py

Lines 759-771

  759         if self.hstmt:
  760             ret = ddbc_bindings.DDBCSQLResetStmt(self.hstmt)
  761             try:
  762                 check_error(ddbc_sql_const.SQL_HANDLE_STMT.value, self.hstmt, ret)
! 763             except Exception:
  764                 logger.warning("_soft_reset_cursor failed; falling back to full reset")
! 765                 self._reset_cursor()
! 766                 self.last_executed_stmt = ""
! 767                 return
  768         self._clear_rownumber()
  769 
  770     def close(self) -> None:
  771         """

Lines 1384-1392

  1384         if reset_cursor:
  1385             if self.hstmt:
  1386                 self._soft_reset_cursor()
  1387             else:
! 1388                 self._reset_cursor()
  1389         else:
  1390             # Close just the ODBC cursor (not the statement handle) so the
  1391             # prepared plan can be reused.  SQLFreeStmt(SQL_CLOSE) releases
  1392             # the cursor associated with hstmt without destroying the

mssql_python/pybind/ddbc_bindings.cpp

Lines 1379-1394

  1379 }
  1380 
  1381 SQLRETURN SQLResetStmt_wrap(SqlHandlePtr statementHandle) {
  1382     if (!statementHandle || !statementHandle->get()) {
! 1383         return SQL_INVALID_HANDLE;
! 1384     }
  1385     if (statementHandle->isImplicitlyFreed()) {
! 1386         return SQL_INVALID_HANDLE;
! 1387     }
  1388     if (!SQLFreeStmt_ptr) {
! 1389         DriverLoader::getInstance().loadDriver();
! 1390     }
  1391     SQLHANDLE hStmt = statementHandle->get();
  1392 
  1393     SQLRETURN rc;
  1394     {

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 67.9%
mssql_python.row.py: 70.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 74.6%
mssql_python.pybind.connection.connection.cpp: 75.8%
mssql_python.__init__.py: 77.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 79.6%
mssql_python.connection.py: 85.3%

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

Pull request overview

This PR adds a lightweight statement “reset” primitive to the pybind ODBC layer and uses it in Cursor.execute() to reduce overhead on the execute hot path (avoid full HSTMT reallocation, skip redundant parameter-style conversion on re-execution, and reduce diagnostic-record collection work).

Changes:

Add DDBCSQLResetStmt pybind export to close the cursor + reset parameter bindings without freeing the HSTMT.
Update Cursor.execute() to use _soft_reset_cursor() and introduce simple prepare caching + reduced conversion/log/diagnostic overhead.
Adjust the perf benchmark script to strip Driver= from the connection string for mssql-python when present.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
mssql_python/pybind/ddbc_bindings.cpp	Adds SQLResetStmt_wrap and exports DDBCSQLResetStmt to Python.
mssql_python/cursor.py	Uses soft reset + prepare caching and optimizes parameter conversion/logging/diagnostics in execute().
benchmarks/perf-benchmarking.py	Normalizes conn string for mssql-python by removing Driver= when present.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add _soft_reset_cursor: SQL_CLOSE + SQL_RESET_PARAMS instead of full HSTMT free/realloc on each execute() call - Add DDBCSQLResetStmt C++ wrapper exposing lightweight reset via pybind11 - Skip SQLPrepare when re-executing the same SQL (prepare caching) - Skip detect_and_convert_parameters on repeated same-SQL calls - Guard DDBCSQLGetAllDiagRecords behind SQL_SUCCESS_WITH_INFO check - Guard per-parameter debug logging behind logger.isEnabledFor(DEBUG) - Fix benchmark script to strip Driver= from mssql-python connection string

### Work Item / Issue Reference > [AB#45159](https://sqlclientdrivers.visualstudio.com/mssql-python/_sprints/taskboard/mssql-python%20Team/mssql-python/Rubidium/May%202026?workitem=45159) ------------------------------------------------------------------- ### Summary **Enhancements** - #548 — manylinux_2_28 build targets for RHEL 8 / glibc 2.28 - #542 — macOS universal2 wheel for Python 3.10 - #526 — UTF-16 string handling via simdutf - #528 — Optimized execute() hot path - #567 — Azure Linux installation docs **Bug Fixes** - #562 — Login failures now raise mssql_python exception instead of RuntimeError - #568 — GIL released during blocking SQLSetConnectAttr calls - #541 — GIL released during blocking ODBC statement/fetch/transaction calls - #560 — executemany RuntimeError when decimals change signs - #495 — Inconsistent CP1252 VARCHAR retrieval Windows vs Linux - #559 — BulkCopy empty string in NVARCHAR(MAX)/VARCHAR(MAX) (via mssql_py_core 0.1.4)

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch from ab57bb8 to 6703b85 Compare April 20, 2026 08:01

github-advanced-security Bot found potential problems Apr 20, 2026

View reviewed changes

Comment thread eng/pipelines/pr-validation-pipeline.yml Dismissed

Comment thread eng/pipelines/pr-validation-pipeline.yml Dismissed

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch 2 times, most recently from 2ee0f68 to c113a03 Compare April 20, 2026 09:59

github-actions Bot added the pr-size: large Substantial code update label Apr 20, 2026

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch 3 times, most recently from 970ce4d to 0604fc5 Compare April 24, 2026 11:13

github-actions Bot added pr-size: medium Moderate update size and removed pr-size: large Substantial code update labels Apr 24, 2026

Gaurav Sharma (bewithgaurav) marked this pull request as ready for review April 28, 2026 05:03

Copilot AI review requested due to automatic review settings April 28, 2026 05:03

Copilot started reviewing on behalf of Gaurav Sharma (bewithgaurav) April 28, 2026 05:03 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Comment thread mssql_python/cursor.py

Comment thread mssql_python/cursor.py Outdated

Comment thread mssql_python/cursor.py Outdated

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

Saurabh Singh (saurabh500) reviewed Apr 28, 2026

View reviewed changes

Comment thread mssql_python/cursor.py

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch 2 times, most recently from 612c063 to c529bc9 Compare April 28, 2026 05:31

Saurabh Singh (saurabh500) reviewed Apr 28, 2026

View reviewed changes

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

Comment thread mssql_python/cursor.py

Comment thread mssql_python/cursor.py

Comment thread benchmarks/perf-benchmarking.py Outdated

Comment thread mssql_python/cursor.py

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch from c529bc9 to 536cde1 Compare April 28, 2026 05:42

Gaurav Sharma (bewithgaurav) force-pushed the bewithgaurav/insertmany-perf-python-fixes branch from 536cde1 to 88329a0 Compare April 28, 2026 05:46

Gaurav Sharma (bewithgaurav) and others added 2 commits April 28, 2026 11:59

Replace regex Driver= strip with simple split/filter per review

8a9e16b

Merge branch 'main' into bewithgaurav/insertmany-perf-python-fixes

a1632bf

gargsaumya reviewed Apr 28, 2026

View reviewed changes

Comment thread mssql_python/pybind/ddbc_bindings.cpp

Enhance diagnostic guard comment per review

18b7517

gargsaumya approved these changes Apr 29, 2026

View reviewed changes

Saurabh Singh (saurabh500) approved these changes Apr 29, 2026

View reviewed changes

Gaurav Sharma (bewithgaurav) merged commit 44afe09 into main Apr 29, 2026
34 checks passed

gargsaumya mentioned this pull request May 15, 2026

RELEASE:1.7.0 #585

Closed

gargsaumya mentioned this pull request May 20, 2026

RELEASE:1.7.1 #593

Merged

PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics by bewithgaurav · Pull Request #528 · microsoft/mssql-python · GitHub

Uh oh!

PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics - #528

PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics #528
Gaurav Sharma (bewithgaurav) merged 4 commits into
mainfrom
bewithgaurav/insertmany-perf-python-fixes

Gaurav Sharma (bewithgaurav) commented Apr 17, 2026 •
edited
Loading

Uh oh!

Uh oh!

github-actions Bot commented Apr 17, 2026 •
edited
Loading

Uh oh!

🔥 Diff Coverage

78%

🎯 Overall Coverage

79%

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics by bewithgaurav · Pull Request #528 · microsoft/mssql-python · GitHub

Uh oh!

PERF: Optimize execute() hot path: soft reset, prepare caching, and guarded diagnostics - #528

Conversation

Gaurav Sharma (bewithgaurav) commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Work Item / Issue Reference

Summary

Uh oh!

github-actions Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Code Coverage Report

🔥 Diff Coverage

78%

🎯 Overall Coverage

79%

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

Summary

mssql_python/cursor.py

mssql_python/pybind/ddbc_bindings.cpp

📋 Files Needing Attention

🔗 Quick Links

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Gaurav Sharma (bewithgaurav) commented Apr 17, 2026 •
edited
Loading

github-actions Bot commented Apr 17, 2026 •
edited
Loading