FEAT: Support for Apache Arrow in Bulk Copy API

[gargsaumya]

Work Item / Issue Reference

AB#46268

GitHub Issue: #<ISSUE_NUMBER>

---

Summary

This pull request introduces a new high-performance bulkcopy_arrow method to the Cursor class for bulk loading data directly from Apache Arrow sources, along with several related improvements and refactorings. It also updates type stubs and documentation to reflect the new API, and improves bulk copy authentication handling by refactoring shared logic.

New feature: Arrow-based bulk copy

• Added Cursor.bulkcopy_arrow(table_name, source) for efficient bulk loading from Arrow sources (e.g., pyarrow.Table, RecordBatch, objects exposing the Arrow C Data Interface). This avoids unnecessary Python row materialization and is significantly faster for Arrow-native data. [1] [2]
• The classic bulkcopy() method now raises a TypeError if given Arrow-shaped data, steering users to the new method.

Codebase refactoring and improvements

• Refactored connection context and Azure AD authentication logic into a shared _build_pycore_context() helper, used by both bulkcopy and bulkcopy_arrow. This ensures consistent authentication handling and reduces code duplication. [1] [2]
• Updated type stubs (mssql_python.pyi) to include the new bulkcopy_arrow method and improved the typing for bulkcopy. [1] [2]

Documentation

• Updated the CHANGELOG.md to document the new Arrow bulk copy feature and its requirements.

Minor improvements

• Minor code formatting improvement in a test file (test_bulkcopy_udt_geometry).

[Copilot]

Pull request overview

Adds Apache Arrow support to the Bulk Copy API by introducing a dedicated Cursor.bulkcopy_arrow() path and refactoring shared connection/auth context construction so both tuple-based and Arrow-based bulk copy use consistent handling.

Changes:

• Added Cursor.bulkcopy_arrow(table_name, source) and Arrow-source detection/steering to prevent Arrow inputs from going through tuple bulkcopy.
• Refactored bulk-copy connection/auth parsing into Cursor._build_pycore_context() for reuse across bulk copy entry points.
• Added comprehensive tests and a benchmark comparing tuple vs Arrow bulk copy throughput.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_024_bulkcopy_arrow.py	Adds unit + live-DB integration coverage for Arrow bulk copy, auth context building, and Arrow-source detection.
mssql_python/mssql_python.pyi	Updates public type stubs to include bulkcopy and new bulkcopy_arrow API.
mssql_python/cursor.py	Implements _build_pycore_context, Arrow source detection/steering, and the new bulkcopy_arrow method.
CHANGELOG.md	Documents the new Arrow bulk copy feature and related requirements/behavior changes.
benchmarks/bench_bulkcopy_arrow.py	Adds a benchmark script comparing tuple-based vs Arrow-based bulk copy performance.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 81%

📈 Total Lines Covered: 7145 out of 8757 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/cursor.py (100%)

Summary

• Total: 80 lines
• Missing: 0 lines
• Coverage: 100%

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.3%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 81.4%
mssql_python.pybind.connection.connection.cpp: 83.7%
mssql_python.connection.py: 84.7%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[bewithgaurav]

no major problems or blockers found - just some improvements to code structure and a nit miss on error message, will approve once done

[bewithgaurav]

this preamble is byte-identical to bulkcopy(), as are three other chunks:
the table_name check, the batch_size and timeout validation below it, and the whole finally block with the credential scrub and cleanup. a few lines duplicated across the two methods.

two static helpers cover all four, _bulkcopy_core_and_validate(table_name, batch_size, timeout) returning the module and _bulkcopy_teardown(pycore_context, pycore_cursor, pycore_connection).

I made changes locally, net 43 lines lighter, no flags and no dynamic dispatch

why I pointed:
the except block a few lines down got changed to a bare raise, bulkcopy still does raise type(e)(str(e)) from None - identical code, fixed in one copy, missed in the other

[bewithgaurav]

nit: str and bytes slip past this block and fail later with a message that doesn't name the cause:

cursor.bulkcopy_arrow("dbo.T", "data.parquet")
TypeError: iterable must yield pyarrow.RecordBatch instances

both are iterable, so they get treated as an iterable of batches and die on the first character. bulkcopy guards for exactly this a couple hundred lines up and says

• "got str. Strings and bytes are not valid row collections."

folding it into the existing None check keeps it to one branch:

Suggested change

	if source is None:
	raise TypeError(
	"source must be a pyarrow Table/RecordBatch/RecordBatchReader, "
	"an iterable of RecordBatch, or an object implementing "
	"__arrow_c_stream__/__arrow_c_array__, got None"
	)
	if source is None or isinstance(source, (str, bytes)):
	got = "None" if source is None else type(source).__name__
	raise TypeError(
	"source must be a pyarrow Table/RecordBatch/RecordBatchReader, "
	"an iterable of RecordBatch, or an object implementing "
	f"__arrow_c_stream__/__arrow_c_array__, got {got}"
	)

everything else reads fine here. a list of tuples gets the RecordBatch message and that's the correct thing to say. it's only these two types which are missed.