Dataframe Interchange Protocol#

The interchange protocol is implemented for pa.Table and pa.RecordBatch and is used to interchange data between PyArrow and other dataframe libraries that also have the protocol implemented. The data structures that are supported in the protocol are primitive data types plus the dictionary data type. The protocol also has missing data support and it supports chunking, meaning accessing the data in batches of rows.

The Python dataframe interchange protocol is designed by the Consortium for Python Data API Standards in order to enable data interchange between dataframe libraries in the Python ecosystem. See more about the standard in the protocol documentation.

From PyArrow to other libraries: __dataframe__() method#

The __dataframe__() method creates a new exchange object that the consumer library can take and construct an object of its own.

>>> import pyarrow as pa
>>> table = pa.table({"n_attendees": [100, 10, 1]})
>>> table.__dataframe__()
<pyarrow.interchange.dataframe._PyArrowDataFrame object at ...>

This is meant to be used by the consumer library when calling the from_dataframe() function and is not meant to be used manually by the user.

From other libraries to PyArrow: from_dataframe()#

With the from_dataframe() function, we can construct a pyarrow.Table from any dataframe object that implements the __dataframe__() method via the dataframe interchange protocol.

We can for example take a polars dataframe and construct a PyArrow table with the use of the interchange protocol:

>>> import polars as pl
>>> from datetime import datetime
>>> arr = [datetime(2023, 5, 20, 10, 0),
...        datetime(2023, 5, 20, 11, 0),
...        datetime(2023, 5, 20, 13, 30)]
>>> df = pl.DataFrame({
...          'Talk': ['About Polars','Intro into PyArrow','Coding in Rust'],
...          'Time': arr,
...      })
>>> df
shape: (3, 2)

 Talk                Time                
 ---                 ---                 
 str                 datetime[s]        

 About Polars        2023-05-20 10:00:00 
 Intro into PyArrow  2023-05-20 11:00:00 
 Coding in Rust      2023-05-20 13:30:00 

>>> from_dataframe(df)
pyarrow.Table
Talk: large_string
Time: timestamp[us]
----
Talk: [["About Polars","Intro into PyArrow","Coding in Rust"]]
Time: [[2023-05-20 10:00:00.000000,2023-05-20 11:00:00.000000,2023-05-20 13:30:00.000000]]