IO: Absorb upsert PyArrow logic into io/pyarrow.py - #3813
Conversation
Sorry, something went wrong.
Move PyArrow-specific table operations (joins, group_by, duplicate detection, row comparison) from table/upsert_util.py into pyiceberg/io/pyarrow.py. The upsert_util module now delegates to helpers in io/pyarrow rather than importing pyarrow directly. This consolidates PyArrow logic behind the io/pyarrow module boundary, which is a precondition for the decomposition proposed in apache#3737. Fixes apache#3812 (PR A)
rambleraptor
left a comment
There was a problem hiding this comment.
This is just a simple code reorg, so it looks fine.
I think it's good to consolidate our PyArrow logic, especially since PyArrow is an optional dependency.
Sorry, something went wrong.
| Or, | ||
| ) | ||
| from pyiceberg.io.pyarrow import ( | ||
| _upsert_get_rows_to_update, |
There was a problem hiding this comment.
These shouldn't be private if we're importing them elsewhere.
Sorry, something went wrong.
There was a problem hiding this comment.
Hi @rambleraptor, thanks for catching that, made the corresponding changes.
Sorry, something went wrong.
rambleraptor
left a comment
There was a problem hiding this comment.
Just a code re-org, so this looks good. Thanks for doing this!
Sorry, something went wrong.
Reviewers
1 more reviewerAt least 1 approving review is required to merge this pull request.
Summary
Move PyArrow-specific table operations (joins, group_by, duplicate detection, row comparison) from table/upsert_util.py into pyiceberg/io/pyarrow.py. The upsert_util module now delegates to helpers in io/pyarrow rather than importing pyarrow directly.
Changes
What stays the same
Why
This consolidates PyArrow logic behind the io/pyarrow module boundary. table/upsert_util.py was the only file in pyiceberg/table/ with a top-level import pyarrow and inline PyArrow compute operations (joins, aggregations, etc.). Moving these operations into io/pyarrow.py means all runtime PyArrow usage routes through one module, which is a precondition for the decomposition proposed in #3737.
See #3812 for the full audit and tracking.
Related