FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Improve performance of Table argsort and Time get_sortable_arrays by taldcroft · Pull Request #20150 · astropy/astropy · GitHub

Improve performance of Table argsort and Time get_sortable_arrays - #20150

Merged
taldcroft merged 7 commits into
astropy:mainfrom
taldcroft:table-unified-sort
Aug 12, 2026
Merged

Improve performance of Table argsort and Time get_sortable_arrays#20150
taldcroft merged 7 commits into
astropy:mainfrom
taldcroft:table-unified-sort

Conversation

taldcroft commented Jul 29, 2026
edited
Loading

Copy link
Copy Markdown
Member

Description

This PR use col.info.get_sortable_arrays() in Table.argsort() for efficient mixin sorting. This fixes the first bullet point of #14942.

Rationale

Table.argsort() (and hence Table.sort()) currently sorts by converting the key columns to a NumPy array via Table.as_array(). For a single plain key column this is just the column data, but for multiple key columns it builds a structured array over all of them, and if any key is a mixin column (like ~astropy.time.Time) that can't be represented as native structured-array dtype, it gets stored as a slow Python object array. Every comparison during the sort then falls back to Python-level object comparisons instead of vectorized NumPy operations, which is extremely slow — see the benchmark below, where combining a Time column with a plain int column is over 1000x slower than sorting on the int column alone.

get_sortable_arrays() already exists as an info method (used internally by table joins) that returns one or more plain NumPy arrays which can be lexically sorted to reproduce the sort order of a column, without needing an object array. For a normal Column this is just [self.data]. For a Time column it is two float arrays (jd1/jd2 split into an approximate value and a remainder) that fully capture time ordering without loss of precision.

This PR reworks Table.argsort() to call get_sortable_arrays() on each key column and flatten the results into a single list of arrays. If there is only one sortable array overall (e.g. a single plain-column key), np.argsort is used directly, preserving support for the kind argument. Otherwise (multiple keys, and/or a mixin column that expands into multiple arrays) np.lexsort is used, which is always stable and does not accept kind; in that case a non-default, non-stable kind is ignored and triggers an AstropyUserWarning.

If a column's get_sortable_arrays() raises NotImplementedError (e.g. SkyCoord, which has no well-defined sort order), Table.argsort() now raises TypeError rather than falling through to a slow/incorrect object array comparison.

As part of this work, Time.info.get_sortable_arrays() was also optimized: it previously computed the remainder term by constructing a new Time object and subtracting, which is roughly as expensive as computing a full Time.argsort(). It now uses the same cheap jd1/jd2 float arithmetic that Time.argsort() already uses internally, avoiding that redundant object construction.

Changes

  • astropy/table/table.py: Table.argsort() rewritten to use col.info.get_sortable_arrays() per key column, dispatching to np.argsort for a single sortable array or np.lexsort otherwise.
  • astropy/time/core.py: TimeInfo.get_sortable_arrays() optimized to avoid constructing an intermediate Time object.
  • astropy/table/tests/test_table.py: updated test_sort_kind for the new kind warning behavior on multi-key sorts, and added test_sort_kind_single_key covering the single-sortable-array code path for all numpy.argsort kind values.
  • astropy/table/tests/test_operations.py: added regression tests for multi-key sorts involving a Time column, the kind warning on multi-key sorts, and the TypeError raised for non-sortable mixin columns (e.g. SkyCoord).
  • docs/changes/table/20149.perf.rst: changelog entry.

Backwards compatibility notes

  • Sorting on a column whose mixin does not support get_sortable_arrays() (raises NotImplementedError, e.g. SkyCoord) now raises TypeError instead of attempting (and likely failing or behaving incorrectly on) an object-array sort.

Performance

Table.argsort() performance: main vs table-unified-sort

Benchmark script: perf_test_argsort_time.py

N = 100,000 rows, 5 repeats per case, median times reported.

Results

Case main table-unified-sort Speedup
time only 0.0153 s 0.0148 s ~1.0x
["time", "int"] 30.57 s 0.0212 s ~1440x
["int1", "int2"] 0.0454 s 0.0142 s ~3.2x
["int1"] only 0.0048 s 0.0047 s ~1.0x

Key findings

  • Pure plain-column cases (single or multiple int columns) are roughly the same or a few times faster on this branch — the get_sortable_arrays() path avoids building a structured array via as_array().
  • The catastrophic case is mixing a Time mixin column with a plain column on main: sorting ["time", "int"] forces as_array(names=keys), which converts the Time column into a structured array of Python objects, making every comparison during the sort go through slow object comparisons — over 1000x slower than this branch's approach of flattening get_sortable_arrays() and using np.lexsort directly on numeric arrays.
  • This confirms the core motivation for gh-14942: mixin columns like Time combined with any other sort key were essentially unusable at scale on main.

AI disclosure

Essentially all of this PR was generated using GitHub CoPilot with Claude Sonnet 5. I reviewed everything carefully and understand all the changes.

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

taldcroft added this to the v8.1.0 milestone Jul 29, 2026
taldcroft requested a review from neutrinoceros as a code owner July 29, 2026 14:18

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

taldcroft changed the title Table unified sort Improve performance of Table argsort and Time get_sortable_arrays Jul 29, 2026
taldcroft requested a review from mhvk July 29, 2026 15:54
pllim added benchmark Run benchmarks for a PR Performance labels Jul 29, 2026

mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

@taldcroft - the crucial bits are great, and my comments are just a request to reconsider the warning added, and one little thing that can be removed.

Comment thread astropy/table/tests/test_table.py Outdated
t = Table()
t["a"] = [2, 1, 3, 2, 3, 1]
a_struct = t["a"].copy()
with warnings.catch_warnings():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is the default when we run tests, so no need to add
(aside: can do catch_warnings(action="error"))

Comment thread astropy/table/table.py Outdated
The column name(s) to order the table by
kind : {'quicksort', 'mergesort', 'heapsort', 'stable'}, optional
Sorting algorithm used by ``numpy.argsort``.
Sorting algorithm used by `numpy.argsort`. This is only used

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It is not a big deal, but my sense would be never to warn and here just write

Passed on to `numpy.argsort` if sorting is on a single column,
ignored for sorting on multiple columns (including mixin columns
that expand into multiple sortable arrays, like |Time|).

My logic is that for sorting a warning is really not that useful, and for people who, say write a package that wraps table sorting, and who just want stable sorts (which is not the default for argsort), it is nice to be able to give it and not worry about how many columns are going to be sorted.

Note that np.argsort is not that different; the kind parameter is only a request (and mergesort doesn't really exist as a separate option anymore):

    Sorting algorithm. The default is 'quicksort'. Note that both 'stable'
    and 'mergesort' use timsort under the covers and, in general, the
    actual implementation will vary with data type. The 'mergesort' option
    is retained for backwards compatibility.

Comment thread astropy/table/table.py Outdated
kwargs = {"kind": kind} if kind else {}
idx = np.argsort(sortable_arrays[0], **kwargs)
else:
if kind is not None and kind not in ("stable", "mergesort"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

So, I'd remove the warning here (which obviously simplifies the tests too).

Comment thread astropy/time/core.py
jd1, jd2 = parent.jd1, parent.jd2
jd_approx = jd1 + jd2
jd_remainder = (jd1 - jd_approx) + jd2
return [jd_approx, jd_remainder]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I really don't know why we didn't do this before!

taldcroft enabled auto-merge (squash) August 12, 2026 15:03

Copy link
Copy Markdown
Member Author

@mhvk - thanks for the review. I implemented your suggestion to remove the warning, excellent idea.

Copy link
Copy Markdown
Member Author

FYI the codecov/patch 75% of diff hit is wrong. It shows no coverage for code that is obviously run, and I explicitly confirmed that the exception handling is also covered. So I don't know why codecov is busted like that.

pllim commented Aug 12, 2026

Copy link
Copy Markdown
Member

When did you last rebase? Might help.

pllim commented Aug 12, 2026

Copy link
Copy Markdown
Member

I see you have enabled auto-merge but it is not the CI blocking the merge. This PR needs at least one approval. FYI.

mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

All ok now!

taldcroft merged commit 7556e8e into astropy:main Aug 12, 2026
34 of 38 checks passed
taldcroft deleted the table-unified-sort branch August 12, 2026 19:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL