Thanks @mtsokol, this is a useful thing to tackle now. I'm thinking we may want to identify the parts tht can be merged straight away and do this in a few different PRs; the _main_namespace_definition.py seems like it'll stay WIP for a while, while some other parts are quite straightforward. I'm thinking first PR the obvious cleanups (I can add review comments on which ones are mergeable now), and a second one only doing import * removals. WDYT?

@rgommers, works for me! Please comment these items - I will work on it tomorrow. Then I guess first PR will be about "cleanup __init__.py", by removing outdated items and of course import *. Then a separate PR will be to introduce this main namespace contract.

@mtsokol I added the comments regarding what can be merged in a first PR. That should make the diff here a lot smaller.

Also note that if you push updates to this PR, it's probably preferable to add [skip ci] in the commit message - no need for a full battery of CI here yet.

@rgommers it works for me! Then this PR will be for the first batch of changes (general cleaning of numpy/__init__.py and removing NumPy's warnings and exceptions from the main namespace).

The second, a separate PR, will cover solving cyclic dependencies and getting rid of from ... import *, then the third one will introduce a separate file/contract for explicit definition of the main namespace (defining globals(), __all__ and __dir__ this way).

I'm running CI here because I prepared first batch of changes (also, I'm working on reflecting them in other libraries).

@rgommers Looks that there's still a RankWarning class that needs to be removed from top-level __init__.pyi. It's originally from numpy.polynomial.polyutils.py, in my opinion it's domain specific to polynomials, so it doesn't need to be moved to numpy.exceptions. WDYT?

@rgommers Looks that there's still a RankWarning class that needs to be removed from top-level __init__.pyi. It's originally from numpy.polynomial.polyutils.py, in my opinion it's domain specific to polynomials, so it doesn't need to be moved to numpy.exceptions. WDYT?

That's not completely clear cut (right now it goes with np.polyfit), so I'd not touch it here and add it to your "tentative" list.

I think it's ready for a review: In files where an exception/warning was used only once I used np.exceptions.<>. In cases where it was used multiple times I added an explicit import.

Just a note, the effective change here was that previously ComplexWarning and some other errors were available as np.ComplexWarning but hidden because we wanted to move to np.exceptions.ComplexWarning. This finalizes the move without a deprecation.
I can see that mostly being relevant for larger libraries who we can expect to deal with it, but if anyone thinks that a wider range of users have such code, I would be fine with keeping the "hidden" status also for a bit longer.

Just a note, the effective change here was that previously ComplexWarning and some other errors were available as np.ComplexWarning but hidden because we wanted to move to np.exceptions.ComplexWarning. This finalizes the move without a deprecation. I can see that mostly being relevant for larger libraries who we can expect to deal with it, but if anyone thinks that a wider range of users have such code, I would be fine with keeping the "hidden" status also for a bit longer.

I can add a custom message about these warnings/exceptions when accessing them from the main namespace (same as __expired_functions__ worked in __init__.py).

I can add a custom message about these warnings/exceptions when accessing them from the main namespace (same as __expired_functions__ worked in __init__.py).

I'd prefer not to do that for now - at least not until/unless we start seeing a real need. The change is trivial and should be easy to find in case one runs into it. If we are going to add messages for all changes, we will again end up with hundreds of lines of cruft in __init__.py that are going to hang around there for years.

We already planned to have a single doc page with all these changes for 2.0; no need to do double work here. Everyone who uses nightlies can easily deal with this.

Right agreed. My concern is currently only about the sum of changes being overwhelming. For some things we have good reasons to do so because they are things that nobody understands or every dev understands that their logic is flawed. For these, they are a bit fuzzy to me: it is basically a file with "legacy aliases" that is just a long list we would keep long enough that users can adopt it without a try/except or if numpy_version >.

The branching is the real reason here, as we have said many times asking users to change things twice isn't great. I don't think this matters for larger libraries, they are used to it, but it does matter for scripts/small libraries.

In other words, the reason I am fine with it unless someone disagrees, is that I think for the users that we should care about (those who are not used to adding such branching), my guess is that there are very few who will notice the changes.

Exactly, I agree with the "no branching for the average user" rule. These are examples I think that are well below the line of usage frequency. Things like widely-used aliases (e.g., absolute as alias of abs) are used enough that we should keep it as a hidden aliases. And the line is somewhere in the middle between those.