More generally, I'm not convinced it's worth the risk to backport non-trivial fixes to issues that solely come up when using the combination of free-threading + subinterpreters.

Hmm, why? This will hurt PEP 734, because Eric's PyPI package for backporting features is targeted towards 3.13+, not 3.14+. I'm not sure how many people are using 3.13t as their working copy, but I would assume that the people who are using it are also the people who would like to use subinterpreters to get around the GIL as well.

• Both free threading and _interpreters are experimental; the combination is especially so.
• I think the scope of getting the two to work robustly together is large; more like a "feature" than a "bug fix"
• Some of these changes are risky; we may introduce regressions to fix something that is not a regression.
• I do not fully understand the thread-safety implications here. From the PR description, I'm not sure you fully understand them either. If you do, then the thread-safety requirements should be more clearly explained in the PR.

• Some of these changes are risky; we may introduce regressions to fix something that is not a regression.

Judging by the failing tests, I see your point :)

What do you suggest we do for 3.13? Slap a big Py_MOD_GIL_USED on _interpreters?

• I do not fully understand the thread-safety implications here. From the PR description, I'm not sure you fully understand them either. If you do, then the thread-safety requirements should be more clearly explained in the PR.

Yeah, I'm pretty sure that the cases I've fixed in this PR are truly fixed, but the test case isn't extensive at all. I'm doing my best to try and cover the bases that aren't thread-safe here, but as far as I can tell, interpreter switching without waiting on the GIL is a gray area. There are things that work right now that shouldn't, and I'm not totally sure why. (I'm not even addressing the C API here, that's probably still a mess--this change is assuming that subinterpreters created by _interpreters are only used by _interpreters.)

@colesbury FWIW, I was being dumb. The exception problem is almost certainly present on the default builds (I said that in the original issue, but not in the description for whatever reason). I'm assuming the same goes for the destroy issue, but I haven't been able to reproduce it yet.

Apparently, there's some sort of race on both builds when creating thread states that's been screwing up my reproducers (I get hit with a thread state already initialized fatal error on both GIL and no-GIL):

from threading import Thread
import _interpreters

interp = _interpreters.create()

def run():
    this_interp = _interpreters.create()
    _interpreters.run_string(this_interp, f"import _interpreters; _interpreters.run_string({interp}, '1')")

threads = [Thread(target=run) for _ in range(1000)]
for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

As far as I can tell, moving HEAD_LOCK here was an incidental fix for that. I'll investigate this further later today.

I'm going to split this into smaller PRs so we can address reviews and backporting on a case-by-case basis.