• Issue: Optimize XIData pickle fallback: up to 17x slower than ProcessPoolExecutor for mutable containers #148072

Summary

Cache pickle.dumps and pickle.loads references per interpreter in _PyXI_state_t, avoiding repeated PyImport_ImportModuleAttrString calls on every cross-interpreter data transfer via the pickle fallback path.

Currently, _PyPickle_Dumps() and _PyPickle_Loads() call PyImport_ImportModuleAttrString("pickle", "dumps"/"loads") on every invocation. By caching the function references in the per-interpreter XI state (initialized lazily on first use, cleared during interpreter finalization), this overhead is eliminated.

Changes

• Include/internal/pycore_crossinterp.h: Add pickle.dumps / pickle.loads cache fields to _PyXI_state_t
• Python/crossinterp.c:
  • Add _get_pickle_dumps() / _get_pickle_loads() helpers (lazy init, per-interpreter)
  • Rewrite _PyPickle_Dumps() / _PyPickle_Loads() to use cached borrowed references
  • Initialize cache in _Py_xi_state_init(), clear with Py_CLEAR() in _Py_xi_state_fini()

Benchmark results

Measured with identity(x): return x submitted repeatedly, max_workers=1, Apple M2.

Mutable type transfer (1,000 iterations)

Large payload transfer (500 iterations)

Design notes

• Each interpreter gets its own cache (stored in interp->xi.pickle), preserving interpreter isolation
• Lazily populated on first pickle operation — no startup cost for interpreters that never use pickle fallback
• Py_CLEAR() in _Py_xi_state_fini() runs before xid_lookup_fini() to ensure correct cleanup ordering
• Thread safety guaranteed by per-interpreter GIL, matching the same pattern used for exception type caching in _PyXI_state_t and module state caching in arraymodule.c, _decimal.c

Test plan

• ./python -m test test_interpreters — 172 tests passed
• ./python -m test test_concurrent_futures — 380 tests passed (including test_interpreter_pool)