makeunicodedata.py currently emits _PyUnicode_ToNumeric() as a switch
with 2,023 case labels. On Windows ARM64, MSVC's /O2 optimizer consumes
excessive memory and the x86-hosted ARM64 compiler fails with C1002 while
compiling Objects/unicodectype.c.
Generate a sorted code-point table, a parallel value-index table, and a
table of the 151 UCD numeric-value entries instead. _PyUnicode_ToNumeric()
now uses binary search and still returns -1.0 for code points without a
numeric value. NUMERIC_MASK and _PyUnicode_IsNumeric() are unchanged.
This is a source-level alternative to the temporary build workaround in #153669. That PR disables optimization for affected MSVC toolsets; this
change keeps optimization enabled by removing the compiler-stressing
generated switch.
Windows ARM64 result
VCTools 14.51.36231, Release /O2, on a Windows 11 ARM64 lab host:
Compiler host
main
This PR
HostX86 -> ARM64
C1002 after 11.487 s / 3.544 GB private
success in 0.971 s / 0.037 GB
HostARM64 -> ARM64
success in 29.482 s / 25.196 GB private
success in 0.598 s / 0.050 GB
Validation
Full native ARM64 Release build: success.
python -m test -j1 test_unicodedata test_str: 212 tests passed.
python -m test -j1 -u cpu test_unicodedata: 73 tests passed.
python -m test -j0 -x test_profiling: 49,207 tests passed across
442 test files. test_profiling was excluded because its sampling
coordinator hangs independently on this lab image.
Exhaustively compared all 1,114,112 Unicode code points with the previous
generated switch; all 2,023 numeric mappings and non-numeric results match.
Tools\patchcheck\patchcheck.py: passed.
AI assistance: OpenAI Codex assisted with the investigation, implementation,
and validation. This PR is opened as a draft so the submitter can complete
the required human review before marking it ready.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #153668.
Summary
makeunicodedata.py currently emits _PyUnicode_ToNumeric() as a switch
with 2,023 case labels. On Windows ARM64, MSVC's /O2 optimizer consumes
excessive memory and the x86-hosted ARM64 compiler fails with C1002 while
compiling Objects/unicodectype.c.
Generate a sorted code-point table, a parallel value-index table, and a
table of the 151 UCD numeric-value entries instead. _PyUnicode_ToNumeric()
now uses binary search and still returns -1.0 for code points without a
numeric value. NUMERIC_MASK and _PyUnicode_IsNumeric() are unchanged.
This is a source-level alternative to the temporary build workaround in
#153669. That PR disables optimization for affected MSVC toolsets; this
change keeps optimization enabled by removing the compiler-stressing
generated switch.
Windows ARM64 result
VCTools 14.51.36231, Release /O2, on a Windows 11 ARM64 lab host:
Validation
442 test files. test_profiling was excluded because its sampling
coordinator hangs independently on this lab image.
generated switch; all 2,023 numeric mappings and non-numeric results match.
AI assistance: OpenAI Codex assisted with the investigation, implementation,
and validation. This PR is opened as a draft so the submitter can complete
the required human review before marking it ready.