[mypyc] Add support for `str.lower()` and `str.upper()` by VaggelisD · Pull Request #20948 · python/mypy · GitHub

Follow up on #19375 with full Unicode support:

ASCII fast path
Shared CPyStr_ChangeCase helper, parameterized by function pointers
_PyUnicode_ToLowerFull/_PyUnicode_ToUpperFull for Unicode which handle 1-to-N expansion (e.g., ß→SS); This was a sticky point with the previous PR which relied on Py_UNICODE_TOLOWER/TOUPPER.
Temporary len * 3 UCS-4 buffer for the Unicode path; This is because each Unicode char may be expanded from 1 byte to 3

Benchmarks (x10 M calls each)

Input	Method	mypyc	CPython	Speedup
"" (empty)	lower	0.070s	0.182s	2.60x
"" (empty)	upper	0.069s	0.181s	2.62x
ASCII len 1 (convert)	lower	0.187s	0.234s	1.25x
ASCII len 1 (convert)	upper	0.161s	0.246s	1.53x
ASCII len 1 (no-op)	lower	0.157s	0.234s	1.49x
ASCII len 1 (no-op)	upper	0.151s	0.245s	1.62x
ASCII len 10	lower	0.189s	0.267s	1.41x
ASCII len 10	upper	0.189s	0.278s	1.47x
ASCII len 100	lower	0.434s	0.510s	1.18x
ASCII len 100	upper	0.431s	0.527s	1.22x
Unicode len 1 (Latin)	lower	0.152s	0.309s	2.03x
Unicode len 1 (Latin)	upper	0.156s	0.320s	2.05x
Unicode len 10 (Latin)	lower	0.438s	0.543s	1.24x
Unicode len 10 (Latin)	upper	0.441s	0.530s	1.20x
Unicode len 100 (Latin)	lower	2.560s	2.897s	1.13x
Unicode len 100 (Latin)	upper	2.564s	2.001s	0.78x
Unicode len 1 (Greek)	lower	0.216s	0.304s	1.41x
Unicode len 1 (Greek)	upper	0.218s	0.313s	1.44x
Unicode len 10 (Greek)	lower	0.526s	0.587s	1.12x
Unicode len 10 (Greek)	upper	0.479s	0.478s	1.00x
1-to-N expansion len 1	lower	0.332s	0.318s	0.96x
1-to-N expansion len 1	upper	0.258s	0.323s	1.25x
1-to-N expansion len 10	lower	0.614s	0.655s	1.07x
1-to-N expansion len 10	upper	0.526s	0.542s	1.03x

Oops, looks like the APIs I based the PR on are not exported past 3.13+, will need to figure out a solution. The str.lower() and str.upper() primitives continue to claim victims even after a year

That was a mistake afaict, CPY_3_13_FEATURES is defined by us so that'd always evaluate to true

I was testing this branch on Python 3.11; For 3.13+ I had to fallback to CPython's generic dispatch for Unicode strings.

This introduces the following regression for the latter versions (tested on 3.14)

Input	Method	mypyc	CPython	Speedup
"" (empty)	lower	0.118s	0.144s	1.22x
"" (empty)	upper	0.104s	0.143s	1.38x
ASCII len 1 (convert)	lower	0.200s	0.234s	1.17x
ASCII len 1 (convert)	upper	0.196s	0.232s	1.18x
ASCII len 1 (no-op)	lower	0.194s	0.224s	1.15x
ASCII len 1 (no-op)	upper	0.199s	0.229s	1.15x
ASCII len 10	lower	0.222s	0.256s	1.15x
ASCII len 10	upper	0.226s	0.256s	1.13x
ASCII len 100	lower	0.476s	0.499s	1.05x
ASCII len 100	upper	0.490s	0.510s	1.04x
Unicode len 1 (Latin)	lower	0.404s	0.305s	0.75x
Unicode len 1 (Latin)	upper	0.420s	0.313s	0.74x
Unicode len 10 (Latin)	lower	0.652s	0.544s	0.83x
Unicode len 10 (Latin)	upper	0.566s	0.481s	0.85x
Unicode len 1 (Greek)	lower	0.406s	0.308s	0.76x
Unicode len 1 (Greek)	upper	0.404s	0.303s	0.75x
1-to-N len 1	lower	0.425s	0.326s	0.77x
1-to-N len 1 (ß→SS)	upper	0.425s	0.326s	0.77x

Thanks for the PR! Can you also compare performance against mypyc master, on Python 3.14?

Since lower and upper calls are very common, we might eventually want to include lower case / upper case tables for at least UCS-2 within librt (that are only included in the build if the primitives are actually used, i.e. as optional files). But the ascii case is very common, so this is already helpful.

I applied the inlining and reran the benchmarks:

For our branch vs Python 3.14, it looks like the regressions disappeared, not entirely sure why, maybe inlining did all that magic (the previous ones were on Python 3.11)

For our branch vs master, the results are logical i.e regression kicks in for large Unicode strings but other than that we're on par or faster.

Input	Method	mypyc (branch)	mypyc (master)	Python	vs master	vs CPython
"" (empty)	lower	0.063s	0.159s	0.211s	2.52x	3.35x
"" (empty)	upper	0.062s	0.163s	0.206s	2.63x	3.32x
ASCII len 1 (convert)	lower	0.157s	0.230s	0.260s	1.46x	1.66x
ASCII len 1 (convert)	upper	0.150s	0.272s	0.272s	1.81x	1.81x
ASCII len 1 (no-op)	lower	0.156s	0.229s	0.269s	1.47x	1.72x
ASCII len 1 (no-op)	upper	0.150s	0.232s	0.272s	1.55x	1.81x
ASCII len 10	lower	0.189s	0.260s	0.310s	1.38x	1.64x
ASCII len 10	upper	0.185s	0.266s	0.302s	1.44x	1.63x
ASCII len 100	lower	0.429s	0.548s	0.554s	1.28x	1.29x
ASCII len 100	upper	0.436s	0.517s	0.557s	1.19x	1.28x
ASCII len 100 (no-op)	lower	0.440s	0.512s	0.551s	1.16x	1.25x
ASCII len 100 (no-op)	upper	0.442s	0.515s	0.563s	1.17x	1.27x
Unicode len 1 (Latin)	lower	0.353s	0.354s	0.392s	1.00x	1.11x
Unicode len 1 (Latin)	upper	0.356s	0.420s	0.380s	1.18x	1.07x
Unicode len 10 (Latin)	lower	0.517s	0.579s	0.571s	1.12x	1.10x
Unicode len 10 (Latin)	upper	0.523s	0.544s	0.586s	1.04x	1.12x
Unicode len 100 (Latin)	lower	2.407s	2.476s	2.431s	1.03x	1.01x
Unicode len 100 (Latin)	upper	2.587s	2.645s	2.637s	1.02x	1.02x
Unicode len 1 (Greek)	lower	0.357s	0.353s	0.396s	0.99x	1.11x
Unicode len 1 (Greek)	upper	0.362s	0.347s	0.392s	0.96x	1.08x
Unicode len 10 (Greek)	lower	0.582s	0.564s	0.643s	0.97x	1.10x
Unicode len 10 (Greek)	upper	0.577s	0.572s	0.672s	0.99x	1.16x
1-to-N len 1 (İ)	lower	0.375s	0.364s	0.415s	0.97x	1.11x
1-to-N len 10 (İ)	lower	0.641s	0.632s	0.676s	0.99x	1.05x
1-to-N len 1 (ß→SS)	upper	0.330s	0.324s	0.370s	0.98x	1.12x
1-to-N len 10 (ß)	upper	0.613s	0.609s	0.648s	0.99x	1.06x
1-to-N len 1 (ffi→FFI)	upper	0.350s	0.341s	0.385s	0.97x	1.10x

Thanks, looks good! I'm glad that we finally have primitives for these common operations.

#21553) 6th PR of #21418. This PR introduces two `i32 -> i32` case-conversion helpers, alongside the existing classifiers. **The constraint to flag**: A single i32 holds one codepoint, but some Unicode case mappings expand to multiple e.g `'ß'.upper()` becomes `'SS'`, `'ﬁ'.upper()` becomes `'FI'` etc. For those inputs the primitive _returns the input unchanged_; This is the same split CPython makes between `Py_UNICODE_TOUPPER` (codepoint) and `str.upper()` (string), with the former returning the **first codepoint** of the expansion. Users needing full Unicode case conversion should call `s.upper()` / `s.lower()` on the string, for which we already have mypyc primitives (#20948). For ASCII benchmarks, the codepoint primitives are ~5x faster than their `str` counterparts, avoiding the 1-char allocation.

VaggelisD added 2 commits March 2, 2026 17:17

[mypyc] Add support for str.lower() and str.upper()

dc11c93

Fix Python 3.13+ which moved API internally

5346404

Fix Python 3.13+ by calling CPython API

4bb387f

VaggelisD commented Mar 2, 2026

View reviewed changes

JukkaL reviewed Mar 2, 2026

View reviewed changes

Comment thread mypyc/lib-rt/str_ops.c Outdated

Comment thread mypyc/lib-rt/str_ops.c Outdated

Inline helpers

bf2239d

JukkaL approved these changes Mar 4, 2026

View reviewed changes

JukkaL merged commit e6d41eb into python:master Mar 4, 2026
18 checks passed

VaggelisD deleted the str_lower_upper branch March 5, 2026 08:11

hauntsaninja mentioned this pull request Mar 14, 2026

[mypyc] Implement str.lower() and str.upper() primitive #19375

Closed

VaggelisD mentioned this pull request May 28, 2026

[mypyc] Add librt.strings.toupper and tolower codepoint primitives #21553

Merged

[mypyc] Add support for `str.lower()` and `str.upper()` by VaggelisD · Pull Request #20948 · python/mypy · GitHub

Repository navigation

[mypyc] Add support for str.lower() and str.upper() - #20948

[mypyc] Add support for str.lower() and str.upper()#20948
JukkaL merged 4 commits into
python:masterfrom
VaggelisD:str_lower_upper

VaggelisD commented Mar 2, 2026

Uh oh!

VaggelisD commented Mar 2, 2026

Uh oh!

VaggelisD Mar 2, 2026

Uh oh!

VaggelisD commented Mar 2, 2026

Uh oh!

JukkaL left a comment

Uh oh!

Uh oh!

Uh oh!

JukkaL commented Mar 2, 2026

Uh oh!

VaggelisD commented Mar 3, 2026 •
edited
Loading

Uh oh!

Uh oh!

JukkaL left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

               // The _PyUnicode_CheckConsistency definition has been moved to the internal API
               // https://github.com/python/cpython/pull/106398
-              #if defined(Py_DEBUG) && defined(CPY_3_13_FEATURES)

[mypyc] Add support for `str.lower()` and `str.upper()` by VaggelisD · Pull Request #20948 · python/mypy · GitHub

Repository navigation

[mypyc] Add support for str.lower() and str.upper() - #20948

[mypyc] Add support for str.lower() and str.upper()#20948JukkaL merged 4 commits intopython:masterpython/mypy:masterfrom VaggelisD:str_lower_upperVaggelisD/sqlglot-mypy:str_lower_upperCopy head branch name to clipboard

Conversation

VaggelisD commented Mar 2, 2026

Benchmarks (x10 M calls each)

Uh oh!

VaggelisD commented Mar 2, 2026

Uh oh!

VaggelisD Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

VaggelisD commented Mar 2, 2026

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JukkaL commented Mar 2, 2026

Uh oh!

VaggelisD commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[mypyc] Add support for str.lower() and str.upper()#20948
JukkaL merged 4 commits into
python:masterfrom
VaggelisD:str_lower_upper

VaggelisD commented Mar 3, 2026 •
edited
Loading