gh-137729: Fix support for locales with @-modifiers by serhiy-storchaka · Pull Request #137253 · python/cpython · GitHub

Issue: locale.getlocale fails if locale is set #87281

Issue: Support locales with @-modifiers #137729

Tests crash on Windows. See #137273.

This may be a dead code, because normalize() above adds encoding.

A few small change requests, but otherwise this looks good. Thanks, @serhiy-storchaka

Please rewrite the comment to:

# Assume ISO8859-15 for @euro locales. Do note that some systems
# may use other encodings for these locales, so this may not always
# be correct.

Please add a comment to this section stating that Linux appears to require keeping the "@euro" modifier in place, even when using the ".ISO8859-15" encoding.

Please add a comment explaining why this is done (to make glibc on Linux happy)

This reverts commit abcbb93.

@malemburg, please look at this PR again. I added documentation and comments as you suggested.

LGTM. Thanks, @serhiy-storchaka

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian Linux Asan 3.x (no tier) has failed when building commit 0c8fecc.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1811/builds/117) and take a look at the build logs.
Check if the failure is related to this commit (0c8fecc) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1811/builds/117

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 19, done.        
remote: Counting objects:   6% (1/15)        
remote: Counting objects:  13% (2/15)        
remote: Counting objects:  20% (3/15)        
remote: Counting objects:  26% (4/15)        
remote: Counting objects:  33% (5/15)        
remote: Counting objects:  40% (6/15)        
remote: Counting objects:  46% (7/15)        
remote: Counting objects:  53% (8/15)        
remote: Counting objects:  60% (9/15)        
remote: Counting objects:  66% (10/15)        
remote: Counting objects:  73% (11/15)        
remote: Counting objects:  80% (12/15)        
remote: Counting objects:  86% (13/15)        
remote: Counting objects:  93% (14/15)        
remote: Counting objects: 100% (15/15)        
remote: Counting objects: 100% (15/15), done.        
remote: Compressing objects:   9% (1/11)        
remote: Compressing objects:  18% (2/11)        
remote: Compressing objects:  27% (3/11)        
remote: Compressing objects:  36% (4/11)        
remote: Compressing objects:  45% (5/11)        
remote: Compressing objects:  54% (6/11)        
remote: Compressing objects:  63% (7/11)        
remote: Compressing objects:  72% (8/11)        
remote: Compressing objects:  81% (9/11)        
remote: Compressing objects:  90% (10/11)        
remote: Compressing objects: 100% (11/11)        
remote: Compressing objects: 100% (11/11), done.        
remote: Total 19 (delta 4), reused 4 (delta 4), pack-reused 4 (from 2)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to '0c8fecc4cfc41e82307a089c474dc69b009bdbcc'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0c8fecc4cfc gh-137729: Fix support for locales with @-modifiers (GH-137253)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

In file included from ./Include/internal/pycore_dict.h:11,
                 from Objects/typeobject.c:7:
In function ‘Py_DECREF_MORTAL’,
    inlined from ‘PyStackRef_XCLOSE’ at ./Include/internal/pycore_stackref.h:730:9,
    inlined from ‘_PyThreadState_PopCStackRef’ at ./Include/internal/pycore_stackref.h:810:5,
    inlined from ‘vectorcall_maybe’ at Objects/typeobject.c:3108:9:
./Include/internal/pycore_object.h:481:8: warning: array subscript 0 is outside array bounds of ‘PyObject[0]’ {aka ‘struct _object[]’} [-Warray-bounds]
  481 |     if (--op->ob_refcnt == 0) {
      |        ^

Timeout (0:05:00)!
Thread 0x0000007f824ef100 [Thread-2] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/setup.py", line 83 in _test_audit_hook
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2057 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007f82cff100 [Thread-1] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2056 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007f8e9f06c0 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/utils.py", line 82 in format_duration
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 464 in get_running
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 551 in _get_result
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 610 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 455 in _run_tests_mp
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 561 in _run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 598 in run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 767 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 775 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/__main__.py", line 2 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
make: *** [Makefile:2494: buildbottest] Error 1

Cannot open file '/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/test-results.xml' for upload

…37253)

serhiy-storchaka requested a review from malemburg July 30, 2025 19:56

bedevere-app Bot mentioned this pull request Jul 30, 2025

locale.getlocale fails if locale is set #87281

Open

pythongh-87281 : Fix support for locales with modifiers

853cfbd

serhiy-storchaka force-pushed the locale-modifiers branch from 11fce5c to 853cfbd Compare July 30, 2025 20:17

serhiy-storchaka commented Jul 31, 2025

View reviewed changes

Skip the crashing tests on Windows.

abcbb93

serhiy-storchaka force-pushed the locale-modifiers branch from 1013430 to abcbb93 Compare July 31, 2025 17:00

serhiy-storchaka mentioned this pull request Aug 2, 2025

gh-87281: Improve documentation for locale.setlocale() and locale.getlocale() #137313

Merged

malemburg approved these changes Aug 3, 2025

View reviewed changes

bedevere-app Bot added the awaiting merge label Aug 3, 2025

serhiy-storchaka changed the title gh-87281: Fix support for locales with modifiers gh-137729: Fix support for locales with @-modifiers Aug 13, 2025

bedevere-app Bot mentioned this pull request Aug 13, 2025

Support locales with @-modifiers #137729

Closed

serhiy-storchaka added 5 commits August 13, 2025 23:43

Revert "Skip the crashing tests on Windows."

e0df7e0

This reverts commit abcbb93.

Merge branch 'main' into locale-modifiers

1e65e5b

Add cocumentation.

4954f97

Add comments.

1096f32

Fix typos.

073572e

serhiy-storchaka requested a review from malemburg August 14, 2025 05:44

serhiy-storchaka marked this pull request as ready for review August 14, 2025 06:01

serhiy-storchaka requested a review from AA-Turner as a code owner August 14, 2025 06:01

bedevere-app Bot added awaiting core review and removed awaiting merge labels Aug 14, 2025

malemburg approved these changes Aug 15, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Aug 15, 2025

Merge branch 'main' into locale-modifiers

8aaf1e7

serhiy-storchaka merged commit 0c8fecc into python:main Aug 18, 2025

bedevere-app Bot removed the awaiting merge label Aug 18, 2025

serhiy-storchaka deleted the locale-modifiers branch August 18, 2025 07:11

Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025

pythongh-137729 : Fix support for locales with @-modifiers (pythonGH-1…

e677823

…37253)

                       if modifier == 'euro' and '.' not in code:
                           # Assume Latin-9 for @euro locales. This is bogus,
                           # since some systems may use other encodings for these
-                          # locales. Also, we ignore other modifiers.
-                          return code, 'iso-8859-15'
+                          # locales.
+                          return code + '@euro', 'ISO8859-15'

@@ @@ -487,11 +487,16 @@ def _parse_localename(localename): @@
                       if modifier == 'euro' and '.' not in code:
                           # Assume Latin-9 for @euro locales. This is bogus,
                           # since some systems may use other encodings for these

                   if modifier == 'euro':
                       if '.' not in code:
-                          return code + '.ISO8859-15'
+                          return code + '.ISO8859-15@euro'

                       # Ignore one letter locale mappings (except for 'c')
                       if len(locale) == 1 and locale != 'c':
                           continue
+                      if '@' in locale and '@' not in alias:

gh-137729: Fix support for locales with @-modifiers by serhiy-storchaka · Pull Request #137253 · python/cpython · GitHub

Repository navigation

gh-137729: Fix support for locales with @-modifiers - #137253

gh-137729: Fix support for locales with @-modifiers#137253
serhiy-storchaka merged 8 commits into
python:mainfrom
serhiy-storchaka:locale-modifiers

serhiy-storchaka commented Jul 30, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

serhiy-storchaka commented Jul 31, 2025

Uh oh!

serhiy-storchaka Jul 30, 2025

Uh oh!

malemburg left a comment

Uh oh!

malemburg Aug 3, 2025 •
edited
Loading

Uh oh!

Uh oh!

malemburg Aug 3, 2025

Uh oh!

malemburg Aug 3, 2025

Uh oh!

serhiy-storchaka commented Aug 14, 2025

Uh oh!

malemburg left a comment

Uh oh!

bedevere-bot commented Aug 18, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-137729: Fix support for locales with @-modifiers by serhiy-storchaka · Pull Request #137253 · python/cpython · GitHub

Repository navigation

gh-137729: Fix support for locales with @-modifiers - #137253

gh-137729: Fix support for locales with @-modifiers#137253serhiy-storchaka merged 8 commits intopython:mainpython/cpython:mainfrom serhiy-storchaka:locale-modifiersserhiy-storchaka/cpython:locale-modifiersCopy head branch name to clipboard

Conversation

serhiy-storchaka commented Jul 30, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Jul 31, 2025

Uh oh!

serhiy-storchaka Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

malemburg left a comment

Choose a reason for hiding this comment

Uh oh!

malemburg Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malemburg Aug 3, 2025

Choose a reason for hiding this comment

Uh oh!

malemburg Aug 3, 2025

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka commented Aug 14, 2025

Uh oh!

malemburg left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 18, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-137729: Fix support for locales with @-modifiers#137253
serhiy-storchaka merged 8 commits into
python:mainfrom
serhiy-storchaka:locale-modifiers

serhiy-storchaka commented Jul 30, 2025 •
edited by bedevere-app Bot
Loading

malemburg Aug 3, 2025 •
edited
Loading