locale.getlocale fails if locale is set · Issue #87281 · python/cpython · GitHub

BPO	43115
Nosy	@malemburg, @pfmoore, @tjguk, @zware, @eryksun, @zooba, @AndersMunch

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-03.13:42:50.296>
labels = ['3.8', '3.10', 'library', '3.9', 'OS-windows']
title = 'locale.getlocale fails if locale is set'
updated_at = <Date 2021-02-23.17:59:57.570>
user = 'https://github.com/AndersMunch'

bugs.python.org fields:

activity = <Date 2021-02-23.17:59:57.570>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2021-02-03.13:42:50.296>
creator = 'AndersMunch'
dependencies = []
files = []
hgrepos = []
issue_num = 43115
keywords = []
message_count = 15.0
messages = ['386203', '387136', '387138', '387147', '387149', '387157', '387164', '387195', '387205', '387208', '387209', '387256', '387273', '387558', '387588']
nosy_count = 8.0
nosy_names = ['lemburg', 'paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'swt2c', 'AndersMunch']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43115'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

getlocale fails with an exception when the string returned by _setlocale is already an RFC 1766 language tag.

Example:

Python 3.10.0a4 (tags/v3.10.0a4:445f7f5, Jan  4 2021, 19:55:53) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en-US')
'en-US'
>>> locale.getlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\flonidan\env\Python310\lib\locale.py", line 593, in getlocale
    return _parse_localename(localename)
  File "C:\flonidan\env\Python310\lib\locale.py", line 501, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: en-US

Expected result:
('en-US', None)

See wxWidgets/Phoenix#1637 for an example of the ensuing problems. wx.Locale calls C setlocale directly, but, as far as I can see, correctly, using dashes in the language code which is consistent with not only RFC 1766 but also the examples in Microsoft's documentation (https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160). CPython seems to assume underscored names such as 'en_US' instead, as shown by getdefaultlocale inserting an underscore.

I discovered that this can happen with underscores as well:

Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en_DE')
'en_DE'
>>> locale.getlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\flonidan\env\Python38-64\lib\locale.py", line 591, in getlocale
    return _parse_localename(localename)
  File "C:\flonidan\env\Python38-64\lib\locale.py", line 499, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: en_DE

locale.setlocale does validate input - if you write nonsense in the second argument then you get an exception, "locale.Error: unsupported locale setting". So I'm guessing the en_DE locale is actually being set here, and the problem is solely about getlocale making unfounded assumptions about the format.

Same thing happens in 3.10.0a4.

On 17.02.2021 10:55, Anders Munch wrote:
>>>> import locale
>>>> locale.setlocale(locale.LC_ALL, 'en_DE')
> 'en_DE'
>>>> locale.getlocale()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "C:\flonidan\env\Python38-64\lib\locale.py", line 591, in getlocale
>     return _parse_localename(localename)
>   File "C:\flonidan\env\Python38-64\lib\locale.py", line 499, in _parse_localename
>     raise ValueError('unknown locale: %s' % localename)
> ValueError: unknown locale: en_DE

The locale module does not know this encoding, so cannot
guess the encoding. Since getlocale() returns the language
code and encoding, this fails.

If you add the encoding, you should be fine:

>>> locale.setlocale(locale.LC_ALL, 'en_DE.UTF-8')
'en_DE.UTF-8'
>>> locale.getlocale()
('en_DE', 'UTF-8')

getlocale is documented to return None for the encoding if no encoding can be determined. There's no need to guess.

I can't change the locale.setlocale call, because where I'm actually having the problem, I'm not even calling locale.setlocale: wxWidgets is calling C setlocale, that's where it comes from.

wxWidgets aside, it doesn't seem right that getlocale fails when setlocale succeeded.

On 17.02.2021 12:36, Anders Munch wrote:

getlocale is documented to return None for the encoding if no encoding can be determined. There's no need to guess.

Well, not quite... the documentation says that None can be returned,
not that it will return None in such cases.

What happens is that the value returned by the C lib's setlocale()
is normalized and parsed. If it doesn't include an encoding,
the table locale_alias is used to provide an encoding. If this
table does not include the locale, you get the ValueError.

We could enhance this to return None for the encoding instead
of raising an exception, but would this really help ?

Alternatively, we could add "en_DE" to the alias table and set
a default encoding to use. I assume this would have to be
ISO8859-15. OTOH, Windows supports UTF-8 for all locales in
more recent Windows versions, so perhaps we should use UTF-8
instead.

BTW: What is wxWidgets doing with the returned values ?

Resources:

setlocale() in Visual C++ on Windows:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160

locale name formats supported by Windows CRT:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-strings?view=msvc-160

Language locale tags known by Windows:

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c

UTF8 support in Windows CRT:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160#utf-8-support

pythongh-87281: Improve documentation for locale.setlocale() and loca…

gh-87281: Improve documentation for locale.setlocale() and locale.get…

pythongh-87281: Improve documentation for locale.setlocale() and loca…

[3.13] gh-87281: Improve documentation for locale.setlocale() and loc…

pythongh-87281: Improve documentation for locale.setlocale() and loca…

[3.14] gh-87281: Improve documentation for locale.setlocale() and loc…

Triage: Is there anything left to do here? All backports are complete.

Yes, this is a large and complex issue. It needs a lot of work.

locale.getlocale fails if locale is set · Issue #87281 · python/cpython · GitHub

Repository navigation

locale.getlocale fails if locale is set #87281

Linked PRs

25 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

locale.getlocale fails if locale is set · Issue #87281 · python/cpython · GitHub

Repository navigation

locale.getlocale fails if locale is set #87281

Description

Linked PRs

Activity

AndersMunch commented on Feb 3, 2021

AndersMunch commented on Feb 17, 2021

malemburg commented on Feb 17, 2021

AndersMunch commented on Feb 17, 2021

malemburg commented on Feb 17, 2021

25 remaining items

StanFromIreland commented on Sep 17, 2025

serhiy-storchaka commented on Sep 17, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions