gh-85168 made urllib.request.pathname2url() and url2pathname() use the filesystem encoding and error handler instead of forcing UTF-8, but the nturl2path module was left unchanged: "No changes are needed in the nturl2path module because Windows always uses UTF-8, per PEP 529".
That is only half true. A Windows path is a sequence of 16-bit code units, and unpaired surrogates are allowed in it. Python decodes such a name with the filesystem error handler surrogatepass, so the resulting string contains a lone surrogate, which urllib.parse.quote() cannot encode with the default strict handler:
>>> import sys, nturl2path
>>> sys.getfilesystemencoding(), sys.getfilesystemencodeerrors()
('utf-8', 'surrogatepass')
>>> nturl2path.pathname2url('C:\\a\\b\udd00')
Traceback (most recent call last):
...
File "Lib\nturl2path.py", line 68, in pathname2url
tail = urllib.parse.quote(comp[1])
UnicodeEncodeError: 'utf-8' codec can't encode character '\udd00' in position 4: surrogates not allowed
quote() and unquote() should be called with encoding=sys.getfilesystemencoding() and errors=sys.getfilesystemencodeerrors(), as in urllib.request.
In 3.13 this is reachable through urllib.request.pathname2url(), which uses nturl2path on Windows. It is why the fix for gh-69371 cannot be backported to 3.13 as is (GH-156610 fails on Windows). Since 3.14 the module is deprecated and urllib.request no longer uses it, so only direct users are affected.
Activity
gh-156713: Use the filesystem encoding in nturl2path (GH-156717)
[3.14] gh-156713: Use the filesystem encoding in nturl2path (GH-156717)…
[3.13] gh-156713: Use the filesystem encoding in nturl2path (GH-156717)…
pythongh-156713: Fix test_nturl2path for non-UTF-8 filesystem encodings
gh-156713: Fix test_nturl2path for non-UTF-8 filesystem encodings (GH…
[3.13] pythongh-156713: Fix test_nturl2path for non-UTF-8 filesystem …
[3.14] gh-156713: Fix test_nturl2path for non-UTF-8 filesystem encodi…
[3.13] gh-156713: Fix test_nturl2path for non-UTF-8 filesystem encodi…
pythongh-156713: Use the filesystem encoding in nturl2path (pythonGH-…
pythongh-156713: Fix test_nturl2path for non-UTF-8 filesystem encodin…
[3.15] gh-156713: Use the filesystem encoding in nturl2path (GH-156717)…