FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

GH-85168: Use filesystem encoding when converting to/from `file` URIs by barneygale · Pull Request #126852 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

GH-85168: Use filesystem encoding when converting to/from file URIs - #126852

Merged
barneygale merged 5 commits into
python:mainfrom
barneygale:gh-85168
Nov 19, 2024
ConversationCommits (5)ChecksFiles changed
Merged

GH-85168: Use filesystem encoding when converting to/from file URIs#126852
barneygale merged 5 commits into
python:mainfrom
barneygale:gh-85168

Conversation

barneygale commented Nov 15, 2024 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

Adjust urllib.request.url2pathname() and pathname2url() to use the filesystem encoding when quoting and unquoting file URIs, rather than forcing use of UTF-8.

No changes are needed in the nturl2path module because Windows always uses UTF-8, per PEP 529.

pythonGH-85168: Use filesystem encoding when converting to/from `file…
11b9602 
…` URIs

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
bedevere-app Bot mentioned this pull request Nov 15, 2024
Closed
barneygale marked this pull request as ready for review November 15, 2024 02:09
barneygale added needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes labels Nov 15, 2024

barneygale commented Nov 15, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

The freedesktop file URI spec is a useful reference here:

There is no specified encoding for filenames, and although we hope
that eventually all filenames will be encoded in UTF8 we can't rely on
this, because then we would be unable to e.g. rename a file with a
misencoded filename.

Also:

Some current apps generate file URIs by converting the filename from
whatever locale the application runs in to UTF8. This behavior means
that URIs can't be converted to filenames without knowing the
locale of the application that produced them, and that not all valid
filenames can be converted to URIs. Such behavior is not allowed,
and should be changed.

serhiy-storchaka reviewed Nov 15, 2024
View reviewed changes

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM in general. But we should support URIs containing not quoted non-ASCII characters.

Comment thread Lib/urllib/request.py Outdated
Comment thread Lib/test/test_urllib.py Outdated

barneygale commented Nov 15, 2024

Copy link
Copy Markdown
Contributor Author

Thanks for the review - I've made the changes you suggested

serhiy-storchaka approved these changes Nov 18, 2024
View reviewed changes

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM.

You can also use fsdecode(TESTFN_UNDECODABLE) and TESTFN_UNENCODABLE in some tests if you wish.

barneygale merged commit c9b399f into python:main Nov 19, 2024

miss-islington-app Bot commented Nov 19, 2024

Copy link
Copy Markdown

Thanks @barneygale for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 19, 2024
pythonGH-85168: Use filesystem encoding when converting to/from `file…
bcb48b0 
…` URIs (pythonGH-126852)

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
(cherry picked from commit c9b399f)

Co-authored-by: Barney Gale <barney.gale@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 19, 2024
pythonGH-85168: Use filesystem encoding when converting to/from `file…
72c3c0f 
…` URIs (pythonGH-126852)

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
(cherry picked from commit c9b399f)

Co-authored-by: Barney Gale <barney.gale@gmail.com>

bedevere-app Bot commented Nov 19, 2024

Copy link
Copy Markdown

GH-127039 is a backport of this pull request to the 3.13 branch.

bedevere-app Bot removed the needs backport to 3.13 only security fixes label Nov 19, 2024

bedevere-app Bot commented Nov 19, 2024

Copy link
Copy Markdown

GH-127040 is a backport of this pull request to the 3.12 branch.

bedevere-app Bot removed the needs backport to 3.12 only security fixes label Nov 19, 2024
barneygale added a commit that referenced this pull request Nov 19, 2024
[3.12] GH-85168: Use filesystem encoding when converting to/from `fil…
7735f58 
…e` URIs (GH-126852) (#127040)

GH-85168: Use filesystem encoding when converting to/from `file` URIs (GH-126852)

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
(cherry picked from commit c9b399f)

Co-authored-by: Barney Gale <barney.gale@gmail.com>
barneygale added a commit that referenced this pull request Nov 20, 2024
[3.13] GH-85168: Use filesystem encoding when converting to/from `fil…
746a0c5 
…e` URIs (GH-126852) (#127039)

GH-85168: Use filesystem encoding when converting to/from `file` URIs (GH-126852)

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
(cherry picked from commit c9b399f)

Co-authored-by: Barney Gale <barney.gale@gmail.com>
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
pythonGH-85168: Use filesystem encoding when converting to/from `file…
91b0e5b 
…` URIs (python#126852)

Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
barneygale mentioned this pull request Jul 19, 2025
Merged

serhiy-storchaka commented Aug 31, 2026

Copy link
Copy Markdown
Member

UTF-8 can fail on surrogates. See #156610.

serhiy-storchaka commented Aug 31, 2026

Copy link
Copy Markdown
Member

Opened #156713.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

serhiy-storchaka serhiy-storchaka approved these changes
+1 more reviewer
ruidazeng ruidazeng approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL