GH-119496: accept UTF-8 BOM in .pth files by ncoghlan · Pull Request #119503 · python/cpython · GitHub

Out-File -Encoding utf8 in Windows Powershell 5.1 emits
UTF-8 with a BOM marker, which the regular utf-8 codec
decodes incorrectly. utf-8-sig accepts a BOM, but also
works correctly without one.

Issue: Improve handling of UTF-16-LE encoded .pth files #119496

`Out-File -Encoding utf8` in Windows Powershell 5.1 emits UTF-8 with a BOM maker, which the regular `utf-8` codec decodes incorrectly. `utf-8-sig` accepts a BOM, but also works correctly without one.

I flagged this as skipping news, as it seems like part of the not-yet-released gh-77102 changes to me.

PowerShell 5.1 is very old and will be EOL in 2025. I am not sure about adding some support for it is worth enough.
But "as we do in source files" seems good reason. LGTM.

+1 from me. Any time we know we're decoding the start of a file, there's no harm in using utf-8-sig.

I wouldn't use it anywhere we write to a file. It's a bit unfortunate we don't have an encoding that will silently drop a BOM at the start of a buffer without also adding one going the other direction. (Or an encoding that will detect a BOM and switch to UTF-8, or else use 'locale', which is the typical way Windows apps handle this.)

Co-authored-by: Inada Naoki <songofacandy@gmail.com>

Thanks @ncoghlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

`Out-File -Encoding utf8` and similar commands in Windows Powershell 5.1 emit UTF-8 with a BOM marker, which the regular `utf-8` codec decodes incorrectly. `utf-8-sig` accepts a BOM, but also works correctly without one. This change also makes .pth files match the way Python source files are handled. (cherry picked from commit bf5b646) Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com> Co-authored-by: Inada Naoki <songofacandy@gmail.com>

GH-119508 is a backport of this pull request to the 3.13 branch.

GH-119509 is a backport of this pull request to the 3.12 branch.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 Refleaks NoGIL 3.x has failed when building commit bf5b646.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1368/builds/1058) and take a look at the build logs.
Check if the failure is related to this commit (bf5b646) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1368/builds/1058

Failed tests:

test_free_threading

Test leaking resources:

test_free_threading: memory blocks
test_free_threading: references

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 7, done.        
remote: Counting objects:  14% (1/7)        
remote: Counting objects:  28% (2/7)        
remote: Counting objects:  42% (3/7)        
remote: Counting objects:  57% (4/7)        
remote: Counting objects:  71% (5/7)        
remote: Counting objects:  85% (6/7)        
remote: Counting objects: 100% (7/7)        
remote: Counting objects: 100% (7/7), done.        
remote: Compressing objects:  25% (1/4)        
remote: Compressing objects:  50% (2/4)        
remote: Compressing objects:  75% (3/4)        
remote: Compressing objects: 100% (4/4)        
remote: Compressing objects: 100% (4/4), done.        
remote: Total 4 (delta 3), reused 1 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'bf5b6467f8cc06759f3396ab1a8ad64fe7d1db2e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at bf5b6467f8 GH-119496: accept UTF-8 BOM in .pth files (GH-119503)
Switched to and reset branch 'main'

In file included from ./Modules/tkappinit.c:17:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33In file included from ./Modules/_tkinter.c:52:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:        int (*synchandler)();   /* Synchronization handler */33
:                          ^ 
warning:                            voida function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]

        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: 
warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
9 warnings generated.
9 warnings generated.

make: *** [buildbottest] Error 2

`Out-File -Encoding utf8` and similar commands in Windows Powershell 5.1 emit UTF-8 with a BOM marker, which the regular `utf-8` codec decodes incorrectly. `utf-8-sig` accepts a BOM, but also works correctly without one. This change also makes .pth files match the way Python source files are handled. Co-authored-by: Inada Naoki <songofacandy@gmail.com>

ncoghlan added 2 commits May 24, 2024 22:21

Accept UTF-8 BOM in .pth files

77a45b0

`Out-File -Encoding utf8` in Windows Powershell 5.1 emits UTF-8 with a BOM maker, which the regular `utf-8` codec decodes incorrectly. `utf-8-sig` accepts a BOM, but also works correctly without one.

Update site.py

1c2b313

ncoghlan requested a review from methane May 24, 2024 12:25

bedevere-app Bot added the awaiting core review label May 24, 2024

bedevere-app Bot mentioned this pull request May 24, 2024

Improve handling of UTF-16-LE encoded .pth files #119496

Open

ncoghlan added skip news needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels May 24, 2024

methane reviewed May 24, 2024

View reviewed changes

Comment thread Lib/site.py Outdated

methane approved these changes May 24, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels May 24, 2024

Simpliy decode call

1828c45

Co-authored-by: Inada Naoki <songofacandy@gmail.com>

ncoghlan enabled auto-merge (squash) May 24, 2024 14:08

ncoghlan merged commit bf5b646 into python:main May 24, 2024

bedevere-app Bot removed the awaiting merge label May 24, 2024

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label May 24, 2024

bedevere-app Bot removed the needs backport to 3.12 only security fixes label May 24, 2024

ncoghlan mentioned this pull request May 24, 2024

.pth files cannot contain folders with utf-8 names #77102

Closed

GH-119496: accept UTF-8 BOM in .pth files by ncoghlan · Pull Request #119503 · python/cpython · GitHub

Uh oh!

Uh oh!

GH-119496: accept UTF-8 BOM in .pth files - #119503

GH-119496: accept UTF-8 BOM in .pth files#119503
ncoghlan merged 3 commits into
python:mainfrom
ncoghlan:gh-119496-accept-utf8-bom-in-pth-files

ncoghlan commented May 24, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

ncoghlan commented May 24, 2024 •
edited
Loading

Uh oh!

Uh oh!

methane commented May 24, 2024

Uh oh!

Uh oh!

zooba commented May 24, 2024

Uh oh!

miss-islington-app Bot commented May 24, 2024

Uh oh!

bedevere-app Bot commented May 24, 2024

Uh oh!

bedevere-app Bot commented May 24, 2024

Uh oh!

bedevere-bot commented May 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GH-119496: accept UTF-8 BOM in .pth files by ncoghlan · Pull Request #119503 · python/cpython · GitHub

Uh oh!

Uh oh!

GH-119496: accept UTF-8 BOM in .pth files - #119503

GH-119496: accept UTF-8 BOM in .pth files#119503ncoghlan merged 3 commits intopython:mainpython/cpython:mainfrom ncoghlan:gh-119496-accept-utf8-bom-in-pth-filesncoghlan/cpython:gh-119496-accept-utf8-bom-in-pth-filesCopy head branch name to clipboard

Conversation

ncoghlan commented May 24, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncoghlan commented May 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

methane commented May 24, 2024

Uh oh!

Uh oh!

zooba commented May 24, 2024

Uh oh!

miss-islington-app Bot commented May 24, 2024

Uh oh!

bedevere-app Bot commented May 24, 2024

Uh oh!

bedevere-app Bot commented May 24, 2024

Uh oh!

bedevere-bot commented May 24, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GH-119496: accept UTF-8 BOM in .pth files#119503
ncoghlan merged 3 commits into
python:mainfrom
ncoghlan:gh-119496-accept-utf8-bom-in-pth-files

ncoghlan commented May 24, 2024 •
edited by bedevere-app Bot
Loading

ncoghlan commented May 24, 2024 •
edited
Loading