gh-98401: Invalid escape sequences emits SyntaxWarning by vstinner · Pull Request #99011 · python/cpython · GitHub

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning.

The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions.
Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

Issue: Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Previous attempt in 2018:

Commit: 6543912
Serhiy only changed the Python parser, not codecs: rationale
The new SyntaxWarning was seen in multiple projects: docutils (fixed), bottle, SymPy (bug in a docstring)
SyntaxWarning are only emitted once, and only when a Python file is parsed: when a cached PYC file is used, not warning is emitted
Serhiy reverted his change

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

$ PYTHONWARNINGS=error::SyntaxWarning ./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

This issue mostly hit code defining regular expressions. Example in BeautifulSoup 3.2.2:

$ pycodestyle --select W605 $(find -name "*.py")
./BeautifulSoup.py:455:52: W605 invalid escape sequence '\d'
./BeautifulSoup.py:455:73: W605 invalid escape sequence '\w'

./BeautifulSoup.py:574:47: W605 invalid escape sequence '\d'
./BeautifulSoup.py:574:66: W605 invalid escape sequence '\w'

./BeautifulSoup.py:1080:38: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1589:36: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1933:16: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1933:46: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1935:35: W605 invalid escape sequence '\s'

Examples of code, re.compile() calls:

    BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
                                           + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
                                           + ")")
(...)
                      (re.compile('<!\s+([^<>]*)>'),
(...)
        xml_encoding_match = re.compile(
            '^<\?.*encoding=[\'"](.*?)[\'"].*\?>').match(xml_data)

SyntaxWarning is not related to codecs. It only should be emitted by the compiler.

Which warning should be emitted if SyntaxWarning is not the best choice? UnicodeWarning?

Does UnicodeWarning make sense for codecs.escape_decode()?

DeprecationWarning, as for all other deprecated features.

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

I ran PYTHONWARNINGS=error::SyntaxWarning make regen-all PYTHON_FOR_REGEN=./python: it works as expected.

@serhiy-storchaka:

Perhaps SyntaxWarning could be raised for invalid octals too. It is a more obvious error, and less common, so it may have lesser deprecation period.

Ok, I updated my PR: Use SyntaxWarning for invalid octal sequence.

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning. Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError. codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning. * The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions. * Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

Hum, I messed up my PR so I squashed two commits and I fixed my PR:

invalid octal sequence and invalid escape sequence now both emit SyntaxWarning (and will emit SyntaxError in the future)
codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged

I also mentioned the future convertion to SyntaxError in the doc (What's New / NEWS entries).

@mdickinson @serhiy-storchaka @hugovk: Would you mind to review my PR?

Did you remove all pyc files?

find -name '*.py[co]' -exec rm -rf '{}' +

Let me try these commands:

export PYTHONWARNINGS=error::SyntaxWarning 
git clean -fdx
./configure --with-pydebug CFLAGS=-O0
make
make regen-all PYTHON_FOR_REGEN=./python
make

./python -c 's = "\477"'

The last command fails as expected with:

  File "<string>", line 1
    s = "\477"
        ^^^^^^
SyntaxError: invalid octal escape sequence '\477'

Oops, test_string_literals didn't work when run with ./python -W error::SyntaxWarning: it's now fixed.

I ran the test suite with:

export PYTHONWARNINGS=error::SyntaxWarning 
./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

Note: I would prefer to run the whole test suite with -Werror, but it fails even without this PR :-( Python currently emits many deprecation warnings for example :-(

Fixes this warning experienced with Python 3.12 (python/cpython#98401, python/cpython#99011): faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.' email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',

Use r-strings for all regular expressions. Fixes these warnings experienced with Python 3.12 (python/cpython#98401, python/cpython#99011, https://docs.python.org/3/whatsnew/3.12.html#other-language-changes point 2): run_tests.py:200: SyntaxWarning: invalid escape sequence '\d' FINAL_LINE_RE = re.compile('status=(\d+)$') run_tests.py:441: SyntaxWarning: invalid escape sequence '\*' re.match('^\* daemon .+ \*$', line) or line == ''): Change-Id: I71ddfb1a2ca62654378ae67a99e9aeb4ce7b7394 Reviewed-on: https://chromium-review.googlesource.com/c/crashpad/crashpad/+/6254063 Commit-Queue: Mark Mentovai <mark@chromium.org> Reviewed-by: Nico Weber <thakis@chromium.org>

All other regular expressions in the file already used r-strings, but this newer one added in beba95a did not. Fixes this warning experienced with Python ≥3.12 (python/cpython#98401, python/cpython#99011, python/cpython@a60ddd31be7f): bisect-builds.py:1260: SyntaxWarning: invalid escape sequence '\.' if not re.search("\.apks?$", apk_name): Change-Id: Ia8f1981efde3b14d5c36c3dfc2bca7737fd4cb89 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6647645 Commit-Queue: Mark Mentovai <mark@chromium.org> Reviewed-by: Kuan Huang <kuanhuang@chromium.org> Cr-Commit-Position: refs/heads/main@{#1474605}

Since 61fad56 (https://chromium-review.googlesource.com/c/5848450, 2024-09-11), autoninja under Python 3.12 presents these warnings: ``` …/autoninja.py:73: SyntaxWarning: invalid escape sequence '\s' m = re.match('instance\s*=\s*projects/([^/]*)/instances/.*', line) …/autoninja.py:92: SyntaxWarning: invalid escape sequence '\s' m = re.match('SISO_PROJECT=\s*(\S*)\s*', line) ``` This warning appears in Python 3.12 ([1], [2], [3]). '\s' and '\S' are not valid escape sequences in strings. r'\s' and r'\S' are valid in regular expressions, but outside of raw strings, they would need to be written as '\\s' and '\\S'. There is no reason to not use raw strings in this case, so the new regular expression pattern strings introduced in 61fad56 are changed to raw strings. [1] https://docs.python.org/3/whatsnew/3.12.html#:~:text=A%20backslash%2Dcharacter%20pair%20that%20is%20not%20a%20valid%20escape%20sequence%20now%20generates%20a%20SyntaxWarning%2C%20instead%20of%20DeprecationWarning. [2] python/cpython#98401 [3] python/cpython#99011 Bug: b/364318216 Change-Id: I0f237976fe9c39208541ae78205f5bdbf126fa82 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5859159 Commit-Queue: Mark Mentovai <mark@chromium.org> Reviewed-by: Philipp Wollermann <philwo@google.com> Auto-Submit: Mark Mentovai <mark@chromium.org>

vstinner requested review from lysnikolaou and pablogsal as code owners November 2, 2022 16:21

bedevere-bot mentioned this pull request Nov 2, 2022

Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Closed

bedevere-bot added the awaiting core review label Nov 2, 2022

This was referenced Nov 2, 2022

gh-98401: Reject invalid escape sequences in strings #98404

Closed

Raise non-silent warning for invalid escape sequences #77093

Closed

lysnikolaou reviewed Nov 2, 2022

View reviewed changes

Comment thread Parser/string_parser.c Outdated

serhiy-storchaka reviewed Nov 3, 2022

View reviewed changes

serhiy-storchaka approved these changes Nov 3, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Nov 3, 2022

Fix test_string_literals

d24f799

vstinner merged commit a60ddd3 into python:main Nov 3, 2022

bedevere-bot removed the awaiting merge label Nov 3, 2022

vstinner deleted the syntax_warning branch November 3, 2022 16:53

chenhengqi mentioned this pull request Oct 31, 2023

tools/*.py: fix invalid escape sequences in regexps iovisor/bcc#4789

Merged

benoit9126 mentioned this pull request Nov 9, 2023

Syntax warning because of "\S" in Python 3.12 keygen-sh/py-machineid#9

Closed

terryjreedy mentioned this pull request May 23, 2024

Invalid escape sequences raise SyntaxWarning #119458

Closed

bebound mentioned this pull request Sep 14, 2024

gh-123858: Improve Doc: SyntaxWarning is emitted during bytecode generation #122844

Merged

braingram mentioned this pull request Nov 13, 2025

ignore reverted warning issued in python 3.11 spacetelescope/romancal#2059

Merged

10 tasks

coderabbitai Bot mentioned this pull request Jul 13, 2026

♻️ Centralize and structure code generation/infrastructure munich-quantum-toolkit/qecc#751

Merged

9 tasks

This was referenced Jul 27, 2026

Feature units concrete ch8 Blueprints-org/blueprints#1093

Merged

Remove obsolete docs and simplify the homepage hero surajmandalcell/subscription-proxy-inator#18

Closed

coderabbitai Bot mentioned this pull request Aug 5, 2026

Fix Gutenberg contents-list ingest and green up CI on Python 3.14 gilesknap/somnia#1

Merged

coderabbitai Bot mentioned this pull request Aug 16, 2026

fix: keep backslashes in the generated SQL constants rayakame/sqlc-gen-better-python#262

Merged

                           b = bytes([i])
                           if b not in b'abfnrtvx':
-                              with self.assertWarns(DeprecationWarning):
+                              with self.assertWarns(SyntaxWarning):

gh-98401: Invalid escape sequences emits SyntaxWarning by vstinner · Pull Request #99011 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-98401: Invalid escape sequences emits SyntaxWarning - #99011

gh-98401: Invalid escape sequences emits SyntaxWarning#99011vstinner merged 2 commits intopython:mainpython/cpython:mainfrom vstinner:syntax_warningCopy head branch name to clipboard

Conversation

vstinner commented Nov 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vstinner commented Nov 2, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

Uh oh!

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

serhiy-storchaka commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-98401: Invalid escape sequences emits SyntaxWarning#99011
vstinner merged 2 commits into
python:mainfrom
vstinner:syntax_warning

vstinner commented Nov 2, 2022 •
edited
Loading