[3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields - #154867
Merged
serhiy-storchaka merged 3 commits intoAug 3, 2026
Merged
[3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields#154867serhiy-storchaka merged 3 commits into
serhiy-storchaka merged 3 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
The regular expressions which look for a quoted field matched its body lazily, so a closing quote which was not followed by a delimiter was retried with every following quote, to the end of the sample. Match the body possessively instead: it ends at the first quote which is not doubled, as it does for a reader.
# Conflicts: # Lib/test/test_csv.py
sniff() now replaces \r\n and \r with \n before these regular expressions are used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
GH-155117 is a backport of this pull request to the 3.14 branch. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
GH-155118 is a backport of this pull request to the 3.13 branch. |
Sorry, something went wrong.
Yhg1s
pushed a commit
that referenced
this pull request
Aug 4, 2026
…H-154867) (#155118) [3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) The regular expressions which look for a quoted field matched its body lazily, so a closing quote which was not followed by a delimiter was retried with every following quote, to the end of the sample. Match the body possessively instead: it ends at the first quote which is not doubled, as it does for a reader. (cherry picked from commit 476fb09) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
|
GH-155166 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
serhiy-storchaka
added a commit
that referenced
this pull request
Aug 4, 2026
…H-154867) (GH-155117) The regular expressions which look for a quoted field matched its body lazily, so a closing quote which was not followed by a delimiter was retried with every following quote, to the end of the sample. Match the body possessively instead: it ends at the first quote which is not doubled, as it does for a reader. (cherry picked from commit 476fb09) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 4, 2026
deadlovelll
pushed a commit
to deadlovelll/cpython
that referenced
this pull request
Aug 5, 2026
…ields (pythonGH-154867) (pythonGH-155117) The regular expressions which look for a quoted field matched its body lazily, so a closing quote which was not followed by a delimiter was retried with every following quote, to the end of the sample. Match the body possessively instead: it ends at the first quote which is not doubled, as it does for a reader. (cherry picked from commit 476fb09) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 11, 2026
Copy link
Copy Markdown
|
GH-155538 is a backport of this pull request to the 3.11 branch. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
GH-155539 is a backport of this pull request to the 3.10 branch. |
Sorry, something went wrong.
pablogsal
pushed a commit
that referenced
this pull request
Aug 11, 2026
pablogsal
pushed a commit
that referenced
this pull request
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four regular expressions which look for a quoted field match its body lazily. A closing quote which is not followed by a delimiter is therefore retried with every following quote, to the very end of the sample, and findall() repeats that from every start position -- so the search is quadratic. On a single column of quoted fields none of them ever matches, so the whole cost is a failing scan.
The body now ends at the first quote which is not doubled, which is the only closing quote a reader would accept, and is matched possessively.
A minimal reproducer needs no quoted fields at all -- ',"x' * n is quadratic too, because two of the four patterns are anchored on a delimiter rather than a line.
Being unambiguous is what makes it linear, so a few matches necessarily change: 16 of 560 files in the CSVsniffer corpora. Against their ground truth the net is positive -- delimiter 73.3% -> 74.5%, quotechar 81.3% -> 81.1%, doublequote unchanged.
main is not affected: the sniffer was rewritten there in gh-83273.