FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-34294: re module, fix wrong capturing groups in rare cases. (GH-1… · native-api/cpython@4a7f44a · GitHub

  1. native-api
  2. cpython

Repository navigation

Commit 4a7f44a

Browse files
animalize
authored and
serhiy-storchaka
committed
bpo-34294: re module, fix wrong capturing groups in rare cases. (pythonGH-11546)
Need to reset capturing groups between two SRE(match) callings in loops, this fixes wrong capturing groups in rare cases. Also add a missing index in re.rst.
1 parent 02c04f2 commit 4a7f44a

5 files changed

Lines changed: 49 additions & 1 deletion

File tree

‎Doc/library/re.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,8 @@ The special characters are:
371371
``(?#...)``
372372
A comment; the contents of the parentheses are simply ignored.
373373

374+
.. index:: single: (?=; in regular expressions
375+
374376
``(?=...)``
375377
Matches if ``...`` matches next, but doesn't consume any of the string. This is
376378
called a :dfn:`lookahead assertion`. For example, ``Isaac (?=Asimov)`` will match

‎Lib/test/test_re.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2067,6 +2067,40 @@ def test_bug_29444(self):
20672067
self.assertEqual(m.group(), b'xyz')
20682068
self.assertEqual(m2.group(), b'')
20692069

2070+
def test_bug_34294(self):
2071+
# Issue 34294: wrong capturing groups
2072+
2073+
# exists since Python 2
2074+
s = "a\tx"
2075+
p = r"\b(?=(\t)|(x))x"
2076+
self.assertEqual(re.search(p, s).groups(), (None, 'x'))
2077+
2078+
# introduced in Python 3.7.0
2079+
s = "ab"
2080+
p = r"(?=(.)(.)?)"
2081+
self.assertEqual(re.findall(p, s),
2082+
[('a', 'b'), ('b', '')])
2083+
self.assertEqual([m.groups() for m in re.finditer(p, s)],
2084+
[('a', 'b'), ('b', None)])
2085+
2086+
# test-cases provided by issue34294, introduced in Python 3.7.0
2087+
p = r"(?=<(?P<tag>\w+)/?>(?:(?P<text>.+?)</(?P=tag)>)?)"
2088+
s = "<test><foo2/></test>"
2089+
self.assertEqual(re.findall(p, s),
2090+
[('test', '<foo2/>'), ('foo2', '')])
2091+
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
2092+
[{'tag': 'test', 'text': '<foo2/>'},
2093+
{'tag': 'foo2', 'text': None}])
2094+
s = "<test>Hello</test><foo/>"
2095+
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
2096+
[{'tag': 'test', 'text': 'Hello'},
2097+
{'tag': 'foo', 'text': None}])
2098+
s = "<test>Hello</test><foo/><foo/>"
2099+
self.assertEqual([m.groupdict() for m in re.finditer(p, s)],
2100+
[{'tag': 'test', 'text': 'Hello'},
2101+
{'tag': 'foo', 'text': None},
2102+
{'tag': 'foo', 'text': None}])
2103+
20702104

20712105
class PatternReprTests(unittest.TestCase):
20722106
def check(self, pattern, expected):

‎Misc/NEWS.d/next/Library/2019-01-14-11-53-10.bpo-34294.3JFdg2.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
re module, fix wrong capturing groups in rare cases. :func:`re.search`,
2+
:func:`re.findall`, :func:`re.sub` and other functions that scan through
3+
string looking for a match, should reset capturing groups between two match
4+
attempts. Patch by Ma Lin.

‎Modules/_sre.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,7 +340,7 @@ _sre_unicode_tolower_impl(PyObject *module, int character)
340340
LOCAL(void)
341341
state_reset(SRE_STATE* state)
342342
{
343-
/* FIXME: dynamic! */
343+
/* state->mark will be set to 0 in SRE_OP_MARK dynamically. */
344344
/*memset(state->mark, 0, sizeof(*state->mark) * SRE_MARK_SIZE);*/
345345

346346
state->lastmark = -1;

‎Modules/sre_lib.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1363,6 +1363,10 @@ SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int toplevel)
13631363
return ret; /* should never get here */
13641364
}
13651365

1366+
/* need to reset capturing groups between two SRE(match) callings in loops */
1367+
#define RESET_CAPTURE_GROUP() \
1368+
do { state->lastmark = state->lastindex = -1; } while (0)
1369+
13661370
LOCAL(Py_ssize_t)
13671371
SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
13681372
{
@@ -1440,6 +1444,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
14401444
if (status != 0)
14411445
return status;
14421446
++ptr;
1447+
RESET_CAPTURE_GROUP();
14431448
}
14441449
return 0;
14451450
}
@@ -1487,6 +1492,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
14871492
/* close but no cigar -- try again */
14881493
if (++ptr >= end)
14891494
return 0;
1495+
RESET_CAPTURE_GROUP();
14901496
}
14911497
i = overlap[i];
14921498
} while (i != 0);
@@ -1510,6 +1516,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
15101516
if (status != 0)
15111517
break;
15121518
ptr++;
1519+
RESET_CAPTURE_GROUP();
15131520
}
15141521
} else {
15151522
/* general case */
@@ -1520,6 +1527,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
15201527
state->must_advance = 0;
15211528
while (status == 0 && ptr < end) {
15221529
ptr++;
1530+
RESET_CAPTURE_GROUP();
15231531
TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
15241532
state->start = state->ptr = ptr;
15251533
status = SRE(match)(state, pattern, 0);

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL