bpo-39209: Manage correctly multi-line tokens in interactive mode (GH… · python-emscripten/cpython@b2e281a · GitHub

1. python-emscripten
2. cpython

Repository navigation

• Code
• Pull requests
• Actions
• Projects
• Security and quality
• Insights

Commit b2e281a

Browse filesBrowse the repository at this point in the historyBrowse files

miss-islington

and

pablogsal

authored

bpo-39209: Manage correctly multi-line tokens in interactive mode (pythonGH-17860)

(cherry picked from commit 5ec91f7) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

1 parent 33cb4a6 commit b2e281a
Copy full SHA for b2e281a

3 files changed

+40Lines changed: 40 additions & 0 deletions

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

• Lib/test
  • test_repl.py
• Misc/NEWS.d/next/Core and Builtins
  • 2020-01-06-10-29-16.bpo-39209.QHAONe.rst
• Parser
  • tokenizer.c

Expand file treeCollapse file tree

Open diff view settings

Collapse file

‎Lib/test/test_repl.py‎

Copy file name to clipboardExpand all lines: Lib/test/test_repl.py

+36Lines changed: 36 additions & 0 deletions

Original file line number	Diff line number	Diff line change
@@ -58,5 +58,41 @@ def test_no_memory(self):
58	58	# Exit code 120: Py_FinalizeEx() failed to flush stdout and stderr.
59	59	self.assertIn(p.returncode, (1, 120))
60	60
	61	+ @cpython_only
	62	+ def test_multiline_string_parsing(self):
	63	+ # bpo-39209: Multiline string tokens need to be handled in the tokenizer
	64	+ # in two places: the interactive path and the non-interactive path.
	65	+ user_input = '''\
	66	+ x = """<?xml version="1.0" encoding="iso-8859-1"?>
	67	+ <test>
	68	+ <Users>
	69	+ <fun25>
	70	+ <limits>
	71	+ <total>0KiB</total>
	72	+ <kbps>0</kbps>
	73	+ <rps>1.3</rps>
	74	+ <connections>0</connections>
	75	+ </limits>
	76	+ <usages>
	77	+ <total>16738211KiB</total>
	78	+ <kbps>237.15</kbps>
	79	+ <rps>1.3</rps>
	80	+ <connections>0</connections>
	81	+ </usages>
	82	+ <time_to_refresh>never</time_to_refresh>
	83	+ <limit_exceeded_URL>none</limit_exceeded_URL>
	84	+ </fun25>
	85	+ </Users>
	86	+ </test>"""
	87	+ '''
	88	+ user_input = dedent(user_input)
	89	+ user_input = user_input.encode()
	90	+ p = spawn_repl()
	91	+ with SuppressCrashReport():
	92	+ p.stdin.write(user_input)
	93	+ output = kill_python(p)
	94	+ self.assertEqual(p.returncode, 0)
	95	+
	96	+
61	97	if __name__ == "__main__":
62	98	unittest.main()

Collapse file

‎Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst‎

Copy file name to clipboard

+2Lines changed: 2 additions & 0 deletions

• Display the source diff
• Display the rich diff

Original file line number	Diff line number	Diff line change
@@ -0,0 +1,2 @@
	1	+ Correctly handle multi-line tokens in interactive mode. Patch by Pablo
	2	+ Galindo.

Collapse file

‎Parser/tokenizer.c‎

Copy file name to clipboardExpand all lines: Parser/tokenizer.c

+2Lines changed: 2 additions & 0 deletions

Original file line number	Diff line number	Diff line change
@@ -886,6 +886,7 @@ tok_nextc(struct tok_state *tok)
886	886	size_t start = tok->start - tok->buf;
887	887	size_t oldlen = tok->cur - tok->buf;
888	888	size_t newlen = oldlen + strlen(newtok);
	889	+ Py_ssize_t cur_multi_line_start = tok->multi_line_start - tok->buf;
889	890	char *buf = tok->buf;
890	891	buf = (char *)PyMem_REALLOC(buf, newlen+1);
891	892	tok->lineno++;
@@ -898,6 +899,7 @@ tok_nextc(struct tok_state *tok)
898	899	}
899	900	tok->buf = buf;
900	901	tok->cur = tok->buf + oldlen;
	902	+ tok->multi_line_start = tok->buf + cur_multi_line_start;
901	903	tok->line_start = tok->cur;
902	904	strcpy(tok->buf + oldlen, newtok);
903	905	PyMem_FREE(newtok);

0 commit comments

Comments

0 (0)