FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively by pablogsal · Pull Request #105070 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively - #105070

Merged
pablogsal merged 7 commits into
python:mainfrom
pablogsal:tokenizer_iter
May 30, 2023
ConversationCommits (7)ChecksFiles changed
Merged

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively#105070
pablogsal merged 7 commits into
python:mainfrom
pablogsal:tokenizer_iter

Conversation

pablogsal commented May 29, 2023 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

bedevere-bot mentioned this pull request May 29, 2023
Closed
pablogsal mentioned this pull request May 29, 2023
Closed
pablogsal force-pushed the tokenizer_iter branch 3 times, most recently from 0a661c4 to 713379f Compare May 29, 2023 20:29
pablogsal marked this pull request as ready for review May 29, 2023 20:30
pablogsal requested a review from lysnikolaou as a code owner May 29, 2023 20:30
pablogsal requested a review from lysnikolaou as a code owner May 29, 2023 20:30
pablogsal commented May 29, 2023
View reviewed changes
Comment thread Lib/test/test_tokenize.py
def get_tokens(string):
return list(_generate_tokens_from_c_tokenizer(string))

self.assertRaises(SyntaxError, get_tokens, "(1+2]")

pablogsal May 29, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This was bothering me 😅

pablogsal commented May 29, 2023
View reviewed changes
Comment thread Lib/tokenize.py
This has the same API as tokenize(), except that it expects the *readline*
callable to return str objects instead of bytes.
"""
def _gen():

pablogsal May 29, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Now that we are taking callables all of these can go :)

pythongh-105069: Add a readline-like callable to the tokenizer to con…
7caac01 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal commented May 30, 2023
View reviewed changes
Comment thread Lib/inspect.py
add(string)
if (string == ','):
add(' ')
clean_signature = ''.join(text).strip()

pablogsal May 30, 2023 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This change is because for some reason the inspect module is relying on the fact that if lines yielded by the generator do not end in \n then they are concatenated together, which is wrong because the contract says "should yield one line at a time" so if the line doesn't end in newline we add one always.

lysnikolaou reviewed May 30, 2023
View reviewed changes

lysnikolaou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Great job! 💯

A couple of comments and it's good to go!

Comment thread Parser/tokenizer.c
Comment thread Lib/tokenize.py Outdated
pablogsal added 2 commits May 30, 2023 17:12
fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-like call…
0935371 
…able to the tokenizer to consume input iteratively
fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-li…
9990b7e 
…ke callable to the tokenizer to consume input iteratively

pablogsal commented May 30, 2023

Copy link
Copy Markdown
Member Author

Fixed the problems and added another test.

@lysnikolaou ready for another review!

pablogsal commented May 30, 2023

Copy link
Copy Markdown
Member Author

CC: @mgmacias95 wanna make a review?

fixup! fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a read…
0598127 
…line-like callable to the tokenizer to consume input iteratively
mgmacias95 approved these changes May 30, 2023
View reviewed changes

mgmacias95 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM

pablogsal merged commit 9216e69 into python:main May 30, 2023
pablogsal deleted the tokenizer_iter branch May 30, 2023 21:43

miss-islington commented May 30, 2023

Copy link
Copy Markdown
Contributor

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington commented May 30, 2023

Copy link
Copy Markdown
Contributor

Sorry @pablogsal, I had trouble checking out the 3.12 backport branch.
Please retry by removing and re-adding the "needs backport to 3.12" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 9216e69a87d16d871625721ed5a8aa302511f367 3.12

pablogsal added needs backport to 3.12 only security fixes and removed needs backport to 3.12 only security fixes labels May 30, 2023

miss-islington commented May 30, 2023

Copy link
Copy Markdown
Contributor

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 30, 2023
pythongh-105069: Add a readline-like callable to the tokenizer to con…
6de88f2 
…sume input iteratively (pythonGH-105070)

(cherry picked from commit 9216e69)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

bedevere-bot commented May 30, 2023

Copy link
Copy Markdown

GH-105119 is a backport of this pull request to the 3.12 branch.

bedevere-bot removed the needs backport to 3.12 only security fixes label May 30, 2023
pablogsal added a commit that referenced this pull request May 31, 2023
[3.12] gh-105069: Add a readline-like callable to the tokenizer to co…
c687946 
…nsume input iteratively (GH-105070) (#105119)

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070)
(cherry picked from commit 9216e69)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
erlend-aasland mentioned this pull request Jun 6, 2023
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
1 more reviewer
mgmacias95 mgmacias95 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

pablogsal

Labels

awaiting core review skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

Back | FazBrowse Home | New Git URL