PEP 701 – Syntactic formalization of f-strings · Issue #102856 · python/cpython · GitHub

Changes in the C tokenizer
Categorize failing tests
Fix failing tests or modify/remove them as needed
Changes in Python tokenizer

Linked PRs

CC: @lysnikolaou @isidentical @mgmacias95

See this for the latest report on errors from @isidentical

Draft PR for the C tokenizer up: #102855

Things for the cleanup of #102855:

Cleaning up the grammar and the action helpers (the names are still ridiculous and there are multiple rules commented out).
Remove the old parsing code and check that we didn't break anything 😅
Clean/refactor the tokenizer struct (better names, factor stuff into its own structure as needed).
Consider factoring out tok_get_fstring_mode because is a monster.

Ok with #102855 we have the following failing tests:

test_ast
test_cmd_line_script
test_eof
test_exceptions
test_fstring
test_tokenize
test_type_comments
test_unparse

Most of these are updating error messages, line numbers and other stuff but some may have actual bugs so we should check them. Please, mention which ones are you working on so we don't clash with one another.

Working on test_tokenize

Hello, Pablo!
Can I get work on test_ast?
Recently I sent some PR's about this file (for example, #102797). So, I have some experience in that =)

I can work with test_type_comments and test_unparse.

@Eclips4 @ramvikrams wonderful! Just make PRs against my fork!

Report here or ping any of us if you find something that could be a bug (don't just fix the tests blindly because there may be bugs lurking).

@lysnikolaou can you work on cleaning up the grammar + the actions?

@isidentical can you work on cleaning up some of the tokenizer layers? (This is quite a lot so we can probably work together here).

@pablogsal
About test_ast.py
Seems thats like there only a one test will be failed, and how I undestand, that's a bug:

cpython/Lib/test/test_ast.py

Lines 779 to 780 in 7f760c2

    
           with self.assertRaises(SyntaxError): 
        
               ast.parse('f"{x=}"', feature_version=(3, 7))

I think, there's two solutions:

Remove this test, because support of python3.7 will be ended soon.
Now errors raised by tokenizer.c instead of string_parser.c, so as I understand, we should change python_gram, is it right? ( We need access to feature_version, which in tokenizer inaccessible )

2. Now errors raised by tokenizer.c instead of string_parser.c, so as I understand, we should change python_gram, is it right? ( We need access to feature_version, which in tokenizer inaccessible )

Probably we can do this but on the other hand I would prefer to not overcomplicate this so I think (1) is better

@lysnikolaou can you work on cleaning up the grammar + the actions?

Will do!

Also, I can take a look at test_cmd_line_script. Seems easy.

Also, I can take a look at test_cmd_line_script. Seems easy.

All yours!

[3.12] gh-102856: Add changes related to PEP 701 in 3.12 What's New d…

Closing this as we already have a What's New entry, C tokenizer, and Python tokenizer. Let's tackle any small remaining items in separate issues from now on.

Probably we need to alter https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals. @lysnikolaou can you take a go at that?

pythongh-102856: Add missing quote to fix doctest (pythonGH-104852)

gh-102856: Add missing quote to fix doctest (#104852)

Probably we need to alter https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals. @lysnikolaou can you take a go at that?

Sure!

[3.12] gh-102856: Add missing quote to fix doctest (GH-104852) (#104854)

pythongh-102856: Update Formatted string literals docs section after …

pythongh-102856: Update "Formatted string literals" docs section afte…

gh-102856: Update "Formatted string literals" docs section after PEP7…

pythongh-102856: Update "Formatted string literals" docs section afte…

[3.12] gh-102856: Update "Formatted string literals" docs section aft…

Ah great! I didn't know someone would continue PEP 536. Happy that it's happening!

Shouldn't PEP 536 be mentioned in this one?

Ah great! I didn't know someone would continue PEP 536. Happy that it's happening!

Shouldn't PEP 536 be mentioned in this one?

It's mentioned in the PEP:

https://peps.python.org/pep-0701/

Btw as heads up: We don't monitor normally closed issues so is very likely that people won't answer to comments when the issue is closed :)

Thanks! Seems like I missed the mention then. Perfect!

PEP 701 – Syntactic formalization of f-strings · Issue #102856 · python/cpython · GitHub

Repository navigation

PEP 701 – Syntactic formalization of f-strings #102856

Linked PRs

52 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	with self.assertRaises(SyntaxError):
	ast.parse('f"{x=}"', feature_version=(3, 7))

PEP 701 – Syntactic formalization of f-strings · Issue #102856 · python/cpython · GitHub

Repository navigation

PEP 701 – Syntactic formalization of f-strings #102856

Description

Linked PRs

Activity

pablogsal commented on Mar 20, 2023

pablogsal commented on Mar 20, 2023

pablogsal commented on Mar 20, 2023

pablogsal commented on Mar 20, 2023

pablogsal commented on Mar 20, 2023

mgmacias95 commented on Mar 20, 2023

Eclips4 commented on Mar 21, 2023

ramvikrams commented on Mar 21, 2023

pablogsal commented on Mar 21, 2023

pablogsal commented on Mar 21, 2023

Eclips4 commented on Mar 21, 2023

pablogsal commented on Mar 21, 2023

lysnikolaou commented on Mar 21, 2023

Eclips4 commented on Mar 21, 2023

pablogsal commented on Mar 21, 2023

52 remaining items

pablogsal commented on May 24, 2023

lysnikolaou commented on May 24, 2023

flying-sheep commented on Oct 2, 2023

pablogsal commented on Oct 3, 2023

flying-sheep commented on Oct 3, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions