gh-116022: Improve `repr()` of AST nodes by tomasr8 · Pull Request #117046 · python/cpython · GitHub

Based on the original issue the new repr() has:

limited depth: nodes which are to deep are represented as Node(...)
compressed list/tuple fields (e.g. body): more than 2 items are shown as [repr(first), ..., repr(last)]

Examples (output is formatted to be more readable):

>>> import ast
>>> ast.parse("x = 3")
Module(
    body=[Assign(targets=[Name(...)], value=Constant(...), type_comment=None)],
    type_ignores=[]
)


>>> ast.parse('x=7; y=4; c,z=3,4')
Module(
    body=[
        Assign(targets=[Name(...)], value=Constant(...), type_comment=None),
        ...,
        Assign(targets=[Tuple(...)], value=Tuple(...), type_comment=None)
    ],
    type_ignores=[]
)


>>> from pathlib import Path
>>> import typing
>>> ast.parse(Path(typing.__file__).read_text())
Module(
    body=[
        Expr(value=Constant(...)),
        ...,
        FunctionDef(
            name="__getattr__",
            args=arguments(...),
            body=[Expr(...), ..., Return(...)],
            decorator_list=[],
            returns=None,
            type_comment=None,
            type_params=[]
        )
    ],
    type_ignores=[]
)

Happy to take feedback :)

Issue: Improve repr() of AST nodes #116022

Co-authored-by: AN Long <aisk@users.noreply.github.com>

Thank you, and sorry I missed this PR!

I think your approach makes sense but there's a number of small things missing in the C code, and we need good unit test coverage for the new code.

Thanks for taking a look!

I think your approach makes sense

If you had a different approach in mind, I'm happy to change it. This is what makes sense to me, but maybe it's not the best way to go about it..

I've been busy with work lately, but I'll try to add more tests to the PR in the coming days :)

I think this should also be documented in the ast module (a versionchanged note saying that the repr() of AST nodes now includes the values of fields) and in the What's New for 3.13. (Or 3.14 if we don't get this ready before Tuesday.)

I moved the tests to a separate data folder and added a CLI toggle to automatically regenerate them with ./python Lib/test/test_ast.py --snapshot-update. This is basically snapshot testing similar to how it's done in pytest for example. unittest unfortunately doesn't have this capability so it must be implemented separately in each module :/

The advantage of snapshot testing is that we don't need to write the test cases manually - both the initial tests and whenever they need to be updated. You can just run ./python Lib/test/test_ast.py --snapshot-update and compare the diff.

@ambv In case you have some thoughts on this

^ fixed the conflicts in tests

Looking pretty good!

You can DECREF name and value earlier so you don't have to repeat this. Looks like value is no longer needed after line 1602, so you could DECREF it there. name could be DECREFed after line 1623.

Indeed, I move those DECREFs up

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

Thanks for pointing that out, should be fixed now :)

There's some CI failures related to conversion warnings, I think there are some changes landing soon around that so I'll wait for now.

Thanks @JelleZijlstra and everyone else for reviews and help on getting this done!

Co-authored-by: AN Long <aisk@users.noreply.github.com> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

tomasr8 added 2 commits March 19, 2024 22:07

Improve repr() of AST nodes

7461655

Add news entry

447261b

tomasr8 requested a review from isidentical as a code owner March 19, 2024 21:29

bedevere-app Bot added the awaiting review label Mar 19, 2024

bedevere-app Bot mentioned this pull request Mar 19, 2024

Improve repr() of AST nodes #116022

Closed

tomasr8 commented Mar 19, 2024

View reviewed changes

Comment thread Python/Python-ast.c Outdated

aisk reviewed Mar 20, 2024

View reviewed changes

Comment thread Parser/asdl_c.py Outdated

Comment thread Misc/NEWS.d/next/Core and Builtins/2024-03-19-22-21-22.gh-issue-116022.iyHENN.rst Outdated

tomasr8 and others added 5 commits March 20, 2024 19:04

Improve news entry

74759bc

Co-authored-by: AN Long <aisk@users.noreply.github.com>

Follow PEP7

4ffc1fb

Co-authored-by: AN Long <aisk@users.noreply.github.com>

Fix test

6094100

Regenerate files

5cde4fc

Merge branch 'main' into ast-repr

00133b8

JelleZijlstra reviewed Apr 24, 2024

View reviewed changes

tomasr8 added 5 commits May 2, 2024 19:53

Fix missing whitespace

e1b7643

Check return values

0308982

Check for retval < 0 rather than == -1

3f2bf3f

Fix missing whitespace

4386a5a

Simplify code

af3b2a3

JelleZijlstra reviewed May 2, 2024

View reviewed changes

Comment thread Parser/asdl_c.py Outdated

JelleZijlstra reviewed May 2, 2024

View reviewed changes

Comment thread Parser/asdl_c.py Outdated

Comment thread Parser/asdl_c.py Outdated

Comment thread Parser/asdl_c.py Outdated

jeremyhylton self-requested a review May 23, 2024 17:59

devdanzin reviewed Jun 3, 2024

View reviewed changes

Comment thread Parser/asdl_c.py Outdated

JelleZijlstra and others added 7 commits June 3, 2024 03:36

Update Parser/asdl_c.py

1aaa1c0

Merge branch 'main' into ast-repr

06e4657

Fix leaked references

df70943

Add an assertion

ed89975

Simplify code

d8b3bff

Add tests

cfd6cdf

Use PyUnicodeWriter instead of string concatenation

f022378

Add test folder to Makefile

44b3303

tomasr8 requested a review from erlend-aasland as a code owner July 14, 2024 19:19

erlend-aasland removed their request for review July 15, 2024 12:05

Switch back to the private API

0b52a65

JelleZijlstra self-requested a review July 17, 2024 13:54

Merge branch 'main' into ast-repr

0b19d2e

vstinner reviewed Aug 5, 2024

View reviewed changes

Comment thread Makefile.pre.in Outdated

tomasr8 added 2 commits August 12, 2024 15:58

Move test data inside test_ast

2ed3f99

Merge branch 'main' into ast-repr

c94c197

JelleZijlstra reviewed Aug 14, 2024

View reviewed changes

tomasr8 added 2 commits August 14, 2024 20:29

Fix tests

6ca6b5b

Apply review suggestions

df34a04

Merge branch 'main' into ast-repr

ab823cc

JelleZijlstra approved these changes Sep 17, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Sep 17, 2024

JelleZijlstra self-assigned this Sep 17, 2024

Merge branch 'main' into ast-repr

abd35b0

JelleZijlstra merged commit 21d2a9a into python:main Sep 18, 2024

bedevere-app Bot removed the awaiting merge label Sep 18, 2024

zware mentioned this pull request Sep 18, 2024

Fix make htmllive target #124219

Merged

tomasr8 deleted the ast-repr branch September 19, 2024 07:49

alex mentioned this pull request Oct 5, 2024

Heap-use-after-free READ 8 · dictkeys_decref in fuzz_ast_literal_eval #125010

Closed

+                      }
+                      if (!value_repr) {
+                          Py_DECREF(name);

gh-116022: Improve `repr()` of AST nodes by tomasr8 · Pull Request #117046 · python/cpython · GitHub

Repository navigation

gh-116022: Improve repr() of AST nodes - #117046

gh-116022: Improve repr() of AST nodes#117046JelleZijlstra merged 39 commits intopython:mainpython/cpython:mainfrom tomasr8:ast-reprtomasr8/cpython:ast-reprCopy head branch name to clipboard

Conversation

tomasr8 commented Mar 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomasr8 commented May 2, 2024

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomasr8 commented Jul 14, 2024

Uh oh!

tomasr8 commented Aug 3, 2024

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Aug 14, 2024

Choose a reason for hiding this comment

Uh oh!

tomasr8 Aug 14, 2024

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Aug 14, 2024

Uh oh!

tomasr8 commented Aug 14, 2024

Uh oh!

JelleZijlstra commented Sep 17, 2024

Uh oh!

tomasr8 commented Sep 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gh-116022: Improve repr() of AST nodes#117046
JelleZijlstra merged 39 commits into
python:mainfrom
tomasr8:ast-repr

tomasr8 commented Mar 19, 2024 •
edited
Loading