CC: @pablogsal @ammaraskar

Someone should try this on Windows with "Courier New" as Terry mentioned in the issue. It will be good to check with a bunch of fonts to see what we are up against.

tokenizer:  python
  3xREGULAR
    TokenInfo(type=3 (STRING), string="'XXX'", start=(1, 0), end=(1, 5), line="'XXX'")
  3xTINY H
    TokenInfo(type=3 (STRING), string="'ʰʰʰ'", start=(1, 0), end=(1, 5), line="'ʰʰʰ'")
  3xEAST ASIAN
    TokenInfo(type=3 (STRING), string="'该该该'", start=(1, 0), end=(1, 5), line="'该该该'")
  3xEMOJIs
    TokenInfo(type=3 (STRING), string="'🐍🐍🐍'", start=(1, 0), end=(1, 5), line="'🐍🐍🐍'")
tokenizer:  c
  3xREGULAR
    TokenInfo(type=3 (STRING), string="'XXX'", start=(1, 0), end=(1, 5), line="'XXX'\n")
  3xTINY H
    TokenInfo(type=3 (STRING), string="'ʰʰʰ'", start=(1, 0), end=(1, 8), line="'ʰʰʰ'\n")
  3xEAST ASIAN
    TokenInfo(type=3 (STRING), string="'该该该'", start=(1, 0), end=(1, 11), line="'该该该'\n")
  3xEMOJIs
    TokenInfo(type=3 (STRING), string="'🐍🐍🐍'", start=(1, 0), end=(1, 14), line="'🐍🐍🐍'\n")

Which then directly mirrored back to AST offets, which makes stuff a bit trickier at the traceback level. We could just patch this on the traceback level though I am not really sure that 'hack' is the proper way to go. We might need to end up a similiar solution to our _PyPegen_byte_offset_to_character_offset in the AST logic too, but I am not too certain about how.

Seems to work now with my last commit

I simply applied the same functionality at the end of the specialization function, and casted the newly retrieved AST offsets back to the regular ones. There are a lot of code that needs to be cleaned away in the final version, though for the prototype at least it works (for now).

This PR is stale because it has been open for 30 days with no activity.

Wow, this is an old PR but @pablogsal @ammaraskar are we interested in reviving it? I think I can rebase and get it ready for 3.12.

I am. I was precisely thinking on reviving it last week so you managed to read my mind! 😁

Hahahaha, perfection! I'll try to get it up to date over this week, and will ping you again for the review 💯

Hahahaha, perfection

Sounds good, happy to re-review when you update :)

Sorry for the slow follow-ups around PEP657 stuff, I've been a little busy and inactive recently :(

I've pushed the initial revision where we handle everything using the unicodedata.east_asian_width (double width on W and F labels, single on rest) as per the discussion here. Before merging this we also have to decide:

• Do we want to give users a way to opt-in/opt-out from this (considering the nature, I'd say opt-out, since I don't think anybody is going to look why behaves like this and turn this feature on)?

• Whether to do some sort of terminal support detection (non-deterministic tracebacks depending on non PYTHON* env variable configuration doesn't sound really nice but no strong opinions if we want to make this opt-In by default and add some sort of scanning)

• Is this applicable when showing the traceback in web? A lot of web frameworks use traceback.* APIs to format tracebacks and then embed them inside an HTML page on error in debug mode. Should we offer a flag in format_* APIs to allow them switch without changing it in the global Python exectuon level.

• Is this a feature or a fix? I am leaning towards a feature since this essentially changes how tracebacks are printed for certain cases and people might depend on the behavior (there are a couple of codebases where tracebacks are also part of the tests themselves in a stringized form)

I guess also looking at what other compilers are doing by default might also help us gain some insight (I recall @pablogsal mentioning rustc; maybe it might worth a shot to check out what they do to decide when to show carets).

CC: @cfbolz (would also love to hear your feedback on the unicode related parts)

I'll take a look at the code!

"Amusingly" the width doesn't line up in my browser's font:

(Looks fantastic in my editor and my terminal though)

Personal opinions on some of your questions:

• imo it's a bug. for people with wide characters the position info was just confusing so far, fixing that is a bug fix.
• maybe we want a way to opt out (I have no clear opinion on that) but the new behaviour should be the default

The code looks reasonable to me.

I've been thinking about it a bit more, and it would be certainly more annoying to implement, but I am wondering whether it wouldn't be an option to use unicode chars 0x3000 (IDEOGRAPHIC SPACE) and 0xFF3E (FULLWIDTH CIRCUMFLEX ACCENT) to do the spaces/underlines under wide chars. Because even if in a font the width of two ascii spaces is not the same as a fullwidth char, the font should at least be consistent with itself and have the fullwidth space be the same width. Example:

Here are the chars:

说明说明📗a 
　　　　　a fullwidth space
＾＾＾＾＾a fullwidth circumflex
          a ascii whitespace
^^^^^^^^^^a ascii circumflex

Screenshot in my Firefox:

(in my terminal all the 'a' line up, so it doesn't matter there).

(Unfortunately it already breaks down in Chrome on my laptop, where the book emoji is even wider)

@isidentical let's push this forward. Could you rebase the PR?

yay, i'm excited for this to land :-)

GH-111345 is a backport of this pull request to the 3.11 branch.

GH-111346 is a backport of this pull request to the 3.12 branch.

GH-111373 is a backport of this pull request to the 3.11 branch.