gh-130273: Fix traceback color output with unicode characters by grayjk · Pull Request #142529 · python/cpython · GitHub

Account for the display width of unicode characters so that colors and underlining in traceback output is correct

Closes #130273

Issue: Traceback colors are shifted when the line contains wide unicode characters #130273

All commit authors signed the Contributor License Agreement.

@serhiy-storchaka: Here is a PR about text width and Unicode characters :-)

Closes python#130273

updated to use @serhiy-storchaka's recently added unicodedata.iter_graphemes

@pablogsal @hauntsaninja as recent reviewers of traceback.py, would you mind taking look

There are conflicts again I'm afraid, and mypy isn't happy either.

@pablogsal You maybe make wrong something :)

Thanks for your PR.

I reverted the move of string-handling utils to traceback, they don't belong there
Now those utils are copied into traceback temporarily so that we can backport the fixes to older bugfix branches (3.13 and 3.14). In main I will do a follow-up change where those utils are moved to string/__init__.py where they belong.

This is because mypy type-checking introduction to the standard library relies on introduction of fully type-annotated libraries. I intend to keep it that way and therefore sprinkling type: ignore is not a solution.

It is now out of scope to format all of traceback.py for 3.15 while string.py is just three small classes and a function (two more functions if we count string.templatelib). When I make this change, I will also undo the unicodedata conditional import shenanigans and use lazy imports. That's also why this will be a main-only change.

There's also a related behavioral change in _display_width that I feel is part of this fix. It introduces handling of pre-existing terminal escape sequences in the exception strings (as well as any stranded ^Z characters but those are admittedly unlikely to be part of tracebacks).

Pull request overview

This PR fixes misaligned color highlighting/underlining in traceback output when the source line contains wide Unicode characters or grapheme clusters, by switching caret/column calculations to use display width rather than raw character offsets.

Changes:

Introduces grapheme-aware zipping between source characters and caret markers to keep colored segments aligned with terminal display width.
Reworks _display_width() to use _wlen()/_str_width() logic (including CTRL-Z and ANSI escape sequence handling) and adds targeted tests for wide/combining/ASCII edge cases.
Adds a NEWS fragment documenting the fix.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
Misc/NEWS.d/next/Library/2025-12-10-15-15-09.gh-issue-130273.iCfiY5.rst	Announces the traceback color alignment fix for Unicode characters.
Lib/traceback.py	Implements grapheme/display-width-aware caret-to-text alignment and updates width calculation helpers.
Lib/test/test_traceback.py	Adds regression tests for colorized tracebacks with wide/combining Unicode and ASCII display-width edge cases.
Lib/_pyrepl/utils.py	Minor whitespace-only change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Thanks @grayjk for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

Thanks @grayjk for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

Sorry, @grayjk and @ambv, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker dfeb160bc35f0ba16800d07b85cb11598d1cd307 3.14

Sorry, @grayjk and @ambv, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker dfeb160bc35f0ba16800d07b85cb11598d1cd307 3.13

There's some trivial conflicts that can be solved here, but the real problem is that unicodedata.iter_graphemes is new for 3.15 and a pure Python reimplementation is close to 200 lines.

WDYT, @serhiy-storchaka, is it worth creating a pure Python version of this segmentation purely for the backports?

To help you decide, I created #148218 so you can see what the implementation would look like.

…ythonGH-142529) Account for the display width of Unicode characters so that colors and underlining in traceback output is correct. Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-app Bot mentioned this pull request Dec 10, 2025

Traceback colors are shifted when the line contains wide unicode characters #130273

Closed

bedevere-app Bot added the awaiting review label Dec 10, 2025

StanFromIreland reviewed Dec 10, 2025

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2025-12-10-15-15-09.gh-issue-130273.iCfiY5.rst

grayjk added 3 commits January 28, 2026 17:45

Fix traceback color output with unicode characters

306a690

Closes python#130273

mv news blurb

8edad11

use unicodedata.iter_graphemes

7947033

grayjk force-pushed the issue-130273 branch from 59ca7f0 to 7947033 Compare January 28, 2026 23:05

grayjk requested a review from StanFromIreland January 28, 2026 23:19

This comment was marked as resolved.

Sign in to view

Merge branch 'main' into issue-130273

e8d23cd

This comment was marked as resolved.

Sign in to view

vstinner reviewed Feb 19, 2026

View reviewed changes

Comment thread Lib/traceback.py Outdated

mv wlen/str_width to traceback

467656e

grayjk requested review from ambv, lysnikolaou and pablogsal as code owners February 19, 2026 16:57

StanFromIreland requested a review from serhiy-storchaka March 31, 2026 17:11

StanFromIreland reviewed Mar 31, 2026

View reviewed changes

Comment thread Lib/traceback.py Outdated

vstinner reviewed Apr 1, 2026

View reviewed changes

Comment thread Lib/traceback.py Outdated

Comment thread Lib/traceback.py

Comment thread Misc/NEWS.d/next/Library/2025-12-10-15-15-09.gh-issue-130273.iCfiY5.rst Outdated

This comment was marked as outdated.

Sign in to view

ambv mentioned this pull request Apr 7, 2026

gh-130273: Fix incorrectly colored error locations when wide unicode characters exists #130277

Closed

ambv added 2 commits April 7, 2026 12:22

Merge branch 'main' into pr-142529

05ad91e

Restore _pyrepl.utils.str_width and .wlen; respond to code review

37a4fd9

Remove ASCII fast track that was too eager

1318217

ambv added 3 commits April 7, 2026 13:59

Add _zip_display_width fast track suggested by Victor

7b3fa87

Move str_width and wlen to the place in the file where they were before

4ddf024

derp

f094135

ambv requested a review from Copilot April 7, 2026 12:49

Copilot started reviewing on behalf of ambv April 7, 2026 12:50 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread Lib/traceback.py

Comment thread Lib/traceback.py

ambv merged commit dfeb160 into python:main Apr 7, 2026
64 checks passed

bedevere-app Bot removed the awaiting review label Apr 7, 2026

ambv added awaiting review needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Apr 7, 2026

miss-islington-app Bot assigned ambv Apr 7, 2026

ambv mentioned this pull request Apr 7, 2026

gh-130273: Add pure Python implementation of unicodedata.iter_graphemes() #148218

Closed

grayjk deleted the issue-130273 branch April 8, 2026 01:39

serhiy-storchaka removed needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jul 15, 2026

gh-130273: Fix traceback color output with unicode characters by grayjk · Pull Request #142529 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-130273: Fix traceback color output with unicode characters - #142529

gh-130273: Fix traceback color output with unicode characters#142529ambv merged 11 commits intopython:mainpython/cpython:mainfrom grayjk:issue-130273grayjk/cpython:issue-130273Copy head branch name to clipboard

Conversation

grayjk commented Dec 10, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot Bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vstinner commented Dec 11, 2025

Uh oh!

grayjk commented Jan 28, 2026

Uh oh!

grayjk commented Feb 18, 2026

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

StanFromIreland commented Mar 31, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Wulian233 commented Apr 5, 2026

Uh oh!

ambv commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miss-islington-app Bot commented Apr 7, 2026

Uh oh!

miss-islington-app Bot commented Apr 7, 2026

Uh oh!

miss-islington-app Bot commented Apr 7, 2026

Uh oh!

miss-islington-app Bot commented Apr 7, 2026

Uh oh!

ambv commented Apr 7, 2026

Uh oh!

ambv commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gh-130273: Fix traceback color output with unicode characters#142529
ambv merged 11 commits into
python:mainfrom
grayjk:issue-130273

grayjk commented Dec 10, 2025 •
edited by bedevere-app Bot
Loading

python-cla-bot Bot commented Dec 10, 2025 •
edited
Loading