gh-74902: Add Unicode Grapheme Cluster Break algorithm by serhiy-storchaka · Pull Request #143076 · python/cpython · GitHub

Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex #29.

Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm.

Issue: Add unicode grapheme cluster break algorithm #74902

Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex #29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

The order of functions in this file doesn’t seem to be alphabetical or topical.
I think another ticket should be created to add a quick links table at the top.

Or we can split it on sections by type and order alphabetically inside a section.

This sentence is not fully right, but I can’t find the right suggestion with both «most of» and «same as».

These functions help compute width?

At least two implementations (in Perl's Unicode::GCString and builtin in C++) use graphemes. Naive implementation in C's wcwidth() does not work well with complex characters and Emoji.

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Sorry if my question was not clear.
Are these functions building blocks that will be used to compute character widths, as needed by the other tickets about pyrepl, str justify, etc?
(maybe iter_graphemes?)

Yes, I think that _PyGraphemeBreak, _Py_InitGraphemeBreak() and _Py_NextGraphemeBreak() will be used in the implementation of unicodedata.width().

Did you mean to use the passed ucd argument?

Suggested change

      
                            result = list(unicodedata.iter_graphemes(input))
          
                            result = list(ucd.iter_graphemes(input))

Suggested change

      
                                result = list(unicodedata.iter_graphemes(input, breaks[i]))
          
                                result = list(ucd.iter_graphemes(input, breaks[i]))

Continues above.

No, it is module-only function.

The above functions already have docstrings?

Should it not be urlfetch resource?

Maybe. The other test (for normalization) uses the network resource).

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

…me_break_tests().

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Stable LTO 3.x (tier-3) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1654/builds/1940) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1654/builds/1940

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-s390x.lto/build/Lib/tempfile.py", line 484, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: Implicitly cleaning up <HTTPError 403: 'Forbidden'>

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2750) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2750

Failed tests:

test_unicodedata

Test leaking resources:

test_unicodedata: memory blocks
test_unicodedata: references

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 24, done.        
remote: Counting objects:   6% (1/16)        
remote: Counting objects:  12% (2/16)        
remote: Counting objects:  18% (3/16)        
remote: Counting objects:  25% (4/16)        
remote: Counting objects:  31% (5/16)        
remote: Counting objects:  37% (6/16)        
remote: Counting objects:  43% (7/16)        
remote: Counting objects:  50% (8/16)        
remote: Counting objects:  56% (9/16)        
remote: Counting objects:  62% (10/16)        
remote: Counting objects:  68% (11/16)        
remote: Counting objects:  75% (12/16)        
remote: Counting objects:  81% (13/16)        
remote: Counting objects:  87% (14/16)        
remote: Counting objects:  93% (15/16)        
remote: Counting objects: 100% (16/16)        
remote: Counting objects: 100% (16/16), done.        
remote: Compressing objects:   9% (1/11)        
remote: Compressing objects:  18% (2/11)        
remote: Compressing objects:  27% (3/11)        
remote: Compressing objects:  36% (4/11)        
remote: Compressing objects:  45% (5/11)        
remote: Compressing objects:  54% (6/11)        
remote: Compressing objects:  63% (7/11)        
remote: Compressing objects:  72% (8/11)        
remote: Compressing objects:  81% (9/11)        
remote: Compressing objects:  90% (10/11)        
remote: Compressing objects: 100% (11/11)        
remote: Compressing objects: 100% (11/11), done.        
remote: Total 24 (delta 5), reused 5 (delta 5), pack-reused 8 (from 2)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'bab1d7a561ab015dd6bb97e255fd12a8ce367edf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at bab1d7a561a gh-74902: Add Unicode Grapheme Cluster Break algorithm (GH-143076)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

make: *** [Makefile:2503: buildbottest] Error 2

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Refleaks 3.x (tier-3) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1613/builds/2681) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1613/builds/2681

Failed tests:

test_unicodedata

Test leaking resources:

test_unicodedata: memory blocks
test_unicodedata: references

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 24, done.        
remote: Counting objects:   6% (1/16)        
remote: Counting objects:  12% (2/16)        
remote: Counting objects:  18% (3/16)        
remote: Counting objects:  25% (4/16)        
remote: Counting objects:  31% (5/16)        
remote: Counting objects:  37% (6/16)        
remote: Counting objects:  43% (7/16)        
remote: Counting objects:  50% (8/16)        
remote: Counting objects:  56% (9/16)        
remote: Counting objects:  62% (10/16)        
remote: Counting objects:  68% (11/16)        
remote: Counting objects:  75% (12/16)        
remote: Counting objects:  81% (13/16)        
remote: Counting objects:  87% (14/16)        
remote: Counting objects:  93% (15/16)        
remote: Counting objects: 100% (16/16)        
remote: Counting objects: 100% (16/16), done.        
remote: Compressing objects:   9% (1/11)        
remote: Compressing objects:  18% (2/11)        
remote: Compressing objects:  27% (3/11)        
remote: Compressing objects:  36% (4/11)        
remote: Compressing objects:  45% (5/11)        
remote: Compressing objects:  54% (6/11)        
remote: Compressing objects:  63% (7/11)        
remote: Compressing objects:  72% (8/11)        
remote: Compressing objects:  81% (9/11)        
remote: Compressing objects:  90% (10/11)        
remote: Compressing objects: 100% (11/11)        
remote: Compressing objects: 100% (11/11), done.        
remote: Total 24 (delta 5), reused 5 (delta 5), pack-reused 8 (from 2)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'bab1d7a561ab015dd6bb97e255fd12a8ce367edf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at bab1d7a561a gh-74902: Add Unicode Grapheme Cluster Break algorithm (GH-143076)
Switched to and reset branch 'main'

…H-143076) Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex python#29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

…ythonGH-143076) Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex #29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. (cherry picked from commit bab1d7a) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

GH-148247 is a backport of this pull request to the 3.14 branch.

serhiy-storchaka requested a review from AA-Turner as a code owner December 22, 2025 16:34

bedevere-app Bot mentioned this pull request Dec 22, 2025

Add unicode grapheme cluster break algorithm #74902

Closed

bedevere-app Bot added the awaiting core review label Dec 22, 2025

Try to fix rst.

b0585f9

AA-Turner mentioned this pull request Dec 22, 2025

gh-74902: add unicode grapheme cluster break algorithm #2673

Closed

AA-Turner reviewed Dec 22, 2025

View reviewed changes

Comment thread Doc/whatsnew/3.15.rst Outdated

grayjk mentioned this pull request Dec 22, 2025

What would wcwidth look like if it were built-in to Python? jquast/wcwidth#94

Closed

StanFromIreland reviewed Dec 22, 2025

View reviewed changes

Comment thread Doc/library/unicodedata.rst Outdated

merwok reviewed Dec 23, 2025

View reviewed changes

serhiy-storchaka and others added 2 commits December 23, 2025 11:37

iConvert the types to heap types.

37fa38f

Update Doc/library/unicodedata.rst

c46e9bd

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

StanFromIreland reviewed Dec 26, 2025

View reviewed changes

serhiy-storchaka and others added 6 commits December 26, 2025 22:06

Apply suggestions from code review

22cacf6

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Add tests for GB12 and GB13. Remove redundant parameter in run_graphe…

a95c3cb

…me_break_tests().

Rename Grapheme to Segment. Add __repr__(), remove __iter__().

ad50831

Merge branch 'main' into grapheme_cluster_break2

365d0c1

Merge branch 'main' into grapheme_cluster_break2

f96c8d0

Polishing.

3b3a50a

serhiy-storchaka enabled auto-merge (squash) January 14, 2026 14:16

serhiy-storchaka merged commit bab1d7a into python:main Jan 14, 2026
47 checks passed

bedevere-app Bot removed the awaiting core review label Jan 14, 2026

serhiy-storchaka deleted the grapheme_cluster_break2 branch July 1, 2026 16:47

+                          self.assertEqual(chunks.pop(), '', line)
+                          input = ''.join(chunks)
+                          with self.subTest(line):
+                              result = list(unicodedata.iter_graphemes(input))

+                              self.assertEqual([x.start for x in result], breaks[:-1], comment)
+                              self.assertEqual([x.end for x in result], breaks[1:], comment)
+                              for i in range(1, len(breaks) - 1):
+                                  result = list(unicodedata.iter_graphemes(input, breaks[i]))

               .. data:: ucd_3_2_0
-                 This is an object that has the same methods as the entire module, but uses the
+                 This is an object that has most of the methods of the entire module, but uses the

	result = list(unicodedata.iter_graphemes(input))
	result = list(ucd.iter_graphemes(input))

	result = list(unicodedata.iter_graphemes(input, breaks[i]))
	result = list(ucd.iter_graphemes(input, breaks[i]))

+                      hdr = testfile.readline()
+                      return unicodedata.unidata_version in hdr
+                  @requires_resource('network')

		``False`` otherwise.

		.. versionadded:: next

		}


		/* XXX Add doc strings. */

gh-74902: Add Unicode Grapheme Cluster Break algorithm by serhiy-storchaka · Pull Request #143076 · python/cpython · GitHub

Repository navigation

gh-74902: Add Unicode Grapheme Cluster Break algorithm - #143076

gh-74902: Add Unicode Grapheme Cluster Break algorithm#143076serhiy-storchaka merged 10 commits intopython:mainpython/cpython:mainfrom serhiy-storchaka:grapheme_cluster_break2serhiy-storchaka/cpython:grapheme_cluster_break2Copy head branch name to clipboard

Conversation

serhiy-storchaka commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merwok commented Dec 23, 2025

Uh oh!

serhiy-storchaka commented Dec 23, 2025

Uh oh!

merwok commented Dec 23, 2025

Uh oh!

serhiy-storchaka commented Dec 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-app Bot commented Apr 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-74902: Add Unicode Grapheme Cluster Break algorithm#143076
serhiy-storchaka merged 10 commits into
python:mainfrom
serhiy-storchaka:grapheme_cluster_break2

serhiy-storchaka commented Dec 22, 2025 •
edited
Loading