gh-95555: Add the CATEGORY_UCD opcode and the simple enumerated properties - #153023
Open
serhiy-storchaka wants to merge 2 commits into
Open
gh-95555: Add the CATEGORY_UCD opcode and the simple enumerated properties#153023serhiy-storchaka wants to merge 2 commits into
serhiy-storchaka wants to merge 2 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
… properties
Introduce the unicodedata capsule used to match \p{...} properties that need
the Unicode Character Database, starting with the simplest tier: the
enumerated properties stored as a single byte in the per-character record --
Bidi_Class (bc), East_Asian_Width (ea), Grapheme_Cluster_Break (gcb) and
Indic_Conjunct_Break (incb).
unicodedata exports the _ucd_re_CAPI capsule (the \N{...} precedent) and
_ucd_re_info(), which lists the property selectors and value names. The new
CATEGORY_UCD opcode packs (negate, property, value) and is matched in C by
sre_category_ucd(). A negated single value is one charset item, so \P{bc=AL}
composes inside a set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copy link
Copy Markdown
Documentation build overview11 files changed · ± 11 modified ± Modified |
Sorry, something went wrong.
The CATEGORY_UCD support adds three process-global tables that are set once and effectively constant: the cached capsule pointer in sre_category_ucd(), the capsule struct built by unicodedata_create_re_capi(), and the enumerated-property table in unicodedata_ucd_re_info(). List them in the c-analyzer ignore file, as done for the name-lookup capsule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of four stacked PRs that complete \p{...} support with the properties backed by the Unicode Character Database, matched in C through a unicodedata capsule. This PR adds the machinery and the simplest tier: the enumerated properties stored as a single byte in the per-character record — Bidi_Class (bc), East_Asian_Width (ea), Grapheme_Cluster_Break (gcb) and Indic_Conjunct_Break (incb).
unicodedata exports the _ucd_re_CAPI capsule (the \N{...} precedent) and _ucd_re_info(), which lists the property selectors and value names. The parser resolves a property name and value to a selector and value index; the new CATEGORY_UCD opcode packs (negate, property, value) into one operand and is matched in C by sre_category_ucd(), which compares the index returned by the capsule. A negated single value is one charset item, so \P{bc=AL} composes inside a set.
The follow-up PRs add, in order: the numeric and binary properties (ccc, Bidi_Mirrored, Extended_Pictographic), the computed properties (Block, Decomposition_Type, Numeric_Type) and the remaining General_Category values and groups (Ll, Lo, M/P/S) with POSIX punct.