Unicode: flag "u" and class \p{...}

JavaScript uses Unicode encoding for strings. Most characters are encoded with 2 bytes, but that allows to represent at most 65536 characters.

That range is not big enough to encode all possible characters, thats why some rare characters are encoded with 4 bytes, for instance like (mathematical X) or (a smile), some hieroglyphs and so on.

Here are the Unicode values of some characters:

So characters like a and occupy 2 bytes, while codes for , and are longer, they have 4 bytes.

Long time ago, when JavaScript language was created, Unicode encoding was simpler: there were no 4-byte characters. So, some language features still handle them incorrectly.

For instance, length thinks that here are two characters:

alert(''.length); // 2
alert(''.length); // 2

But we can see that theres only one, right? The point is that length treats 4 bytes as two 2-byte characters. Thats incorrect, because they must be considered only together (so-called surrogate pair, you can read about them in the article Strings).

By default, regular expressions also treat 4-byte long characters as a pair of 2-byte ones. And, as it happens with strings, that may lead to odd results. Well see that a bit later, in the article Sets and ranges [...].

Unlike strings, regular expressions have flag u that fixes such problems. With such flag, a regexp handles 4-byte characters correctly. And also Unicode property search becomes available, well get to it next.

Unicode properties \p{}

Every character in Unicode has a lot of properties. They describe what category the character belongs to, contain miscellaneous information about it.

For instance, if a character has Letter property, it means that the character belongs to an alphabet (of any language). And Number property means that its a digit: maybe Arabic or Chinese, and so on.

We can search for characters with a property, written as \p{}. To use \p{}, a regular expression must have flag u.

For instance, \p{Letter} denotes a letter in any language. We can also use \p{L}, as L is an alias of Letter. There are shorter aliases for almost every property.

In the example below three kinds of letters will be found: English, Georgian and Korean.

let str = "A  ";

alert( str.match(/\p{L}/gu) ); // A,,
alert( str.match(/\p{L}/g) ); // null (no matches, \p doesn't work without the flag "u")

Heres the main character categories and their subcategories:

• Letter L:
  • lowercase Ll
  • modifier Lm,
  • titlecase Lt,
  • uppercase Lu,
  • other Lo.
• Number N:
  • decimal digit Nd,
  • letter number Nl,
  • other No.
• Punctuation P:
  • connector Pc,
  • dash Pd,
  • initial quote Pi,
  • final quote Pf,
  • open Ps,
  • close Pe,
  • other Po.
• Mark M (accents etc):
  • spacing combining Mc,
  • enclosing Me,
  • non-spacing Mn.
• Symbol S:
  • currency Sc,
  • modifier Sk,
  • math Sm,
  • other So.
• Separator Z:
  • line Zl,
  • paragraph Zp,
  • space Zs.
• Other C:
  • control Cc,
  • format Cf,
  • not assigned Cn,
  • private use Co,
  • surrogate Cs.

So, e.g. if we need letters in lower case, we can write \p{Ll}, punctuation signs: \p{P} and so on.

There are also other derived categories, like:

• Alphabetic (Alpha), includes Letters L, plus letter numbers Nl (e.g. a character for the roman number 12), plus some other symbols Other_Alphabetic (OAlpha).
• Hex_Digit includes hexadecimal digits: 0-9, a-f.
• And so on.

Unicode supports many different properties, their full list would require a lot of space, so here are the references:

• List all properties by a character: https://unicode.org/cldr/utility/character.jsp.
• List all characters by a property: https://unicode.org/cldr/utility/list-unicodeset.jsp.
• Short aliases for properties: https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt.
• A full base of Unicode characters in text format, with all properties, is here: https://www.unicode.org/Public/UCD/latest/ucd/.

Example: hexadecimal numbers

For instance, lets look for hexadecimal numbers, written as xFF, where F is a hex digit (09 or AF).

A hex digit can be denoted as \p{Hex_Digit}:

let regexp = /x\p{Hex_Digit}\p{Hex_Digit}/u;

alert("number: xAF".match(regexp)); // xAF

Example: Chinese hieroglyphs

Lets look for Chinese hieroglyphs.

Theres a Unicode property Script (a writing system), that may have a value: Cyrillic, Greek, Arabic, Han (Chinese) and so on, heres the full list.

To look for characters in a given writing system we should use Script=<value>, e.g. for Cyrillic letters: \p{sc=Cyrillic}, for Chinese hieroglyphs: \p{sc=Han}, and so on:

let regexp = /\p{sc=Han}/gu; // returns Chinese hieroglyphs

let str = `Hello   123_456`;

alert( str.match(regexp) ); // ,

Example: currency

Characters that denote a currency, such as $, , , have Unicode property \p{Currency_Symbol}, the short alias: \p{Sc}.

Lets use it to look for prices in the format currency, followed by a digit:

let regexp = /\p{Sc}\d/gu;

let str = `Prices: $2, 1, 9`;

alert( str.match(regexp) ); // $2,1,9

Later, in the article Quantifiers +, *, ? and {n} well see how to look for numbers that contain many digits.

Summary

Flag u enables the support of Unicode in regular expressions.

That means two things:

1. Characters of 4 bytes are handled correctly: as a single character, not two 2-byte characters.
2. Unicode properties can be used in the search: \p{}.

With Unicode properties we can look for words in given languages, special characters (quotes, currencies) and so on.