Repository navigation
Add multibyte encoding support to pyexpat #62259
Description
| BPO | 18059 |
|---|---|
| Nosy | @malemburg, @doerwalter, @amauryfa, @ncoghlan, @scoder, @vstinner, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-05-25.18:41:28.221>
labels = ['extension-modules', 'expert-XML', 'type-feature', '3.7']
title = 'Add multibyte encoding support to pyexpat'
updated_at = <Date 2017-03-27.10:33:59.091>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2017-03-27.10:33:59.091>
actor = 'doerwalter'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'XML']
creation = <Date 2013-05-25.18:41:28.221>
creator = 'serhiy.storchaka'
dependencies = []
files = ['30381', '31759']
hgrepos = []
issue_num = 18059
keywords = ['patch']
message_count = 16.0
messages = ['189989', '189995', '190011', '190022', '190024', '190064', '190070', '197686', '197708', '197725', '203839', '203898', '203906', '203919', '290483', '290576']
nosy_count = 7.0
nosy_names = ['lemburg', 'doerwalter', 'amaury.forgeotdarc', 'ncoghlan', 'scoder', 'vstinner', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18059'
versions = ['Python 3.7']
Activity
serhiy-storchaka commented on May 25, 2013
It is possible to add the support of most multibyte encodings to pyexpat.
There are several ways to do this:
Generate maps with a special script and add generated file to repository. After adding or updating a multibyte encoding this file should be regenerated.
Generate maps on fly. It requires more time for first use of the encoding, but allows support of arbitrary encoding which compatible with expat.
amauryfa commented on May 25, 2013
I guess GB18030 can't be supported at all?
serhiy-storchaka commented on May 25, 2013
Here is a patch which implements first way.
Yes, looks as followed encodings could not be supported at all: euc-kr, gb18030, iso2022-kr, utf-7, cp037, cp424, cp500, cp864, cp875, cp1026, cp1140, utf_32, utf_32_be, utf_32_le.
amauryfa commented on May 25, 2013
Then you should also remove the "Make it as simple as possible" comment :-/
serhiy-storchaka commented on May 25, 2013
It is still simple enough.
serhiy-storchaka commented on May 26, 2013
Patch updated. Fixed an error in the encodings generator and added additional compatibility check for 8-bit encodings in PyUnknownEncodingHandler().
Feel free to bikesheed the encodings generator.
serhiy-storchaka commented on May 26, 2013
Patch updated. Yet some tests added and yet some bugs fixed.
scoder commented on Sep 14, 2013
I don't think I have my head deep enough in the encodings implementation to say that this is the correct/best way to do it, but the patch looks mostly reasonable to me and would be a helpful addition.
I have two comments on the pyexpat_encoding_convert() function:
I can't see a safe-guard against reading beyond the data buffer. What if s already points to the last byte and we are trying to read two or three bytes to decode them? I wouldn't be surprised to see that this kind of input can be crafted.
Creating a throw-away Unicode object through a named decoder looks like a huge overhead for decoding two bytes. It might be considered an optimisation to change that, but if you are really trying to parse a longer XML document with lots of Japanese text in it (i.e. if you actually *need* this feature), it will most likely end up being way too slow to make any real use of it.
I think that both points should be addressed before this gets added.
serhiy-storchaka commented on Sep 14, 2013
Expat itself responsible for this guard. It has all necessary information and provides an input of required size for custom converter.
Yes, this is a problem. I'm working on another approach, when full encoding table built at first request for the encoding (and than cache it). It makes decoding individual characters fast, but requires about 0.5 sec for initialization. Is such approach more suitable?
serhiy-storchaka commented on Sep 14, 2013
Here is a totally rewritten patch, which builds decoding table at first request for encoding and save it in the cache. Decoding should be very fast.
Do you have large testing XML files with multibyte encodings? Could you please measure the time of parsing this files and for comparision the time of parsing this files encoded with utf-8 and utf-16?
serhiy-storchaka commented on Nov 22, 2013
If anybody is interested in support of multibyte encodings in XML parser, it is time to make a review.
vstinner commented on Nov 22, 2013
I'm not sure that multibyte encodings other than UTF-8 are used in the world. I'm not convinced that we should support them. If the changes are small, it's maybe not a bad thing. Do you know which applications use such codecs?
pyexpat_encoding_create() looks like an heuristic. How many multibyte codecs can be used with your patch? A whitelist of multibyte codecs may be less reliable. What do you think?
serhiy-storchaka commented on Nov 22, 2013
I don't use any of them but I heard some of them are still widely used.
This issue was provoked by bpo-13612. See also related bpo-15877.
All codecs which can be supported by expat.
"""
Every ASCII character that can appear in a well-formed XML document,
other than the characters
$@^`{}~
must be represented by a single byte, and that byte must be the
same byte that represents that character in ASCII.
No character may require more than 4 bytes to encode.
All characters encoded must have Unicode scalar values <=
0xFFFF, (i.e., characters that would be encoded by surrogates in
UTF-16 are not allowed). Note that this restriction doesn't
apply to the built-in support for UTF-8 and UTF-16.
No Unicode character may be encoded by more than one distinct
sequence of bytes.
"""
14 Python encodings satisfy these criteria: big5, big5hkscs, cp932, cp949, cp950, euc-jp, euc-jis-2004, euc-jisx0213, gb2312, gbk, johab, shift-jis, shift-jis-2004, shift-jisx0213.
pyexpat_multibyte_encodings_4.patch implements this way. It hardcodes a list of supported encodings with minimal required tables.
pyexpat_multibyte_encodings_5.patch supports any encoding which satisfy expat criteria and builds all needed data at first access (tens kilobytes). After heavy start it works much faster than previous patch.
malemburg commented on Nov 22, 2013
On 22.11.2013 23:03, STINNER Victor wrote:
I'm not sure what you mean with multibyte encodings. There's UTF-16 which
is a popular 2-byte encoding and then there are a whole lot of variable
length encodings such as UTF-8 and many of the Asian codecs in the stdlib.
While you see those used a lot for text, I'm not sure whether the
same is true for XML documents, where UTF-8 is the standard,
but other encodings can be specified if needed.
Serhiy: Apart from this being a nice-to-have feature, where do you see
the practical use ?
serhiy-storchaka commented on Mar 25, 2017
Marc-Andre, there are at least two issues about supporting East Asian encodings (bpo-13612 and bpo-15877). I think this means that that encodings are used in XML in wild. Current support of encodings (8-bit + UTF-8 + UTF-16) is enough for my needs, but I never have deal with East Asian languages.
Currently the CodecInfo object has an optional flag _is_text_encoding. I think we can add more private attributes (flags and precomputed tables) for using with the expat parser. If they are not set (third-party encodings) the current autodetection code can be used as a fallback.
doerwalter commented on Mar 27, 2017
This looks to me like a limited reimplementation of the codec machinery. Why not use incremental codecs as a preprocessor? Would this be to slow?
serhiy-storchaka commented on May 15, 2026
#149860 is an extension of the idea of my old patches. It also replaces #148911, although these PRs can complement one other. The differences between them:
The limitation of Expat is that it only supports BMP characters (U+0000 - U+FFFF), so some encodings are only supported partially. This includes non-standard aliases of UTF-8, like "UTF8" or "utf-8-sig".
gh-62259: Add support of multi-byte encodings in the XML parser (GH-1…
serhiy-storchaka commented on May 27, 2026
#150503 adds to Tools/unicode/ a cleaned-up version of the script I used to generate these tables, in case we add new encodings or change the form of these attributes.
Some encodings are incompatible with Expat:
I have not found any information about compatibility of EBCDIC or other ASCII-incompatible encodings (except UTF-16/UTF-32) with XML, but they are not supported in Expat. I do not know if they are supported in other XML parsers.
gh-62259: Add Tools/unicode/gen_expat_table.py (GH-150503)
pythongh-62259: Add Tools/unicode/gen_expat_table.py (pythonGH-150503)