Repository navigation
gh-148821: Always reject known multi-byte encodings in pyexpat - #148911
serhiy-storchaka wants to merge 13 commits into
Conversation
Sorry, something went wrong.
The XML parser (pyexpat) now raises ValueError for known unsupported multi-byte encodings such us "ISO-2022-JP", "utf8" (without hyphen) or "raw-unicode-escape" instead of failing later, when encounter non-ASCII data.
|
It is a draft, because I have a technical issue here. I need to use a private C API, but simply adding PyAPI_FUNC in its declaration and including the corresponding private header does not work -- I get a linker error. But copying the declaration to a public header fixes build. I do not understand this. @vstinner, do you have any ideas? Can you please help? |
Sorry, something went wrong.
|
I see Python/codecs.c is missing #include "pycore_codecs.h", could that be it? |
Sorry, something went wrong.
|
Python/codecs.c must include pycore_codecs.h to export _PyCodec_LookupTextEncoding() correctly. You can apply the patch: diff --git a/Include/codecs.h b/Include/codecs.h
index d14f527dee7..512a3c723ec 100644
--- a/Include/codecs.h
+++ b/Include/codecs.h
@@ -170,12 +170,6 @@ PyAPI_FUNC(PyObject *) PyCodec_NameReplaceErrors(PyObject *exc);
PyAPI_DATA(const char *) Py_hexdigits;
#endif
-#ifndef Py_LIMITED_API
-PyAPI_FUNC(PyObject*) _PyCodec_LookupTextEncoding(
- const char *encoding,
- const char *alternate_command);
-#endif
-
#ifdef __cplusplus
}
#endif
diff --git a/Python/codecs.c b/Python/codecs.c
index 0bde56c0ac6..a522e6b8806 100644
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -10,6 +10,7 @@ Copyright (c) Corporation for National Research Initiatives.
#include "Python.h"
#include "pycore_call.h" // _PyObject_CallNoArgs()
+#include "pycore_codecs.h" // export _PyCodec_LookupTextEncoding()
#include "pycore_interp.h" // PyInterpreterState.codec_search_path
#include "pycore_pyerrors.h" // _PyErr_FormatNote()
#include "pycore_pystate.h" // _PyInterpreterState_GET() |
Sorry, something went wrong.
Sorry, something went wrong.
vstinner
left a comment
There was a problem hiding this comment.
I would prefer to rename _is_single_byte to _is_multibyte.
Sorry, something went wrong.
|
I do not know what wrong with the CIFuzz tests. I do not even know where are these tests located. |
Sorry, something went wrong.
|
python/library-fuzzers#55 will fix it, please ping there when this lands. |
Sorry, something went wrong.
|
#149860 replaces this PR. It uses _expat_decoding_table=False to mark incompatible with Expat encodings, so setting _is_multibyte=True is no longer needed for this issue. Although we can still set it if it is useful for other purposes. |
Sorry, something went wrong.
The XML parser (pyexpat) now raises ValueError for known unsupported multi-byte encodings such us "ISO-2022-JP", "utf8" (without hyphen) or "raw-unicode-escape" instead of failing later, when encounter non-ASCII data.