Issue #9738: Document encodings of AST, compiler, parser and PyRun fu… · python-emscripten/cpython@00676d1 · GitHub

1. python-emscripten
2. cpython

Repository navigation

• Code
• Pull requests
• Actions
• Projects
• Security and quality
• Insights

Commit 00676d1

Browse filesBrowse the repository at this point in the historyBrowse files

Victor Stinner

committed

Issue python#9738: Document encodings of AST, compiler, parser and PyRun functions

1 parent dc2081f commit 00676d1
Copy full SHA for 00676d1

6 files changed

+111-49Lines changed: 111 additions & 49 deletions

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

• Doc/c-api
  • veryhigh.rst
• Include
  • ast.h
  • compile.h
  • parsetok.h
  • pythonrun.h
  • symtable.h

Expand file treeCollapse file tree

Open diff view settings

Collapse file

‎Doc/c-api/veryhigh.rst‎

Copy file name to clipboardExpand all lines: Doc/c-api/veryhigh.rst

+20-11Lines changed: 20 additions & 11 deletions

• Display the source diff
• Display the rich diff

Original file line number	Diff line number	Diff line change
@@ -66,8 +66,9 @@ the same library that the Python runtime is using.
66	66	If *fp* refers to a file associated with an interactive device (console or
67	67	terminal input or Unix pseudo-terminal), return the value of
68	68	:c:func:`PyRun_InteractiveLoop`, otherwise return the result of
69		- :c:func:`PyRun_SimpleFile`. If *filename* is *NULL*, this function uses
70		- ``"???"`` as the filename.
	69	+ :c:func:`PyRun_SimpleFile`. *filename* is decoded from the filesystem
	70	+ encoding (:func:`sys.getfilesystemencoding`). If *filename* is *NULL*, this
	71	+ function uses ``"???"`` as the filename.
71	72
72	73
73	74	.. c:function:: int PyRun_SimpleString(const char *command)
@@ -110,9 +111,10 @@ the same library that the Python runtime is using.
110	111	.. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
111	112
112	113	Similar to :c:func:`PyRun_SimpleStringFlags`, but the Python source code is read
113		- from *fp* instead of an in-memory string. *filename* should be the name of the
114		- file. If *closeit* is true, the file is closed before PyRun_SimpleFileExFlags
115		- returns.
	114	+ from *fp* instead of an in-memory string. *filename* should be the name of
	115	+ the file, it is decoded from the filesystem encoding
	116	+ (:func:`sys.getfilesystemencoding`). If *closeit* is true, the file is
	117	+ closed before PyRun_SimpleFileExFlags returns.
116	118
117	119
118	120	.. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename)
@@ -125,7 +127,10 @@ the same library that the Python runtime is using.
125	127
126	128	Read and execute a single statement from a file associated with an
127	129	interactive device according to the *flags* argument. The user will be
128		- prompted using ``sys.ps1`` and ``sys.ps2``. Returns ``0`` when the input was
	130	+ prompted using ``sys.ps1`` and ``sys.ps2``. *filename* is decoded from the
	131	+ filesystem encoding (:func:`sys.getfilesystemencoding`).
	132	+
	133	+ Returns ``0`` when the input was
129	134	executed successfully, ``-1`` if there was an exception, or an error code
130	135	from the :file:`errcode.h` include file distributed as part of Python if
131	136	there was a parse error. (Note that :file:`errcode.h` is not included by
@@ -142,7 +147,8 @@ the same library that the Python runtime is using.
142	147
143	148	Read and execute statements from a file associated with an interactive device
144	149	until EOF is reached. The user will be prompted using ``sys.ps1`` and
145		- ``sys.ps2``. Returns ``0`` at EOF.
	150	+ ``sys.ps2``. *filename* is decoded from the filesystem encoding
	151	+ (:func:`sys.getfilesystemencoding`). Returns ``0`` at EOF.
146	152
147	153
148	154	.. c:function:: struct _node* PyParser_SimpleParseString(const char *str, int start)
@@ -164,7 +170,8 @@ the same library that the Python runtime is using.
164	170	Parse Python source code from *str* using the start token *start* according to
165	171	the *flags* argument. The result can be used to create a code object which can
166	172	be evaluated efficiently. This is useful if a code fragment must be evaluated
167		- many times.
	173	+ many times. *filename* is decoded from the filesystem encoding
	174	+ (:func:`sys.getfilesystemencoding`).
168	175
169	176
170	177	.. c:function:: struct _node* PyParser_SimpleParseFile(FILE *fp, const char *filename, int start)
@@ -217,7 +224,8 @@ the same library that the Python runtime is using.
217	224	.. c:function:: PyObject* PyRun_FileExFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit, PyCompilerFlags *flags)
218	225
219	226	Similar to :c:func:`PyRun_StringFlags`, but the Python source code is read from
220		- *fp* instead of an in-memory string. *filename* should be the name of the file.
	227	+ *fp* instead of an in-memory string. *filename* should be the name of the file,
	228	+ it is decoded from the filesystem encoding (:func:`sys.getfilesystemencoding`).
221	229	If *closeit* is true, the file is closed before :c:func:`PyRun_FileExFlags`
222	230	returns.
223	231
@@ -241,8 +249,9 @@ the same library that the Python runtime is using.
241	249	code which can be compiled and should be :const:`Py_eval_input`,
242	250	:const:`Py_file_input`, or :const:`Py_single_input`. The filename specified by
243	251	*filename* is used to construct the code object and may appear in tracebacks or
244		- :exc:`SyntaxError` exception messages. This returns *NULL* if the code cannot
245		- be parsed or compiled.
	252	+ :exc:`SyntaxError` exception messages, it is decoded from the filesystem
	253	+ encoding (:func:`sys.getfilesystemencoding`). This returns *NULL* if the
	254	+ code cannot be parsed or compiled.
246	255
247	256	The integer *optimize* specifies the optimization level of the compiler; a
248	257	value of ``-1`` selects the optimization level of the interpreter as given by

Collapse file

‎Include/ast.h‎

Copy file name to clipboardExpand all lines: Include/ast.h

+5-2Lines changed: 5 additions & 2 deletions

Original file line number	Diff line number	Diff line change
@@ -4,8 +4,11 @@
4	4	extern "C" {
5	5	#endif
6	6
7		- PyAPI_FUNC(mod_ty) PyAST_FromNode(const node *, PyCompilerFlags *flags,
8		- const char *, PyArena *);
	7	+ PyAPI_FUNC(mod_ty) PyAST_FromNode(
	8	+ const node *n,
	9	+ PyCompilerFlags *flags,
	10	+ const char *filename, /* decoded from the filesystem encoding */
	11	+ PyArena *arena);
9	12
10	13	#ifdef __cplusplus
11	14	}

Collapse file

‎Include/compile.h‎

Copy file name to clipboardExpand all lines: Include/compile.h

+6-2Lines changed: 6 additions & 2 deletions

Original file line number	Diff line number	Diff line change
@@ -30,8 +30,12 @@ typedef struct {
30	30
31	31	struct _mod; /* Declare the existence of this type */
32	32	#define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)
33		- PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(struct _mod *, const char *,
34		- PyCompilerFlags *, int, PyArena *);
	33	+ PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(
	34	+ mod_ty mod,
	35	+ const char *filename, /* decoded from the filesystem encoding */
	36	+ PyCompilerFlags *flags,
	37	+ int optimize,
	38	+ PyArena *arena);
35	39	PyAPI_FUNC(PyFutureFeatures *) PyFuture_FromAST(struct _mod *, const char *);
36	40
37	41

Collapse file

‎Include/parsetok.h‎

Copy file name to clipboardExpand all lines: Include/parsetok.h

+20-11Lines changed: 20 additions & 11 deletions

Original file line number	Diff line number	Diff line change
@@ -9,10 +9,10 @@ extern "C" {
9	9
10	10	typedef struct {
11	11	int error;
12		- const char *filename;
	12	+ const char *filename; /* decoded from the filesystem encoding */
13	13	int lineno;
14	14	int offset;
15		- char *text;
	15	+ char *text; /* UTF-8-encoded string */
16	16	int token;
17	17	int expected;
18	18	} perrdetail;
@@ -39,23 +39,32 @@ PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int,
39	39
40	40	PyAPI_FUNC(node *) PyParser_ParseStringFlags(const char *, grammar *, int,
41	41	perrdetail *, int);
42		- PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *,
	42	+ PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *,
43	43	const char*, grammar *,
44	44	int, char *, char *,
45	45	perrdetail *, int);
46		- PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(FILE *, const char *,
47		- const char*, grammar *,
48		- int, char *, char *,
49		- perrdetail *, int *);
	46	+ PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(
	47	+ FILE *fp,
	48	+ const char *filename, /* decoded from the filesystem encoding */
	49	+ const char *enc,
	50	+ grammar *g,
	51	+ int start,
	52	+ char *ps1,
	53	+ char *ps2,
	54	+ perrdetail *err_ret,
	55	+ int *flags);
50	56
51	57	PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilename(const char *,
52	58	const char *,
53	59	grammar *, int,
54	60	perrdetail *, int);
55		- PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilenameEx(const char *,
56		- const char *,
57		- grammar *, int,
58		- perrdetail *, int *);
	61	+ PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilenameEx(
	62	+ const char *s,
	63	+ const char *filename, /* decoded from the filesystem encoding */
	64	+ grammar *g,
	65	+ int start,
	66	+ perrdetail *err_ret,
	67	+ int *flags);
59	68
60	69	/* Note that he following function is defined in pythonrun.c not parsetok.c. */
61	70	PyAPI_FUNC(void) PyParser_SetError(perrdetail *);

Collapse file

‎Include/pythonrun.h‎

Copy file name to clipboardExpand all lines: Include/pythonrun.h

+54-20Lines changed: 54 additions & 20 deletions

Original file line number	Diff line number	Diff line change
@@ -38,19 +38,41 @@ PyAPI_FUNC(void) Py_EndInterpreter(PyThreadState *);
38	38	#ifndef Py_LIMITED_API
39	39	PyAPI_FUNC(int) PyRun_SimpleStringFlags(const char *, PyCompilerFlags *);
40	40	PyAPI_FUNC(int) PyRun_AnyFileFlags(FILE *, const char *, PyCompilerFlags *);
41		- PyAPI_FUNC(int) PyRun_AnyFileExFlags(FILE *, const char *, int, PyCompilerFlags *);
42		- PyAPI_FUNC(int) PyRun_SimpleFileExFlags(FILE *, const char *, int, PyCompilerFlags *);
43		- PyAPI_FUNC(int) PyRun_InteractiveOneFlags(FILE *, const char *, PyCompilerFlags *);
44		- PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(FILE *, const char *, PyCompilerFlags *);
45		-
46		- PyAPI_FUNC(struct _mod *) PyParser_ASTFromString(const char *, const char *,
47		- int, PyCompilerFlags *flags,
48		- PyArena *);
49		- PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(FILE *, const char *,
50		- const char*, int,
51		- char *, char *,
52		- PyCompilerFlags *, int *,
53		- PyArena *);
	41	+ PyAPI_FUNC(int) PyRun_AnyFileExFlags(
	42	+ FILE *fp,
	43	+ const char *filename, /* decoded from the filesystem encoding */
	44	+ int closeit,
	45	+ PyCompilerFlags *flags);
	46	+ PyAPI_FUNC(int) PyRun_SimpleFileExFlags(
	47	+ FILE *fp,
	48	+ const char *filename, /* decoded from the filesystem encoding */
	49	+ int closeit,
	50	+ PyCompilerFlags *flags);
	51	+ PyAPI_FUNC(int) PyRun_InteractiveOneFlags(
	52	+ FILE *fp,
	53	+ const char *filename, /* decoded from the filesystem encoding */
	54	+ PyCompilerFlags *flags);
	55	+ PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(
	56	+ FILE *fp,
	57	+ const char *filename, /* decoded from the filesystem encoding */
	58	+ PyCompilerFlags *flags);
	59	+
	60	+ PyAPI_FUNC(struct _mod *) PyParser_ASTFromString(
	61	+ const char *s,
	62	+ const char *filename, /* decoded from the filesystem encoding */
	63	+ int start,
	64	+ PyCompilerFlags *flags,
	65	+ PyArena *arena);
	66	+ PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(
	67	+ FILE *fp,
	68	+ const char *filename, /* decoded from the filesystem encoding */
	69	+ const char* enc,
	70	+ int start,
	71	+ char *ps1,
	72	+ char *ps2,
	73	+ PyCompilerFlags *flags,
	74	+ int *errcode,
	75	+ PyArena *arena);
54	76	#endif
55	77
56	78	#ifndef PyParser_SimpleParseString
@@ -68,20 +90,32 @@ PyAPI_FUNC(struct _node *) PyParser_SimpleParseFileFlags(FILE *, const char *,
68	90	PyAPI_FUNC(PyObject *) PyRun_StringFlags(const char *, int, PyObject *,
69	91	PyObject *, PyCompilerFlags *);
70	92
71		- PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int,
72		- PyObject *, PyObject *, int,
73		- PyCompilerFlags *);
	93	+ PyAPI_FUNC(PyObject *) PyRun_FileExFlags(
	94	+ FILE *fp,
	95	+ const char *filename, /* decoded from the filesystem encoding */
	96	+ int start,
	97	+ PyObject *globals,
	98	+ PyObject *locals,
	99	+ int closeit,
	100	+ PyCompilerFlags *flags);
74	101	#endif
75	102
76	103	#ifdef Py_LIMITED_API
77	104	PyAPI_FUNC(PyObject *) Py_CompileString(const char *, const char *, int);
78	105	#else
79	106	#define Py_CompileString(str, p, s) Py_CompileStringExFlags(str, p, s, NULL, -1)
80	107	#define Py_CompileStringFlags(str, p, s, f) Py_CompileStringExFlags(str, p, s, f, -1)
81		- PyAPI_FUNC(PyObject *) Py_CompileStringExFlags(const char *, const char *, int,
82		- PyCompilerFlags *, int);
83		- #endif
84		- PyAPI_FUNC(struct symtable *) Py_SymtableString(const char *, const char *, int);
	108	+ PyAPI_FUNC(PyObject *) Py_CompileStringExFlags(
	109	+ const char *str,
	110	+ const char *filename, /* decoded from the filesystem encoding */
	111	+ int start,
	112	+ PyCompilerFlags *flags,
	113	+ int optimize);
	114	+ #endif
	115	+ PyAPI_FUNC(struct symtable *) Py_SymtableString(
	116	+ const char *str,
	117	+ const char *filename, /* decoded from the filesystem encoding */
	118	+ int start);
85	119
86	120	PyAPI_FUNC(void) PyErr_Print(void);
87	121	PyAPI_FUNC(void) PyErr_PrintEx(int);

Collapse file

‎Include/symtable.h‎

Copy file name to clipboardExpand all lines: Include/symtable.h

+6-3Lines changed: 6 additions & 3 deletions

Original file line number	Diff line number	Diff line change
@@ -16,7 +16,8 @@ typedef enum _block_type { FunctionBlock, ClassBlock, ModuleBlock }
16	16	struct _symtable_entry;
17	17
18	18	struct symtable {
19		- const char *st_filename; /* name of file being compiled */
	19	+ const char *st_filename; /* name of file being compiled,
	20	+ decoded from the filesystem encoding */
20	21	struct _symtable_entry *st_cur; /* current symbol table entry */
21	22	struct _symtable_entry *st_top; /* symbol table entry for module */
22	23	PyObject *st_blocks; /* dict: map AST node addresses
@@ -60,8 +61,10 @@ PyAPI_DATA(PyTypeObject) PySTEntry_Type;
60	61
61	62	PyAPI_FUNC(int) PyST_GetScope(PySTEntryObject *, PyObject *);
62	63
63		- PyAPI_FUNC(struct symtable *) PySymtable_Build(mod_ty, const char *,
64		- PyFutureFeatures *);
	64	+ PyAPI_FUNC(struct symtable *) PySymtable_Build(
	65	+ mod_ty mod,
	66	+ const char *filename, /* decoded from the filesystem encoding */
	67	+ PyFutureFeatures *future);
65	68	PyAPI_FUNC(PySTEntryObject *) PySymtable_Lookup(struct symtable *, void *);
66	69
67	70	PyAPI_FUNC(void) PySymtable_Free(struct symtable *);

0 commit comments

Comments

0 (0)