@pablogsal @lysnikolaou @iritkatriel: Do you see any reason to expose token.h as part of the public C API? Or are you fine with removing this header file from the public C API?

Issue: #92651

These functions and _PyParser_TokenNames symbols are no longer exported.

I tried to no longer export these symbols, but it broke test_peg_generator. I reverted this change.

Update also the default value in make_h().

Oh right, I updated my PR.

Should PyAPI_FUNC and PyAPI_DATA be used in declarations?

As I wrote in my previous comment, first, I removed PyAPI_FUNC() and PyAPI_DATA() but it broke test_peg_generator. This test builds C extensions which are linked dynamically.

I added // Symbols exported for test_peg_generator comment to future readers.

Do you see any reason to expose token.h as part of the public C API?

No, and is a bad idea to have it as part of the public C API, so if we can move it to the internal heafers folder, that would be great :)

This PR doesn't respect PEP 387 deprecation process because:

• token.h is not included by Python.h and so is part of "the Python C API"
• it was never possible to get tokens from any public Python C API function: this header file is basically useless

In Python 3.10, Parser/tokenizer.h defines these functions:

extern struct tok_state *PyTokenizer_FromString(const char *, int);
extern struct tok_state *PyTokenizer_FromUTF8(const char *, int);
extern struct tok_state *PyTokenizer_FromFile(FILE *, const char*,
                                              const char *, const char *);
extern void PyTokenizer_Free(struct tok_state *);
extern int PyTokenizer_Get(struct tok_state *, const char **, const char **);

Python is now built with -fvisibility=hidden: functions which are not declared with PyAPI_FUNC() are not exported. So these functions are not exported.

In Python 3.11, these functions were renamed to get a "_Py" prefix. For example, PyTokenizer_FromString() was renamed to _PyTokenizer_FromString().

These functions are not exported and are private since Python 3.11 (_Py prefix).

Merged. Thanks for the reviews!