Repository navigation
Argument Clinic's [python input] blocks do not support empty lines in comments #128152
Description
Bug report
Bug description:
The following is not a valid Python input clinic but should be:
/*[python input]
# Line 1
#
# Line 3
[python start generated code]*/This results in an assertion error:
$ python ./Tools/clinic/clinic.py --force --make --exclude Lib/test/clinic.test.c --srcdir .
Traceback (most recent call last):
File "/lib/python/cpython/./Tools/clinic/clinic.py", line 11, in <module>
main()
File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 226, in main
run_clinic(parser, args)
File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 205, in run_clinic
parse_file(path,
File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 84, in parse_file
cooked = clinic.parse(raw)
^^^^^^^^^^^^^^^^^
File "/lib/python/cpython/Tools/clinic/libclinic/app.py", line 186, in parse
for block in self.block_parser:
^^^^^^^^^^^^^^^^^
File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 132, in __next__
return_value = self.parse_clinic_block(self.dsl_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 194, in parse_clinic_block
line = self._line()
^^^^^^^^^^^^
File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 155, in _line
self.language.parse_line(line)
File "/lib/python/cpython/Tools/clinic/libclinic/clanguage.py", line 70, in parse_line
self.cpp.writeline(line)
File "/lib/python/cpython/Tools/clinic/libclinic/cpp.py", line 139, in writeline
assert line
^^^^
AssertionError
make: *** [Makefile:1055: clinic] Error 1CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
picnixz commented on Dec 21, 2024
I've had a quick look at the failure and this happens before we've detected the [python input] block. Similarly, #128153 faces the same issue, namely Clinic assumes that we're still processing a C file and doesn't check for the desired DSL.
@erlend-aasland I don't really know how to fix this and it's not really an issue since I can just write some fake characters to simulate an empty line (e.g., # ::) but I'd like to know if first detecting the DSL before attempting to parse it would be a good alternative or if the current design of AC is just too intricate for this feature.
StanFromIreland commented on Dec 21, 2024
I've opened a PR for this issue. I think the problem was that empty lines were breaking the logic so if they don't reach it there should be no more error.
picnixz commented on Dec 21, 2024
As I said on the PR, I'm not sure this is the best approach. The "empty lines" that are detected at this stage are actually lines of the form <space>#</space>, so it's not entirely clear whether they should be ignored at this point. The reason is that we first parse a file before deciding whether this new line is part of a python input block. If this is the case, we should switch to another parsing method (for instance, I haven't actually checked what happens if I write #if defined(...) inside the Python input clinic block).
erlend-aasland commented on Jan 3, 2025
This has nothing to do with [python input]; you'll get the same error with [clinic input]:
[clinic input] repropicnixz commented on Jan 3, 2025
Yes, that's why I'm asking if this is possible to pre-determine the DSL before parsing the C file (a two-pass parsing) so that python input blocks are parsed using the Python parser (namely, if we're in a python input block, we skip the special handling of # which assumes that we're parsing pre-processor guards). Because the error happens before this (I'm not sure it's possible)
pythongh-128152: Argument Clinic: don't parse CPP directives inside C…
erlend-aasland commented on Jan 3, 2025
The problem seems to be in cpp.py's incorrect parsing of comments: it tries to parse directives inside comments. The fix is straightforward: ignore CPP directives inside C comments.
gh-128152: Argument Clinic: ignore pre-processor directives inside C …
pythongh-128152: Argument Clinic: ignore pre-processor directives ins…
[3.13] gh-128152: Argument Clinic: ignore pre-processor directives in…
pythongh-128152: Argument Clinic: ignore pre-processor directives ins…
pythongh-128152: Argument Clinic: ignore pre-processor directives ins…