gh-111348: Fix direct invocation of `test_doctest`; remove `test_doctest.test_coverage` by sobolevn · Pull Request #111349 · python/cpython · GitHub

Instead of this coverage (which is almost 20 years old https://github.com/python/cpython/blame/78e6d72e38ef4b490f0098b644454031f20ae361/Lib/test/test_doctest.py#L3365-L3373) we should work on proper coverage.py integration in our test suite.

Issue: test_doctest cannot be run directly: let's remove this option #111348

…ve `test_coverage`

That's what I got in the mastermain and on this branch with ./python -m test -T test_doctest:

Using random seed: 3478682052
0:00:00 load avg: 0.96 Run 1 test sequentially
0:00:00 load avg: 0.96 [1/1] test_doctest
test_doctest passed in 38.9 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 38.9 sec
Total tests: run=63
Total test files: run=1/1
Result: SUCCESS
Not printing coverage data for '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py': [Errno 2] No such file or directory: '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/sk/src/cpython/Lib/test/__main__.py", line 2, in <module>
    main(_add_python_opts=True)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 665, in main
    Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 657, in main
    exitcode = self.run_tests(selected, tests)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 494, in run_tests
    return self._run_tests(selected, tests)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 474, in _run_tests
    self.finalize_tests(tracer)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 374, in finalize_tests
    results.write_results(show_missing=True, summary=True,
  File "/home/sk/src/cpython/Lib/trace.py", line 273, in write_results
    with open(filename, 'rb') as fp:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py'

After removing problem doctest (on this branch)

Details

diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
index e51f54ef35..72f49a6834 100644
--- a/Lib/test/test_doctest.py
+++ b/Lib/test/test_doctest.py
@@ -2843,39 +2843,6 @@ def test_lineendings(): r"""
     TestResults(failed=0, attempted=1)
     >>> os.remove(fn)
 
-Now we test with a package loader that has a get_data method, since that
-bypasses the standard universal newline handling so doctest has to do the
-newline conversion itself; let's make sure it does so correctly (issue 1812).
-We'll write a file inside the package that has all three kinds of line endings
-in it, and use a package hook to install a custom loader; on any platform,
-at least one of the line endings will raise a ValueError for inconsistent
-whitespace if doctest does not correctly do the newline conversion.
-
-    >>> from test.support import os_helper
-    >>> import shutil
-    >>> dn = tempfile.mkdtemp()
-    >>> pkg = os.path.join(dn, "doctest_testpkg")
-    >>> os.mkdir(pkg)
-    >>> os_helper.create_empty_file(os.path.join(pkg, "__init__.py"))
-    >>> fn = os.path.join(pkg, "doctest_testfile.txt")
-    >>> with open(fn, 'wb') as f:
-    ...     f.write(
-    ...         b'Test:\r\n\r\n'
-    ...         b'  >>> x = 1 + 1\r\n\r\n'
-    ...         b'Done.\r\n'
-    ...         b'Test:\n\n'
-    ...         b'  >>> x = 1 + 1\n\n'
-    ...         b'Done.\n'
-    ...         b'Test:\r\r'
-    ...         b'  >>> x = 1 + 1\r\r'
-    ...         b'Done.\r'
-    ...     )
-    95
-    >>> with test_hook(dn):
-    ...     doctest.testfile("doctest_testfile.txt", package="doctest_testpkg", verbose=False)
-    TestResults(failed=0, attempted=3)
-    >>> shutil.rmtree(dn)
-
 """
 
 def test_testmod(): r"""

I got

$ ./python -m test -T test_doctest | grep doctest
0:00:00 load avg: 0.44 [1/1] test_doctest
test_doctest passed in 38.3 sec
 1092    69%   doctest   (/home/sk/src/cpython/Lib/doctest.py)
    5    80%   test.doctest_aliases   (/home/sk/src/cpython/Lib/test/doctest_aliases.py)
   17    82%   test.doctest_lineno   (/home/sk/src/cpython/Lib/test/doctest_lineno.py)
   13   100%   test.sample_doctest   (/home/sk/src/cpython/Lib/test/sample_doctest.py)
    3    66%   test.sample_doctest_no_docstrings   (/home/sk/src/cpython/Lib/test/sample_doctest_no_docstrings.py)
    5    80%   test.sample_doctest_no_doctests   (/home/sk/src/cpython/Lib/test/sample_doctest_no_doctests.py)
  184    80%   test.test_doctest   (/home/sk/src/cpython/Lib/test/test_doctest.py)
Warning -- Unraisable exception
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError:

So, at least we have some coverage for the doctest module.

With coveragepy:

python -m coverage run --branch --source=doctest Lib/test/regrtest.py test_doctest
0:00:00 load avg: 1.33 Run 1 test sequentially
0:00:00 load avg: 1.33 [1/1] test_doctest

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.0 sec
Total tests: run=59
Total test files: run=1/1
Result: SUCCESS
Warning -- Unraisable exception
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError: 
$ python -m coverage report
Name             Stmts   Miss Branch BrPart  Cover
--------------------------------------------------
Lib/doctest.py     983    136    448     64    84%
--------------------------------------------------
TOTAL              983    136    448     64    84%
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError:

Apparently, we have some coverage statistics for this module. (And few new bugs...)

@skirpichev I think that -T is not related to this issue. This one is about direct module invocation. However, this does look like a bug. Can you please open a new issue about it?

Because the failure that is still there is rather complex:

.....................................F...........................
======================================================================
FAIL: basics (test.test_doctest.test_DocTestFinder)
Doctest: test.test_doctest.test_DocTestFinder.basics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_doctest.test_DocTestFinder.basics
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 445, in basics

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 469, in test.test_doctest.test_DocTestFinder.basics
Failed example:
    print(tests)  # doctest: +ELLIPSIS
Expected:
    [<DocTest sample_func from test_doctest.py:32 (1 example)>]
Got:
    [<DocTest sample_func from /Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py:32 (1 example)>]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 576, in test.test_doctest.test_DocTestFinder.basics
Failed example:
    for t in tests:
        print('%2s  %s' % (len(t.examples), t.name))
Expected:
     1  some_module
     3  some_module.SampleClass
     3  some_module.SampleClass.NestedClass
     1  some_module.SampleClass.NestedClass.__init__
     1  some_module.SampleClass.__init__
     1  some_module.SampleClass.a_cached_property
     2  some_module.SampleClass.a_classmethod
     1  some_module.SampleClass.a_classmethod_property
     1  some_module.SampleClass.a_property
     1  some_module.SampleClass.a_staticmethod
     1  some_module.SampleClass.double
     1  some_module.SampleClass.get
     1  some_module.__test__.c
     2  some_module.__test__.d
     1  some_module.sample_func
Got:
     1  some_module
     1  some_module.__test__.c
     2  some_module.__test__.d

We would need to have some compat code there as well, while it does not bring much value (while making the test more complex and brittle)

I think that -T is not related to this issue

Hmm. That's an alias to --coverage (documented as a stdlib alternative to coveragepy).

Can you please open a new issue about it?

#111355

Because the failure that is still there is rather complex

Yeah, but what we gain raising a RuntimeError? Someone someday will open a bug if #111348 will be closed.

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

After merging in main, all tests pass for me locally with this PR branch when I run python Lib/test/test_doctest.py. Nice work @sobolevn and @skirpichev!

Thanks @sobolevn for the PR, and @AlexWaygood for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

…t_doctest.test_coverage` (pythonGH-111349) (cherry picked from commit 31c05b7) Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

GH-111359 is a backport of this pull request to the 3.12 branch.

GH-111360 is a backport of this pull request to the 3.11 branch.

…st_doctest.test_coverage` (GH-111349) (#111359) gh-111348: Fix direct invocation of `test_doctest`; remove `test_doctest.test_coverage` (GH-111349) (cherry picked from commit 31c05b7) Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…st_doctest.test_coverage` (GH-111349) (#111360) gh-111348: Fix direct invocation of `test_doctest`; remove `test_doctest.test_coverage` (GH-111349) (cherry picked from commit 31c05b7) Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…t_doctest.test_coverage` (python#111349) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

pythongh-111348 : Do not allow test_doctest to be run directly, remo…

63b1f95

…ve `test_coverage`

sobolevn requested review from iritkatriel and serhiy-storchaka October 26, 2023 08:33

bedevere-app Bot mentioned this pull request Oct 26, 2023

test_doctest cannot be run directly: let's remove this option #111348

Closed

bedevere-app Bot added awaiting review tests Tests in the Lib/test dir labels Oct 26, 2023

sobolevn added the skip news label Oct 26, 2023

skirpichev reviewed Oct 26, 2023

View reviewed changes

Comment thread Lib/test/test_doctest.py

skirpichev reviewed Oct 26, 2023

View reviewed changes

Comment thread Lib/test/test_doctest.py Outdated

Update Lib/test/test_doctest.py

776dff1

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

AlexWaygood approved these changes Oct 26, 2023

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 26, 2023

AlexWaygood changed the title gh-111348: Do not allow test_doctest to be run directly, remove test_coverage gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage Oct 26, 2023

AlexWaygood enabled auto-merge (squash) October 26, 2023 12:34

AlexWaygood added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Oct 26, 2023

AlexWaygood merged commit 31c05b7 into python:main Oct 26, 2023

bedevere-app Bot removed the awaiting merge label Oct 26, 2023

bedevere-app Bot removed the needs backport to 3.12 only security fixes label Oct 26, 2023

bedevere-app Bot removed the needs backport to 3.11 only security fixes label Oct 26, 2023

aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024

pythongh-111348 : Fix direct invocation of test_doctest ; remove `tes…

14aa5f5

…t_doctest.test_coverage` (python#111349) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024

pythongh-111348 : Fix direct invocation of test_doctest ; remove `tes…

498ba7a

…t_doctest.test_coverage` (python#111349) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

gh-111348: Fix direct invocation of `test_doctest`; remove `test_doctest.test_coverage` by sobolevn · Pull Request #111349 · python/cpython · GitHub

Repository navigation

gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage - #111349

gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage#111349
AlexWaygood merged 2 commits into
python:mainfrom
sobolevn:sobolevn-patch-3

sobolevn commented Oct 26, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

skirpichev commented Oct 26, 2023

Uh oh!

sobolevn commented Oct 26, 2023

Uh oh!

Uh oh!

sobolevn commented Oct 26, 2023

Uh oh!

skirpichev commented Oct 26, 2023

Uh oh!

Uh oh!

AlexWaygood left a comment

Uh oh!

miss-islington-app Bot commented Oct 26, 2023

Uh oh!

bedevere-app Bot commented Oct 26, 2023

Uh oh!

bedevere-app Bot commented Oct 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-111348: Fix direct invocation of `test_doctest`; remove `test_doctest.test_coverage` by sobolevn · Pull Request #111349 · python/cpython · GitHub

Repository navigation

gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage - #111349

gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage#111349AlexWaygood merged 2 commits intopython:mainpython/cpython:mainfrom sobolevn:sobolevn-patch-3sobolevn/cpython:sobolevn-patch-3Copy head branch name to clipboard

Conversation

sobolevn commented Oct 26, 2023 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skirpichev commented Oct 26, 2023

Uh oh!

sobolevn commented Oct 26, 2023

Uh oh!

Uh oh!

sobolevn commented Oct 26, 2023

Uh oh!

skirpichev commented Oct 26, 2023

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington-app Bot commented Oct 26, 2023

Uh oh!

bedevere-app Bot commented Oct 26, 2023

Uh oh!

bedevere-app Bot commented Oct 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage#111349
AlexWaygood merged 2 commits into
python:mainfrom
sobolevn:sobolevn-patch-3

sobolevn commented Oct 26, 2023 •
edited by bedevere-app Bot
Loading