FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Speedup: build docs in parallel by hugovk · Pull Request #92733 · python/cpython · GitHub

/ cpython Public

Speedup: build docs in parallel - #92733

Merged
JulienPalard merged 1 commit into
python:mainfrom
hugovk:speedup-docs-build
May 16, 2022
Merged

Speedup: build docs in parallel#92733
JulienPalard merged 1 commit into
python:mainfrom
hugovk:speedup-docs-build

Conversation

hugovk commented May 12, 2022
edited
Loading

Copy link
Copy Markdown
Member

sphinx-build has a -j N option:

Distribute the build over N processes in parallel, to make building on multiprocessor machines more effective. Note that not all parts and not all builders of Sphinx can be parallelized. If auto argument is given, Sphinx uses the number of CPUs as N.

https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-j

Test on a new 10-core Mac

make -C Doc clean venv; time make -C Doc html

Results of two runs before

make -C Doc html  64.83s user 0.84s system 98% cpu 1:06.79 total
make -C Doc html  64.61s user 0.84s system 99% cpu 1:06.04 total

Results of two runs after

make -C Doc html  73.28s user 3.09s system 219% cpu 34.826 total
make -C Doc html  74.05s user 3.09s system 221% cpu 34.791 total

That's nearly twice as fast: 1m06s -> 35s

Test on an old dual-core Mac

Results of one run before

make -C Doc html  204.87s user 5.07s system 94% cpu 3:41.28 total

Results of one runs before

make -C Doc html  278.01s user 13.18s system 196% cpu 2:28.38 total

That's about 1.5x faster: 3m42s -> 2m29s

AA-Turner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Faster is better!

A

Copy link
Copy Markdown
Member

@hugovk Should we backport to 3.9 and 3.10 too? -j auto was added in Sphinx 1.7, so I don't believe there would be any compatability concerns.

A

hugovk commented May 14, 2022

Copy link
Copy Markdown
Member Author

Yes, labels added.

AlexWaygood requested a review from JulienPalard May 14, 2022 21:38

Copy link
Copy Markdown
Member

I'll have to dig into this a bit later, but at first glance it can't be backported to 3.9. I don't remember all the details but in 3.9 we used a non-compatibile sphinx extention (in pyspecific.py). It has been fixed around 3.10.

That's why on docs.python.org we only use parallel builds since 3.10:

https://github.com/python/docsbuild-scripts/blob/30b98030d78db56b28f8590b877f3c33f79b913d/build_docs.py#L197

We'll also have to check how it goes with the already present -j4 in build_docs.py, (Yes I'll gladly drop the flag in build_docs.py to use the one from the cpython Makefile).

hugovk commented May 16, 2022

Copy link
Copy Markdown
Member Author

I'll have to dig into this a bit later, but at first glance it can't be backported to 3.9. I don't remember all the details but in 3.9 we used a non-compatibile sphinx extention (in pyspecific.py). It has been fixed around 3.10.

That's why on docs.python.org we only use parallel builds since 3.10:

python/docsbuild-scripts@30b9803/build_docs.py#L197

Thanks, let's keep it simple and drop the 3.9 backport, this change is mainly beneficial for developers building docs, and I expect that will mostly be main and newer versions.

We'll also have to check how it goes with the already present -j4 in build_docs.py, (Yes I'll gladly drop the flag in build_docs.py to use the one from the cpython Makefile).

Yep, what's a good way to check it?

Copy link
Copy Markdown
Member

Yep, what's a good way to check it?

By running make SPHINXOPTS="-j4" autobuild-dev-html which is mostly what docsbuild-scripts does.

And it goes well, so it looks good to me. It starts sphinx-build -b html -d build/doctrees -j auto -j4 -Ea -A daily=1 -W . build/html.

we'll just have to remove the -j4 from docsbuild scripts once its merged, but in the meantime sphinx is not surprised by the presence of two -js.

Copy link
Copy Markdown
Contributor

Thanks @hugovk for the PR, and @JulienPalard for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 16, 2022
(cherry picked from commit a487623)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

Copy link
Copy Markdown
Contributor

Sorry, @hugovk and @JulienPalard, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker a487623c6b784847a8a1e47b4597b0ae2b8def87 3.10

bedevere-bot removed the needs backport to 3.11 only security fixes label May 16, 2022

Copy link
Copy Markdown

GH-92847 is a backport of this pull request to the 3.11 branch.

miss-islington added a commit that referenced this pull request May 16, 2022
(cherry picked from commit a487623)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
hugovk deleted the speedup-docs-build branch May 16, 2022 12:19
hugovk added a commit to hugovk/cpython that referenced this pull request May 16, 2022
bedevere-bot removed the needs backport to 3.10 only security fixes label May 16, 2022

Copy link
Copy Markdown

GH-92850 is a backport of this pull request to the 3.10 branch.

hugovk commented May 16, 2022

Copy link
Copy Markdown
Member Author

Flag drop from build_docs.py: python/docsbuild-scripts#129

ambv pushed a commit that referenced this pull request May 16, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants


Back | FazBrowse Home | New Git URL