Defer imports for faster overall import time: 19 ms -> 5 ms - #234
Conversation
Sorry, something went wrong.
Codecov ReportAll modified and coverable lines are covered by tests ✅ Additional details and impacted files @@ Coverage Diff @@
## main #234 +/- ##
==========================================
- Coverage 99.49% 99.48% -0.01%
==========================================
Files 11 11
Lines 785 782 -3
==========================================
- Hits 781 778 -3
Misses 4 4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry.
|
Sorry, something went wrong.
|
@hugovk care to share how you measured those things and made graphs? I want to check similarly in my project. |
Sorry, something went wrong.
|
Sure! Before each test I ran python3 -c 'import humanize' to make sure everything was cached after making source changes, so any pyc files had been created etc. Then I ran python3 -X importtime -c "import humanize" 2> import.log. The log file looks like this, and shows each import time and the total: import time: self [us] | cumulative | imported package import time: 114 | 114 | _io import time: 18 | 18 | marshal import time: 146 | 146 | posix import time: 376 | 653 | _frozen_importlib_external import time: 354 | 354 | time import time: 149 | 502 | zipimport import time: 32 | 32 | _codecs import time: 363 | 394 | codecs import time: 273 | 273 | encodings.aliases import time: 475 | 1141 | encodings import time: 108 | 108 | encodings.utf_8 import time: 49 | 49 | _signal import time: 17 | 17 | _abc import time: 112 | 128 | abc import time: 108 | 236 | io import time: 28 | 28 | _stat import time: 53 | 80 | stat import time: 491 | 491 | _collections_abc import time: 34 | 34 | errno import time: 36 | 36 | genericpath import time: 88 | 157 | posixpath import time: 308 | 1035 | os import time: 53 | 53 | _sitebuiltins import time: 128 | 128 | encodings.utf_8_sig import time: 296 | 296 | types import time: 93 | 93 | importlib import time: 121 | 121 | importlib._abc import time: 105 | 318 | importlib.util import time: 26 | 26 | importlib.machinery import time: 234 | 234 | _distutils_hack import time: 89 | 89 | __future__ import time: 110 | 199 | pre_commit_uv import time: 452 | 452 | sitecustomize import time: 74 | 74 | usercustomize import time: 3351 | 6159 | site import time: 132 | 132 | linecache import time: 88 | 88 | humanize.filesize import time: 36 | 36 | _operator import time: 161 | 197 | operator import time: 62 | 62 | itertools import time: 85 | 85 | keyword import time: 112 | 112 | reprlib import time: 34 | 34 | _collections import time: 640 | 931 | collections import time: 63 | 63 | _functools import time: 395 | 1388 | functools import time: 803 | 2191 | enum import time: 56 | 56 | _sre import time: 145 | 145 | re._constants import time: 269 | 414 | re._parser import time: 65 | 65 | re._casefix import time: 284 | 817 | re._compiler import time: 130 | 130 | copyreg import time: 325 | 3462 | re import time: 626 | 4283 | gettext import time: 228 | 228 | warnings import time: 136 | 136 | _weakrefset import time: 394 | 758 | threading import time: 78 | 5118 | humanize.i18n import time: 68 | 68 | humanize.lists import time: 110 | 110 | humanize.number import time: 266 | 266 | humanize.time import time: 63 | 63 | humanize._version import time: 137 | 5847 | humanize Rather than poring over the log file, you can install https://github.com/nschloe/tuna and run that on the log: tuna import.log That opens the visualisation in the browser. It's a nice tool, you can click and zoom in to see the sub-imports, and click the top to zoom out again. Then change your code to defer some imports and repeat :) See also https://medium.com/alan/how-we-improved-our-python-backend-start-up-time-2c33cd4873c8 Another good tool is https://github.com/sharkdp/hyperfine, which repeatedly runs a command to get an average, for example: ❯ hyperfine --warmup 3 "python3 -c 'import humanize'"
Benchmark 1: python3 -c 'import humanize'
Time (mean ± σ): 21.4 ms ± 0.7 ms [User: 15.9 ms, System: 4.6 ms]
Range (min … max): 20.2 ms … 24.1 ms 127 runsMore advanced use allows you to give it two commands and it'll compare them both. You can give different setup commands as well, like tell it to checkout a Git branch before each test. This is for broader use as it includes the Python interpreter startup overhead, whereas -X importtime is more focused. |
Sorry, something went wrong.
Defer imports, especially avoid typing import for normal use, which is getting slower in 3.14, to make the overall import time of humanize faster.
Before: 19 ms
After: 5 ms
Also move mypy dependency to autoupdateable requirements file.