| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
@Wulian233 In your benchmark the sqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one. Also time is a builtin module. Is that one really a bottleneck? |
Sorry, something went wrong.
|
I'm also a bit surprised that time would be a bottleneck. When I run ./python -X importtime -c 'import sqlite3' on my system, I get import time: self [us] | cumulative | imported package
[...]
import time: 80 | 80 | time
[...]
import time: 2011 | 8096 | sqlite3
Which, if I'm reading it right, says that importing time can only account for only a small fraction sqlite3's import time? |
Sorry, something went wrong.
I think I got the name wrong at the time, it was a copy of the previous PR content, and here is the test I just run D:\Python313>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'"
Benchmark 1: ./python -c 'from sqlite3 import *'
Time (mean ± σ): 697.1 µs ± 1059.7 µs [User: 2577.5 µs, System: 3063.5 µs]
Range (min … max): 0.0 µs … 10960.7 µs 391 runs
Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
Warning: Ignoring non-zero exit code.
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: ./python -c 'from sqlite3_new import *'
Time (mean ± σ): 635.0 µs ± 755.4 µs [User: 1821.0 µs, System: 3691.3 µs]
Range (min … max): 0.0 µs … 3622.5 µs 399 runs
Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
Warning: Ignoring non-zero exit code.
Summary
./python -c 'from sqlite3_new import *' ran
1.10 ± 2.12 times faster than ./python -c 'from sqlite3 import *'
|
Sorry, something went wrong.
| # 3. This notice may not be removed or altered from any source distribution. | ||
|
|
||
| import datetime | ||
| import time |
There was a problem hiding this comment.
What's the point of the PR if the import is not removed? Lazy imports are therefore not used, then why should any speedup be expected, a slowdown is more likely?
Sorry, something went wrong.
There was a problem hiding this comment.
Already pointed out in #131796 (comment) :)
(But the PR author self-reviewed the review comment by declaring the review comment to be resolved, which is a controversial feature for github to even allow.)
Sorry, something went wrong.
|
To be precise, it's not importing sqlite3 that is faster, it's importing sqlite3 and loading its content in the global namespace (the benchmarks are done for from sqlite3 import * not for a plain import sqlite3 statement). Also, the benchmarks are quite unstable. The standard deviation is much higher than the mean itself! In practice, I think we don't gain anything else than stability (the stdev becomes smaller with the new implementation but again this could be noise). Can you check if import sqlite3 has less noise? And to be precise, we're not gaining a 2x speed-up. On average, we're only gaining a 1.1x speed-up and the standard deviation of this speed-up is $\pm2$, which is again not really indicative =/ And yes, I'm also surprised that we're gaining "that much" when we just make import time local. Considering it's a built-in module (though not sure if it's present at startup), I don't think we need to do this change (we're maybe gaining a small speed-up but maybe not much; and sqlite3 is already is a heavy module to import). So benchmarks using -X importtime here are much more important IMO compared to interpreter's startup as well. |
Sorry, something went wrong.
|
@Wulian233, can you address @picnixz's last remark? |
Sorry, something went wrong.
|
I've reworked another version and it's going to be faster now I ran the command D:\Python314>python -X importtime -c "import sqlite3" and obtained the following output. Below is a summary table comparing sqlite3 and sqlite3_new import times (GPT helped):
Raw output: Details import time: self [us] | cumulative | imported package import time: 296 | 296 | winreg import time: 248 | 248 | _io import time: 70 | 70 | marshal import time: 312 | 312 | nt import time: 1637 | 2266 | _frozen_importlib_external import time: 1103 | 1103 | time import time: 1089 | 2191 | zipimport import time: 75 | 75 | _codecs import time: 793 | 868 | codecs import time: 1926 | 1926 | encodings.aliases import time: 797 | 797 | encodings._win_cp_codecs import time: 3400 | 6989 | encodings import time: 960 | 960 | encodings.utf_8 import time: 75 | 75 | _codecs_cn import time: 126 | 126 | _multibytecodec import time: 1389 | 1590 | encodings.gbk import time: 86 | 86 | _signal import time: 59 | 59 | _abc import time: 323 | 382 | abc import time: 104 | 104 | _stat import time: 527 | 630 | stat import time: 1745 | 1745 | _collections_abc import time: 126 | 126 | genericpath import time: 229 | 229 | _winapi import time: 3109 | 3463 | ntpath import time: 2543 | 8760 | os import time: 584 | 584 | _sitebuiltins import time: 1004 | 1004 | sitecustomize import time: 563 | 563 | usercustomize import time: 2043 | 12953 | site import time: 1104 | 1104 | linecache import time: 787 | 787 | _datetime import time: 1003 | 1790 | datetime import time: 704 | 704 | itertools import time: 1147 | 1147 | keyword import time: 128 | 128 | _operator import time: 1412 | 1540 | operator import time: 2293 | 2293 | reprlib import time: 164 | 164 | _collections import time: 4104 | 9950 | collections import time: 2917 | 12866 | collections.abc import time: 64 | 64 | _types import time: 1134 | 1197 | types import time: 136 | 136 | _functools import time: 2652 | 3984 | functools import time: 2658 | 6641 | _sqlite3 import time: 74 | 74 | _contextvars import time: 1406 | 1479 | _py_warnings import time: 2508 | 3987 | warnings import time: 6431 | 31714 | sqlite3.dbapi2 import time: 8592 | 40306 | sqlite3D:\Python314>python -X importtime -c "import sqlite3_new" hyperfine test: D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'
Benchmark 1: ./python -c 'from sqlite3 import *'
Time (mean ± σ): 176.8 µs ± 179.3 µs [User: 1985.8 µs, System: 2886.4 µs]
Range (min … max): 0.0 µs … 1018.1 µs 406 runs
Benchmark 2: ./python -c 'from sqlite3_new import *'
Time (mean ± σ): 245.4 µs ± 200.7 µs [User: 914.9 µs, System: 2002.8 µs]
Range (min … max): 0.0 µs … 1787.5 µs 401 runs
Summary
./python -c 'from sqlite3 import *' ran
1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'
|
Sorry, something went wrong.
| def main(*args): | ||
| from argparse import ArgumentParser | ||
| from textwrap import dedent | ||
|
|
There was a problem hiding this comment.
I do not see a valid reason to do code churn and delay imports for the sake of speeding up import sqlite3.__main__ while assuming that sqlite3.__main__.main() is never called.
Sorry, something went wrong.
There was a problem hiding this comment.
I presume the following: execute is used by the REPL in addition to CLI; user code could import to use SQLiteInteractiveConsole. Whether worth it, not my call.
Sorry, something went wrong.
There was a problem hiding this comment.
The REPL is of course part of calling main(), which makes my point quite well. :)
User code certainly could import SQLiteInteractiveConsole, as it lies within their technical capability. I think any user code importing an interactive console from a module named __main__ ought to be prepared to use it without worrying about the startup time of importing argparse and textwrap. A module called __main__ feels rather special to me in that sense -- one ought not casually resort to importing it.
Sorry, something went wrong.
There was a problem hiding this comment.
I agree with @eli-schwartz. This change looks meaningless.
Sorry, something went wrong.
|
I agree with @picnixz and @eli-schwartz. The effect of moving import time looks not more than a noise. And it does not make much sense to optimize "import" time of sqlite3.__main__ -- it does not affect the total time. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Improve import time of sqlite3 by 1.5x faster
Benchmark on Windows10, CPython 3.14.0b1
D:\Python314>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *' Benchmark 1: ./python -c 'from sqlite3 import *' Time (mean ± σ): 176.8 µs ± 179.3 µs [User: 1985.8 µs, System: 2886.4 µs] Range (min … max): 0.0 µs … 1018.1 µs 406 runs Benchmark 2: ./python -c 'from sqlite3_new import *' Time (mean ± σ): 245.4 µs ± 200.7 µs [User: 914.9 µs, System: 2002.8 µs] Range (min … max): 0.0 µs … 1787.5 µs 401 runs Summary ./python -c 'from sqlite3 import *' ran 1.39 ± 1.81 times faster than ./python -c 'from sqlite3_new import *'I just realized today that I accidentally deleted an unmerged branch last month, which resulted in closing #129118 . My apologies for the oversight. I've now recreated the pull request - the content remains exactly the same as before