gh-109653: Improve the import time of `email.utils` by AlexWaygood · Pull Request #109824 · python/cpython · GitHub

This patch reduces the import time of email.utils by around 46%.

Why do I care about email.utils? Well, email.utils is imported by email.message, and email.message is imported by lots of other stdlib modules: urllib.parse, mailbox, and importlib.metadata. Improving the import time of this module has a cascading effect through lots of the rest of the standard library.

A lot of the import cost of email.utils has to do with the make_msgid function, which requires the random and socket modules. This function is used by many third-party libraries, but isn't used at all by any of the stdlib modules that import email.utils. This patch moves the function into a separate submodule.

Issue: Improve import time of various stdlib modules #109653

Co-authored-by: Nikita Sobolev <mail@sobolevn.me>

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

In general I don't like performance hacks like this, but they do have their place. I can't speak to whether or not this is a worthwhile performance hack, you should seek approval from the maintainers of the impacted modules for that.

That said, the stdlib itself makes no use of make_msgid, and email.utils is not itself considered part of the new email API. Moving it into a separate module and making that part of the non-legacy public API of the email module would actually make some sense. I guess we'd just call it 'msgid'? Then this code should issue a deprecation warning pointing to the new way to import make_msgid. It feels kind of weird to have a module with just one function, but there isn't really anything else related to it that I can think of. (I wonder...maybe make_msgid actually belongs in the UUID module? Probably not. Wrong RFC.)

This feels a little strange to me. Should we instead just defer the imports? Might even be faster to do so

This feels a little strange to me. Should we instead just defer the imports? Might even be faster to do so

Yeah, that might be a better shout. Have been meaning to give it a go — just haven't got round to it yet :)

I'm not really enthused about the idea of a painful deprecation period in order to change the place where people are "meant" to import it from. This function is pretty widely used by third-party packages, and the improvement in import time doesn't seem worth the disruption to me :)

…eaper now

…/cpython into email-message-import

I've updated the PR. I abandoned the idea of adding a new submodule; now I just defer the problematic random, socket and os imports to inside make_msgid(). The reason I didn't do that originally was because I was concerned it would slow down the make_msgid() function, but local experimentation doesn't show that. (I ran python -m timeit -s "from email.utils import make_msgid" "make_msgid()" both with and without the change to test.) Shows the importance of benchmarking!

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Looks good

Not sure I like this one, only fine with it since formataddr doesn't look too widely used (a lot of the time it's nice to pay these costs upfront, predictable performance is important, e.g. don't want the first request your webserver serves to be randomly slow)

(a lot of the time it's nice to pay these costs upfront, predictable performance is important

agreed. On the other hand, though, the email package goes in quite heavily for lazy imports in some other places, so this does seem in keeping with that general philosophy:

cpython/Lib/email/__init__.py

Lines 28 to 61 in f786029

    
           # Some convenience routines.  Don't import Parser and Message as side-effects 
        
           # of importing email since those cascadingly import most of the rest of the 
        
           # email package. 
        
           def message_from_string(s, *args, **kws): 
        
               """Parse a string into a Message object model. 
        
               Optional _class and strict are passed to the Parser constructor. 
        
               """ 
        
               from email.parser import Parser 
        
               return Parser(*args, **kws).parsestr(s) 
        
           def message_from_bytes(s, *args, **kws): 
        
               """Parse a bytes string into a Message object model. 
        
               Optional _class and strict are passed to the Parser constructor. 
        
               """ 
        
               from email.parser import BytesParser 
        
               return BytesParser(*args, **kws).parsebytes(s) 
        
           def message_from_file(fp, *args, **kws): 
        
               """Read a file and parse its contents into a Message object model. 
        
               Optional _class and strict are passed to the Parser constructor. 
        
               """ 
        
               from email.parser import Parser 
        
               return Parser(*args, **kws).parse(fp) 
        
           def message_from_binary_file(fp, *args, **kws): 
        
               """Read a binary file and parse its contents into a Message object model. 
        
               Optional _class and strict are passed to the Parser constructor. 
        
               """ 
        
               from email.parser import BytesParser 
        
               return BytesParser(*args, **kws).parse(fp)

I'd be happy to change it so it's imported at the top of the function if you think that'd be better?

Nah that'd be worse, module level or here. I'm fine with this as is!

…essage-import

)

AlexWaygood added 2 commits September 25, 2023 10:40

WIP

0aca553

news

f3c815d

AlexWaygood added type-feature A feature request or enhancement performance Performance or resource usage topic-email 3.13 bugs and security fixes labels Sep 25, 2023

AlexWaygood requested a review from a team as a code owner September 25, 2023 10:03

bedevere-app Bot added the awaiting review label Sep 25, 2023

bedevere-app Bot mentioned this pull request Sep 25, 2023

Improve import time of various stdlib modules #109653

Closed

sobolevn reviewed Sep 25, 2023

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2023-09-25-10-47-22.gh-issue-109653.TUHrId.rst Outdated

I can spell my own name

992274f

Co-authored-by: Nikita Sobolev <mail@sobolevn.me>

AA-Turner reviewed Sep 25, 2023

View reviewed changes

Comment thread Lib/email/_msgid.py Outdated

Comment thread Lib/email/_msgid.py Outdated

Comment thread Lib/email/utils.py Outdated

AA-Turner reviewed Sep 25, 2023

View reviewed changes

Comment thread Lib/email/utils.py Outdated

Apply suggestions from code review

493aa33

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bitdancer reviewed Sep 25, 2023

View reviewed changes

AlexWaygood and others added 5 commits October 3, 2023 15:45

Merge branch 'main' into email-message-import

a697d9d

Abandon the separate submodule idea

03233f1

stats are slightly less impressive now because importing random is ch…

f11bc98

…eaper now

Merge branch 'email-message-import' of https://github.com/AlexWaygood…

c5e7a61

…/cpython into email-message-import

Merge branch 'main' into email-message-import

bcaeb4c

AA-Turner reviewed Oct 3, 2023

View reviewed changes

Comment thread Lib/email/utils.py

Update Lib/email/utils.py

b387dcd

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

hauntsaninja approved these changes Oct 4, 2023

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Oct 4, 2023

AlexWaygood added 2 commits October 4, 2023 22:08

Merge branch 'main' of https://github.com/python/cpython into email-m…

c72ee5c

…essage-import

eagerly import os, it is free

e201028

hauntsaninja merged commit aa3f419 into python:main Oct 12, 2023

bedevere-app Bot removed the awaiting core review label Oct 12, 2023

AlexWaygood deleted the email-message-import branch October 12, 2023 22:26

AlexWaygood restored the email-message-import branch February 22, 2024 16:37

AlexWaygood deleted the email-message-import branch March 7, 2024 14:35

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024

pythongh-109653 : Improve the import time of email.utils (python#109824

7867907

)

+                  if attr == "make_msgid":
+                      from email._msgid import make_msgid
+                      return make_msgid
+                  raise AttributeError(f"module {__name__!r} has no attribute {attr!r}")

                       except UnicodeEncodeError:
                           if isinstance(charset, str):
+                              # lazy import to improve module import time
+                              from email.charset import Charset

	# Some convenience routines. Don't import Parser and Message as side-effects
	# of importing email since those cascadingly import most of the rest of the
	# email package.
	def message_from_string(s, args, *kws):
	"""Parse a string into a Message object model.

	Optional _class and strict are passed to the Parser constructor.
	"""
	from email.parser import Parser
	return Parser(args, *kws).parsestr(s)

	def message_from_bytes(s, args, *kws):
	"""Parse a bytes string into a Message object model.

	Optional _class and strict are passed to the Parser constructor.
	"""
	from email.parser import BytesParser
	return BytesParser(args, *kws).parsebytes(s)

	def message_from_file(fp, args, *kws):
	"""Read a file and parse its contents into a Message object model.

	Optional _class and strict are passed to the Parser constructor.
	"""
	from email.parser import Parser
	return Parser(args, *kws).parse(fp)

	def message_from_binary_file(fp, args, *kws):
	"""Read a binary file and parse its contents into a Message object model.

	Optional _class and strict are passed to the Parser constructor.
	"""
	from email.parser import BytesParser
	return BytesParser(args, *kws).parse(fp)

gh-109653: Improve the import time of `email.utils` by AlexWaygood · Pull Request #109824 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-109653: Improve the import time of email.utils - #109824

gh-109653: Improve the import time of email.utils#109824hauntsaninja merged 12 commits intopython:mainpython/cpython:mainfrom AlexWaygood:email-message-importAlexWaygood/cpython:email-message-importCopy head branch name to clipboard

Conversation

AlexWaygood commented Sep 25, 2023 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bitdancer Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

hauntsaninja commented Oct 3, 2023

Uh oh!

AlexWaygood commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood commented Oct 3, 2023

Uh oh!

Uh oh!

hauntsaninja left a comment

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-109653: Improve the import time of email.utils#109824
hauntsaninja merged 12 commits into
python:mainfrom
AlexWaygood:email-message-import

AlexWaygood commented Sep 25, 2023 •
edited by bedevere-app Bot
Loading

AlexWaygood commented Oct 3, 2023 •
edited
Loading