Repository navigation
Closed
Conversation
Copy link
Copy Markdown
Collaborator
Sorry, something went wrong.
Mypyc doesn't generate very efficient code for enums, so switch to a regular class. We can later revert the change if/when we can improve enum support in mypyc. Operations related to ArgKind were pretty prominent in the op trace log (#19457). By itself this improves performance by ~1.7%, based on `perf_compare.py`, which is significant: ``` master 4.168s (0.0%) | stdev 0.037s HEAD 4.098s (-1.7%) | stdev 0.028s ``` This is a part of a set of micro-optimizations that improve self check performance by ~5.5%.
Copy link
Copy Markdown
Contributor
|
Diff from mypy_primer, showing the effect of this PR on open source code: django-stubs (https://github.com/typeddjango/django-stubs)
+ django-stubs/contrib/sites/models.pyi:16: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
+ https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
+ Please report a bug at https://github.com/python/mypy/issues
+ version: 1.18.0+dev.3194db2e144e768800ecb9650111103a3f6e9bdf
+ django-stubs/contrib/sites/models.pyi:16: : note: use --pdb to drop into pdb
+ Traceback (most recent call last):
+ File "", line 8, in <module>
+ sys.exit(console_entry())
+ File "/__main__.py", line 15, in console_entry
+ main()
+ File "/main.py", line 127, in main
+ res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
+ File "/main.py", line 211, in run_build
+ res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
+ File "/build.py", line 191, in build
+ result = _build(
+ File "/build.py", line 267, in _build
+ graph = dispatch(sources, manager, stdout)
+ File "/build.py", line 2939, in dispatch
+ process_graph(graph, manager)
+ File "/build.py", line 3337, in process_graph
+ process_stale_scc(graph, scc, manager)
+ File "/build.py", line 3438, in process_stale_scc
+ graph[id].type_check_first_pass()
+ File "/build.py", line 2311, in type_check_first_pass
+ self.type_checker().check_first_pass()
+ File "/checker.py", line 483, in check_first_pass
+ self.accept(d)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1259, in accept
+ return visitor.visit_class_def(self)
+ File "/checker.py", line 2560, in visit_class_def
+ self.accept(defn.defs)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1340, in accept
+ return visitor.visit_block(self)
+ File "/checker.py", line 3021, in visit_block
+ self.accept(s)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1427, in accept
+ return visitor.visit_assignment_stmt(self)
+ File "/checker.py", line 3070, in visit_assignment_stmt
+ self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
+ File "/checker.py", line 3282, in check_assignment
+ rvalue_type = self.expr_checker.accept(rvalue, type_context=type_context)
+ File "/checkexpr.py", line 5982, in accept
+ typ = node.accept(self)
+ File "/nodes.py", line 2080, in accept
+ return visitor.visit_call_expr(self)
+ ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
+ File "/checkexpr.py", line 520, in visit_call_expr
+ return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/checkexpr.py", line 654, in visit_call_expr_inner
+ ret_type = self.check_call_expr_with_callee_type(
+ callee_type, e, fullname, object_type, member
+ )
+ File "/checkexpr.py", line 1504, in check_call_expr_with_callee_type
+ ret_type, callee_type = self.check_call(
+ ~~~~~~~~~~~~~~~^
+ callee_type,
+ ^^^^^^^^^^^^
+ ...<6 lines>...
+ object_type=object_type,
+ ^^^^^^^^^^^^^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1606, in check_call
+ return self.check_callable_call(
+ ~~~~~~~~~~~~~~~~~~~~~~~~^
+ callee,
+ ^^^^^^^
+ ...<6 lines>...
+ object_type,
+ ^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1851, in check_callable_call
+ new_ret_type = self.apply_function_plugin(
+ callee,
+ ...<7 lines>...
+ context,
+ )
+ File "/checkexpr.py", line 1278, in apply_function_plugin
+ return callback(
+ FunctionContext(
+ ...<8 lines>...
+ )
+ )
+ File "_django_plugin/transformers/fields.py", line 255, in transform_into_proper_return_type
+ return set_descriptor_types_for_field_callback(ctx, django_context)
+ File "_django_plugin/transformers/fields.py", line 136, in set_descriptor_types_for_field_callback
+ return set_descriptor_types_for_field(ctx)
+ File "_django_plugin/transformers/fields.py", line 145, in set_descriptor_types_for_field
+ null_expr = helpers.get_call_argument_by_name(ctx, "null")
+ File "_django_plugin/lib/helpers.py", line 206, in get_call_argument_by_name
+ if kind == ArgKind.ARG_NAMED and argname == name:
+ ^^^^^^^^^^^^^^^^^
+ AttributeError: type object 'ArgKind' has no attribute 'ARG_NAMED'
|
Sorry, something went wrong.
Copy link
Copy Markdown
Collaborator
|
How difficult would be the relevant changes to enum handling in mypyc? I can try to work on that. This PR incurs a huge maintainability and readability penalty IMO, but probably 1.7% is worth that:( |
Sorry, something went wrong.
Copy link
Copy Markdown
Collaborator
Author
|
Yeah this is pretty unfortunate. Since this change seems to break some plugins, it may be better to improve the enum support in mypyc instead and forget about this. I can create a mypyc issue detailing the requirements needed to make this better. |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
|
@sterliakov I think we can optimize enums in general in mypyc (e.g. use some static pointers for enum values, use pointer identity comparison for ==, and inline enum references when compiling, i.e. SomeEnum.some_val will become _CPy_SomeEnum_some_val). Most likely the perf benefit is from not having to go through attribute access logic, but TBH I didn't really think much about this. That said I agree having optimized enums in mypyc is IMO worth it. |
Sorry, something went wrong.
Copy link
Copy Markdown
Collaborator
Author
|
@sterliakov @ilevkivskyi I created a mypyc issue about this: mypyc/mypyc#1121 Calling methods of enum classes is the biggest bottleneck that I can see. Accessing enum values is already optimized. |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mypyc doesn't generate very efficient code for enums, so switch to a regular class. We can later revert the change if/when we can improve enum support in mypyc.
Operations related to ArgKind were pretty prominent in the op trace log (#19457).
By itself this improves performance by ~1.7%, based on perf_compare.py, which is significant:
This is a part of a set of micro-optimizations that improve self check performance by ~5.5%.