Repository navigation
Allow extracting (deeply) nested calls in Python and Javascript - #1127
dylankiss wants to merge 2 commits into
Conversation
Sorry, something went wrong.
|
During the refactor, the order of extraction was also changed, as you can see in this test: It is now the same as the extraction order of xgettext. Also the comments extraction was fixed to be the same as xgettext and apply to all gettext calls (also nested ones) on the same line. e.g. # NOTE: Main Comment
_("Hello %s",
# NOTE: Nested Comment
_("Nested Gettext")
)Both terms would get their right comment extracted. |
Sorry, something went wrong.
|
Not saying this is not worth fixing, but out of curiosity, do nested gettext calls actually come up often? I don't think I've ever come across one.. |
Sorry, something went wrong.
|
@tomasr8 In our own codebase with lots of developers, people assume it works and it happens from time to time that they add in nested gettext calls. Even the deeply nested ones happen, like this example: https://github.com/odoo/odoo/pull/149921/files#diff-e073b7fa9d45d46ba8d7f011257b0e77e1f87bf47982abc63dd618ff05dddb1aL267-L268 |
Sorry, something went wrong.
|
UPDATE: I added an extra commit to also allow nested calls in the Javascript extractor. If it's better to open a separate PR for that, no problem. |
Sorry, something went wrong.
akx
left a comment
There was a problem hiding this comment.
Some initial comments within, including some that would make this easier to review for me 😄
Sorry, something went wrong.
| function_stack.append({ | ||
| 'function_line_no': line_no, | ||
| 'function_name': last_name, | ||
| 'message_line_no': None, | ||
| 'messages': [], | ||
| 'translator_comments': cur_translator_comments, | ||
| }) |
There was a problem hiding this comment.
I think a typing.NamedTuple or a dataclass would be more appropriate than a dict for this state.
Sorry, something went wrong.
There was a problem hiding this comment.
Makes sense. I changed it into a dataclass, since it's supposed to be mutable.
Sorry, something went wrong.
| # Keep track of the (split) strings encountered | ||
| message_buffer = [] | ||
|
|
||
| for token, value, (line_no, _), _, _ in tokens: |
There was a problem hiding this comment.
Tiny thing, but could the local line_no be renamed back to lineno? It would make reviewing easier since the diff is smaller 😅
(Similarly, line_no elsewhere should maybe be lineno for consistency and compat.)
Sorry, something went wrong.
There was a problem hiding this comment.
You're absolutely right. I was a bit too eager with the renaming here. I changed everything back to their original name.
Sorry, something went wrong.
| jsx=options.get('jsx', True), | ||
| template_string=options.get('template_string', True), |
There was a problem hiding this comment.
Spurious changes, please revert?
Sorry, something went wrong.
There was a problem hiding this comment.
Reverted
Sorry, something went wrong.
| assert messages[0][2] == ('Hello, {name}!', None) | ||
| assert messages[0][2] == 'Foo Bar' | ||
| assert messages[0][3] == ['NOTE: First'] | ||
| assert messages[1][2] == 'Foo Bar' | ||
| assert messages[1][3] == [] | ||
| assert messages[2][2] == ('Hello, {name1} and {name2}!', None) | ||
| assert messages[1][2] == ('Hello, {name}!', None) | ||
| assert messages[1][3] == ['NOTE: First'] | ||
| assert messages[2][2] == 'Heungsub' | ||
| assert messages[2][3] == ['NOTE: Second'] | ||
| assert messages[3][2] == 'Heungsub' | ||
| assert messages[3][2] == 'Armin' | ||
| assert messages[3][3] == [] | ||
| assert messages[4][2] == 'Armin' | ||
| assert messages[4][3] == [] | ||
| assert messages[5][2] == ('Hello, {0} and {1}!', None) | ||
| assert messages[4][2] == ('Hello, {name1} and {name2}!', None, None) | ||
| assert messages[4][3] == ['NOTE: Second'] | ||
| assert messages[5][2] == 'Heungsub' | ||
| assert messages[5][3] == ['NOTE: Third'] | ||
| assert messages[6][2] == 'Heungsub' | ||
| assert messages[6][2] == 'Armin' | ||
| assert messages[6][3] == [] | ||
| assert messages[7][2] == 'Armin' | ||
| assert messages[7][3] == [] | ||
| assert messages[7][2] == ('Hello, {0} and {1}!', None, None) | ||
| assert messages[7][3] == ['NOTE: Third'] | ||
| assert messages[8][2] == 'Person' | ||
| assert messages[8][3] == ['NOTE: Fourth'] | ||
| assert messages[9][2] == ('Hello %(person)', None) | ||
| assert messages[9][3] == ['NOTE: Fourth'] | ||
| assert messages[10][2] == 'Person 1' | ||
| assert messages[10][3] == [] | ||
| assert messages[11][2] == 'Person 2' | ||
| assert messages[11][3] == [] | ||
| assert messages[12][2] == ('Hello %(people)', None) | ||
| assert messages[12][3] == ['NOTE: Fifth'] |
There was a problem hiding this comment.
Could this test be rewritten in a... less verbose way? Looks like it's only looking at indices 2 and 3 of each message, so maybe redo it as something like
assert [(m[2], m[3]) for m in messages] == [
(..., ...),
(..., ...),
(..., ...),
...
]?
I reckon it would be easy to generate the ... segment by doing assert [(m[2], m[3]) for m in messages] == 8 or similar and copy-pasting the complaint pytest -vv would inevitably throw :)
Sorry, something went wrong.
There was a problem hiding this comment.
Very good point. I changed it like that, and it looks a lot cleaner.
Sorry, something went wrong.
|
I'm not a big fan of the token-based extractor getting even more complex. I'm thinking we might be able to replace the python extractor with a NodeVisitor which would simplify the code and it would also solve all of the issues with nested calls, f-strings etc., once and for all. |
Sorry, something went wrong.
|
So I did some investigation and an AST-based extractor cuts down the complexity quite a bit. However, it's about twice as slow compared to the current extractor. @akx Given the slowdown, is this something worth pursuing in your opinion? |
Sorry, something went wrong.
|
@tomasr8 @akx Depending on what we want, I can adapt the PR accordingly. I agree it's not the nicest and most robust way of traversing through a code file, but if the performance is degraded that much by using an AST-based extractor it might still be best to continue this way 🤷♂️ |
Sorry, something went wrong.
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## master #1127 +/- ##
==========================================
+ Coverage 91.37% 91.46% +0.08%
==========================================
Files 27 27
Lines 4672 4673 +1
==========================================
+ Hits 4269 4274 +5
+ Misses 403 399 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness.
|
Sorry, something went wrong.
Currently the Python extractor does not support deeply nested gettext
calls (deeper than as a direct argument to the top-level gettext call).
e.g.
```py
_("Hello %s", _("Person"))
_("Hello %s",
random_function(", ".join([_("Person 1"), _("Person 2")])))
```
The extraction code was refactored quite a bit to simplify the flow and
support this use-case.
Fixes python-babel#1125
(meanwhile also fixes python-babel#1123)
Currently the Javascript extractor does not support nested gettext calls at all. The extraction code was refactored a bit to resemble the Python code as much as possible and support this use-case.
|
(since you said you missed the other ping.. 😆 ) @akx any opinions on this? FTR, I recently made the switch in CPython as well: https://github.com/python/cpython/blob/main/Tools/i18n/pygettext.py#L421 |
Sorry, something went wrong.
Currently the Python extractor does not support deeply nested gettext calls (deeper than as a direct argument to the top-level gettext call).
e.g.
The extraction code was refactored quite a bit to simplify the flow and support this use-case.
Currently the Javascript extractor does not support nested gettext calls at all.
The extraction code was refactored a bit to resemble the Python code as much as possible and support this use-case.
Fixes #1125 (meanwhile also fixes #1123)