JSX/regexp ambiguity in jslexer · Issue #640 · python-babel/babel · GitHub

After a regular expression like />/, Babel ignores all messages until the next one.

Mapping:

[javascript: **.js]
encoding = utf-8
extract_messages = gettext

Source file:

alert(gettext("First message"));
var expr = />/g;
alert(gettext("Second message"));
var expr2 = /'/g;
alert(gettext("Third message"));

Result POT:

"Generated-By: Babel 2.6.0\n"

#: test.js:1
msgid "First message"
msgstr ""

#: test.js:5
msgid "Third message"
msgstr ""

Work around Babel bug

The JavaScript lexer in Babel is far from perfect, unfortunately...
If you feel like trying to hack at the lexer to fix this, by all means please do!

It might be a good idea for a future version to use e.g. https://github.com/Kronuz/esprima-python when it's available (i.e. an optional dependency) for more robust parsing.

Thanks, I might take a look. I am low on time unfortunately right now.

For reference, lexer is jslexer.py

Seems that /> gets recognized as a JSX tag.

I don't see how to fix this. The tokenizer can't possibly tell if /> is the start of a regexp or the end of a JSX tag, they are the same tokens. The Javascript grammar is too ambiguous for this to be decided at the tokenizer level.

A parser would be able to tell if it's currently in a JSX tag or not and resolve the ambiguity, but there is no such parsing being done in Babel. Simply keeping track of JSX open/close tokens wouldn't be enough because regexps can appear inside JSX tags (<hr class={ /\w+/.exec(var)[0] }/>).

I think I'll have to leave that alone, sorry!

Wrote a test:

def test_regex():
    assert list(jslexer.tokenize('''
        re = />/g;
    ''')) == [
        ('name', 're', 2),
        ('operator', '=', 2),
        ('regexp', '/>/', 2),
        ('operator', ';', 2),
    ]

Work around Babel bug

[FIX] xml: escape chars in regex

JSX/regexp ambiguity in jslexer · Issue #640 · python-babel/babel · GitHub

Repository navigation

JSX/regexp ambiguity in jslexer #640

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

JSX/regexp ambiguity in jslexer · Issue #640 · python-babel/babel · GitHub

Repository navigation

JSX/regexp ambiguity in jslexer #640

Description

Activity

akx commented on May 27, 2019

remram44 commented on May 29, 2019

remram44 commented on Jun 15, 2019

remram44 commented on Jun 15, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions