JavaScript: recognize regex literals after => and ... - #3266
Open
ekanshul wants to merge 1 commit into
Open
Conversation
Copy link
Copy Markdown
Sorry, something went wrong.
The root state matched `=>` and `...` as punctuation without entering the slashstartsregex state, so `x => /re/` lexed the regex as two division operators and emitted Error tokens for any backslash inside it. Both tokens are always followed by an expression, so a slash after them starts a regular expression literal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copy link
Copy Markdown
Author
|
Data point: with this change, lexing all 4,346 .ts files of gitea's web_src plus a Next.js monorepo produces zero Error tokens (the TypeScript lexer inherits the rule). Before, a single .filter((line) => /^[a-z0-9]+: '/.test(line)) desynchronized the lexer for the remaining ~800 lines of the file because the regex was read as division and the ' inside it opened a string. |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The JavaScript root state matched => and ... as punctuation without entering the slashstartsregex state, so a regex literal right after an arrow (a very common shape, e.g. const isUrl = s => /^https?:\/\//.test(s)) was lexed as two division operators, and every backslash inside it became an Error token. The same happens after a spread.
Both tokens are always followed by an expression, so a slash after them can only start a regular expression literal. This change makes both rules push slashstartsregex, like the other operators do. It applies to all lexers derived from JavascriptLexer (TypeScript, JSX, TSX, ...).
Found by lexing real-world code (gitea's web_src) and looking for Error tokens; for example gitea/web_src/js/utils/url.ts:
A snippet test is added covering an arrow with a block-less body, an arrow followed by a method call on the regex, and a spread.