lexer
package
This package is not in the latest version of its module.
Go to latest Published: Jul 31, 2026 License: MITDetails
-
Valid go.mod file
The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
-
Redistributable license
Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
-
Tagged version
Modules with tagged versions give importers more predictable builds.
-
Stable version
When a project reaches major version v1 it is considered stable.
- Learn more about best practices
Repository
Links
Open Source Insights
Documentation
¶
Overview ¶
Package lexer implements SQLite's tokenizer.
It is a direct port of sqlite3GetToken() in src/tokenize.c together with the token post-processing sqlite3RunParser() performs around it: runs of whitespace and comments are dropped, and the WINDOW/OVER/FILTER keywords are resolved against their neighbours (analyzeWindowKeyword and friends).
Illegal input is not reported here. sqlite3RunParser aborts at the first TK_ILLEGAL token it reaches, which is exactly as far as the parser had got, so meyer keeps ILLEGAL tokens in the stream and lets the parser report "unrecognized token" when it reaches one. That preserves SQLite's ordering between tokenizer errors and syntax errors.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Lex ¶
Lex splits src into tokens. Whitespace and comments are dropped; the returned slice always ends with a single EOF token whose Pos is the end of the consumed input.
A NUL byte terminates the input, matching SQLite, whose tokenizer walks a NUL-terminated buffer.
Types ¶
This section is empty.
Source Files