Support CREATE VECTOR INDEX - #2437
Open
moshap-firebolt wants to merge 3 commits into
Open
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
LucaCappelletti94
suggested changes
Aug 18, 2026
LucaCappelletti94
left a comment
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
At this time, this PR edits would suggest that CREATE VECTOR INDEX is a BigQuery-specific syntax, but it is not, as it is supported also by:
- Oracle SQL
- Microsoft SQL
- Therefore also Generic Dialect
Therefore, move the tests to sqlparser common and ensure that all dialects that support the syntax successfully parse it. It may be acceptable to maintainers that dialects that do not support the syntax also parse it given the current preference for permissive parsing. I suggest you also review the possible syntax variations that these other dialects may have on this syntax, so as to cover more of its variants completely.
Sorry, something went wrong.
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
from
1e9b099 to
8c5ccd7
Compare
August 18, 2026 18:51
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
2 times, most recently
from
de8c8a1 to
1e9b099
Compare
August 18, 2026 19:31
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
from
b6aa124 to
dd1de78
Compare
August 19, 2026 00:27
Copy link
Copy Markdown
Contributor
Author
|
@LucaCappelletti94 - done, added tests demonstrating different syntaxes. Note - didn't implement full blown Oracle's one. |
Sorry, something went wrong.
BigQuery can create a vector index for approximate nearest-neighbor search over
an embedding column:
CREATE [OR REPLACE] VECTOR INDEX [IF NOT EXISTS] <name>
ON <table>(<column>)
OPTIONS(index_type = 'IVF', distance_type = 'COSINE', ...)
`VECTOR` is not a keyword, so `parse_create` previously failed with
`Expected: an object type after CREATE, found: VECTOR`.
- Add `vector`, `or_replace` and `options` fields to `CreateIndex`. `VECTOR` is
a modifier on `CREATE INDEX` (like `EXTERNAL` on `CREATE TABLE`), so it reuses
the existing node rather than a new statement variant. `Display` renders
`CREATE [OR REPLACE ]VECTOR INDEX ...` plus a trailing `OPTIONS(...)`.
- Parse the form in `parse_create` via a small `parse_create_vector_index`
helper; the `OPTIONS(...)` clause reuses `parse_options` / `SqlOption`, so it
parses and renders the same way as `CREATE TABLE` / `CREATE VIEW` OPTIONS.
- Plain `CREATE INDEX` is unchanged (all three fields default to false/empty).
- New test `parse_bigquery_create_vector_index` in `tests/sqlparser_bigquery.rs`
verifies the round-trip and covers `OR REPLACE`, `IF NOT EXISTS`, multi-part
names and the OPTIONS-less form.
Docs: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_vector_index_statement
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback that `CREATE VECTOR INDEX` is not BigQuery-specific (Oracle, SQL Server, MariaDB and TiDB also have it, hence Generic too). - Route the statement through `parse_create_index` instead of a separate BigQuery helper, so it inherits the standard index trailers (`USING`, `INCLUDE`, `WITH`, expression targets, index options) that cover the Oracle / SQL Server / TiDB variants, plus the BigQuery `OPTIONS(...)` clause. - Align `Display` order (OPTIONS after WITH) with parse order so `INCLUDE` + `OPTIONS` combinations round-trip. - Drop the BigQuery-specific doc-comment framing. - Move the test to `tests/sqlparser_common.rs` as `parse_create_vector_index`, running across all dialects and covering `OR REPLACE`, `IF NOT EXISTS`, schema-qualified names, `OPTIONS(...)`, the `INCLUDE` trailer, and an expression target. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Extend the shared `CREATE VECTOR INDEX` parsing to the trailers used by the other dialects, and add per-dialect tests: - `STORING(...)` covering-column clause (BigQuery); new `storing` field on `CreateIndex`, rendered after `INCLUDE`. - Accept a `WITH (...)` options clause on a vector index in every dialect (SQL Server `WITH (METRIC = ..., TYPE = ..., MAXDOP = ...)`), not only the dialects that enable it for a plain `CREATE INDEX`. Tests: - `tests/sqlparser_common.rs` — the generic core plus the shared trailers (`INCLUDE`, `STORING`, `WITH`, `USING`, expression targets) across all dialects. - `tests/sqlparser_bigquery.rs` — `OPTIONS(...)` with index_type / distance_type / JSON tuning keys, and `STORING(...)`. - `tests/sqlparser_mssql.rs` — bracket-quoted names with `WITH (...)`. - `tests/sqlparser_mysql.rs` — TiDB's distance-function target with `USING`. - `tests/sqlparser_oracle.rs` — the core, an expression target and an `INCLUDE` list (Oracle's `ORGANIZATION` / `DISTANCE` / `WITH TARGET ACCURACY` / `PARAMETERS` clauses are not yet parsed). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
from
dd1de78 to
e30e003
Compare
August 24, 2026 21:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 more reviewer2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CREATE VECTOR INDEX creates an index for approximate nearest-neighbor search over an embedding column. It is not specific to one dialect — BigQuery, Oracle, SQL Server, MariaDB and TiDB all have it (so the Generic dialect should too). VECTOR is not a keyword, so parse_create previously failed with Expected: an object type after CREATE, found: VECTOR.
The statement shares a common core across dialects and then diverges into dialect-specific trailers. It is parsed permissively for every dialect.
Common core
VECTOR is treated as a modifier on CREATE INDEX (like UNIQUE), reusing the existing CreateIndex node rather than a new statement variant. It is routed through parse_create_index, so it inherits the standard index trailers — USING <method>, INCLUDE (...), WITH (...), expression targets and index options.
Per-dialect forms
The two follow-up rows need dedicated grammar with real keyword-collision hazards — WITH TARGET ACCURACY overloads the same WITH keyword as SQL Server's WITH (...), and MariaDB's bare single-letter M collides with identifiers — so they are left out of this change.
Changes
Tests
All are round-trip (verified_stmt) tests.
Docs: BigQuery · Oracle · SQL Server · MariaDB · TiDB