The SQLite online store answers retrieve_online_documents and
retrieve_online_documents_v2 by copying the feature view's rows into a
vec0 (vector) or FTS5 (keyword) table and querying that table. The table
lived in the online store database and was never cleared, so every
search added another copy of every row:
- Vector search only worked once. Later searches failed with "UNIQUE
constraint failed on vec_table primary key" (v2) or "table vec_table
already exists" (v1), and once a write committed the leftover rows,
even the first search of a new process failed.
- Keyword search returned fewer documents on every call, as copies of
the best matches filled top_k, and kept matching text that had since
been overwritten.
The search also left its transaction open, so writes from other
processes failed with "database is locked".
Build the index in the connection's temp schema, drop and refill it for
every search, and commit it right away.
Signed-off-by: Mohammad Hijjawi <mohammad.hijjawi1997@gmail.com>
What this PR does / why we need it:
The SQLite online store answers retrieve_online_documents and retrieve_online_documents_v2 (and so the feature server's /search) by copying the feature view's rows into a vec0 (vector) or FTS5 (keyword) table and querying that. The table lives in the online store database and is never cleared, so every search adds another copy of every row:
Repro on master, same data and query each time:
This PR builds the index in the connection's temp schema, drops and refills it for every search, and commits it right away. Every call now returns the same top_k from the current rows, a rewritten document no longer matches its old text, and nothing is left in the online store file or locked. The vector index build is shared by both APIs.
Which issue(s) this PR fixes:
No existing issue.
Checks
Testing Strategy
Misc
Added two tests to sdk/python/tests/unit/online_store/test_online_retrieval.py; both fail on master and pass with this change:
All sqlite tests in that file pass on Python 3.10 with SQLite 3.53 (vec0 KNN queries need SQLite >= 3.41 for LIMIT, as on ubuntu-latest), as does test_sqlite_get_online_documents when run directly. I did not run the Milvus tests (milvus-lite has no Windows build). ruff and mypy on sqlite.py are clean.