Repository navigation
make vector search configs part of feature definition #5652
Description
Activity
The Milvus support four different patterns of multi-tenancy: https://milvus.io/docs/multi_tenancy.md
Maybe we can suppor the following two:
1, database-level
2, collection-levelSuggesting this issue's scope should cover the index build parameters, not only the search configs, and that they should be typed rather than a dict. Two pieces of evidence from working with vector indexes recently.
1. metric is half-migrated; index_type and the build params are not
The Milvus store already honours a per-field metric but takes everything else from global config — milvus.py:409-410:
metric_type=metric or config.online_store.metric_type, # Field can override index_type=config.online_store.index_type, # global only
So the split today is:
parameter source vector_search_metric Field, falling back to global index_type global feature_store.yaml only nlist, M, efConstruction, … global only, inside an untyped dict That last row is milvus.py:227-229:
# Index build params for vector fields, e.g. {"M": 16, "efConstruction": 200}. index_params: Optional[Dict[str, Any]] = None
Which means the multi-tenant problem this issue opens with applies to build parameters at least as strongly as to embedding_dim: two feature views needing different M or nlist cannot coexist, and Field has nowhere to put them — it carries only vector_index, vector_length and vector_search_metric (field.py:48-50).
2. An untyped dict makes a whole class of error invisible
Worth arguing for typed fields rather than extending Dict[str, Any], because unknown keys are silently dropped by the engines underneath.
Measured against pylance 12.0.0 — create_index takes HNSW parameters through **kwargs and forwards them to Rust, where the names are m / ef_construction. The Java/Scala binding for the same library spells them M / efConstruction. Passing the Java spellings:
m=16, ef_construction=200 -> index_stats: m=16 ef_construction=200 correct M=16, efConstruction=200 -> index_stats: m=20 ef_construction=150 silently defaulted totally_bogus_param=123 -> index_stats: m=20 ef_construction=150 no error raised
The build succeeds and reports success while producing a default-parameter index. The only way to tell is to read index_stats back afterwards. Lance's defaults here happen not to match the values that were asked for, so this is not a theoretical difference.
A Dict[str, Any] on the config object cannot catch that; a typed field on Field can, at feast apply time.
3. Why this matters beyond multi-tenancy
Reviewing an unrelated production index pipeline recently, its HNSW build parameters were encoded in the name of the build step — the tuple efConstruction=2000, ef=500, M=20 appeared only as a substring of a job name, with nothing in any schema or registry. Nothing recorded which parameters produced which artifact, and recall was never measured, so a parameter change was indistinguishable from a data change after the fact.
That is the failure mode worth designing against: if build parameters live anywhere other than the versioned feature definition, there is no object to attach a measurement to. "This version scored recall@10 = 0.95" is only meaningful if "this version" also pins the parameters that produced it.
So concretely, in addition to what this issue already proposes:
- typed build-parameter fields on Field (or a small typed object it holds), covering at least index_type and the per-type build parameters
- Field carrying them means they are in the registry and move with the feature view version
- engines read them from the feature definition, with global config as fallback for compatibility
Happy to take the Field-side change if the direction looks right. The engine-side plumbing is larger and probably wants splitting per store.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Hello everyone,
I have a question about the recommended pattern for using Milvus as an online store in a multi-team environment.
The feature_store.yaml configuration for Milvus defines global configs, like:
embedding_dim
index_type
metric_type
Our use case involves multiple teams (e.g., marketing, sales) sharing the feature store, but their vector embeddings have different requirements. For example:
The Marketing team might use a 768-dimension vector with COSINE similarity.
The Sales team might use a 128-dimension vector with L2 similarity.
Since these configurations seem to be set at the online_store level, it implies we can only support one type of vector across the entire store. This would prevent us from sharing one feature server across teams with different vector models.
What is the best way to support multiple, diverse vector feature views (with different dimensions, metrics, etc.) in a single Milvus online store?
Is the intended pattern to create a separate online_store definition for each unique vector configuration, or am I misunderstanding how these parameters should be used?
Thanks for any insights!
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.