FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: Add MLflow integration support to Feast operator by Vperiodt · Pull Request #6611 · feast-dev/feast · GitHub

16 changes: 8 additions & 8 deletions .secrets.baseline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -957,7 +957,7 @@
"filename": "infra/feast-operator/api/v1/featurestore_types.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 970
"line_number": 1029
}
],
"infra/feast-operator/api/v1/zz_generated.deepcopy.go": [
Expand All @@ -966,21 +966,21 @@
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "f914fc9324de1bec1ad13dec94a8ea2ddb41fc87",
"is_verified": false,
"line_number": 842
"line_number": 909
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 903
"line_number": 970
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "c2028031c154bbe86fd69bef740855c74b927dcf",
"is_verified": false,
"line_number": 1605
"line_number": 1672
}
],
"infra/feast-operator/api/v1alpha1/featurestore_types.go": [
Expand Down Expand Up @@ -1165,14 +1165,14 @@
"filename": "infra/feast-operator/internal/controller/services/repo_config.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 133
"line_number": 137
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/internal/controller/services/repo_config.go",
"hashed_secret": "e2fb052132fd6a07a56af2013e0b62a1f510572c",
"is_verified": false,
"line_number": 241
"line_number": 245
}
],
"infra/feast-operator/internal/controller/services/services.go": [
Expand All @@ -1181,7 +1181,7 @@
"filename": "infra/feast-operator/internal/controller/services/services.go",
"hashed_secret": "36dc326eb15c7bdd8d91a6b87905bcea20b637d1",
"is_verified": false,
"line_number": 184
"line_number": 233
}
],
"infra/feast-operator/internal/controller/services/tls_test.go": [
Expand Down Expand Up @@ -1564,5 +1564,5 @@
}
]
},
"generated_at": "2026-08-18T09:56:16Z"
"generated_at": "2026-08-18T12:17:21Z"
}
1 change: 1 addition & 0 deletions docs/SUMMARY.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
* [5 — Security](how-to-guides/feast-operator/05-security.md)
* [6 — Batch & Jobs](how-to-guides/feast-operator/06-batch-and-jobs.md)
* [7 — OpenLineage & Materialization](how-to-guides/feast-operator/07-openlineage-and-materialization.md)
* [8 — MLflow Integration](how-to-guides/feast-operator/08-mlflow-integration.md)
* [Feast Production Deployment Topologies](how-to-guides/production-deployment-topologies.md)
* [Online Server Performance Tuning](how-to-guides/online-server-performance-tuning.md)
* [Customizing Feast](how-to-guides/customizing-feast/README.md)
Expand Down
193 changes: 193 additions & 0 deletions docs/how-to-guides/feast-operator/08-mlflow-integration.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# Guide 8 — MLflow Integration

The operator auto-discovers MLflow on RHOAI/ODH clusters and enables experiment tracking
for every FeatureStore deployment. When the MLflow operator is present and healthy, Feast
pods receive MLflow configuration automatically — no manual YAML editing required.

---

## Auto-discovery

The operator lists all `MLflow` CRs (`mlflow.opendatahub.io/v1`) in the cluster and uses
the first one with an `Available=True` or `Ready=True` condition. When found, it populates
`tracking_uri` from `status.address.url` and `ui_url` from `status.url`.

If the MLflow CR does not report conditions (older operator versions), auto-discovery will
not activate. Set `trackingUri` explicitly in that case.

> **No MLflow?** The FeatureStore stays Ready. Non-MLflow FeatureViews and all other Feast
> services are completely unaffected.

---

## FeatureStore CR configuration

### Auto-enabled (default when MLflow is present)

No `spec.mlflow` needed. The operator auto-enables when an Available MLflow CR is detected:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: my-store
spec:
feastProject: my_project
services:
onlineStore: {}
registry: {}
ui: {}
```

### Explicit configuration

Override defaults or enable additional features:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: my-store
spec:
feastProject: my_project
services:
onlineStore: {}
registry: {}
ui: {}
mlflow:
enabled: true
trackingUri: "https://custom-mlflow.example.com:8443"
uiUrl: "https://dashboard.example.com/mlflow"
trackingAuth: "kubernetes-namespaced"
autoLog: true
autoLogEntityDf: true
entityDfMaxRows: 50000
logOperations: true
opsExperimentSuffix: "-feast-ops"
```

### Opt-out

Disable MLflow even when the MLflow operator is present:

```yaml
spec:
mlflow:
enabled: false
```

---

## Field reference

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | auto-detected | Master switch for MLflow integration |
| `trackingUri` | string | auto-discovered | MLflow tracking server URI (in-cluster, from `status.address.url`) |
| `uiUrl` | string | auto-discovered | Browser-reachable MLflow URL for Feast UI lineage links (from `status.url`) |
| `trackingAuth` | *string | `"kubernetes-namespaced"` | Auth method for Feast pods calling MLflow |
| `autoLog` | *bool | `true` | Auto-log feature metadata on every retrieval |
| `autoLogEntityDf` | *bool | `false` | Save entity DataFrame as artifact |
| `entityDfMaxRows` | *int32 | `100000` | Skip artifact for large DataFrames |
| `logOperations` | *bool | `false` | Log `feast apply` / `materialize` to ops experiment |
| `opsExperimentSuffix` | *string | `"-feast-ops"` | Ops experiment name suffix |
| `extraConfig` | map[string]string | — | Additional YAML fields (coerced to native types) |

---

## Authentication

The operator injects `MLFLOW_TRACKING_AUTH` into all Feast pod containers. The MLflow
Python client's auth plugin system uses this env var to attach credentials to tracking
server requests.

| `trackingAuth` value | Behavior |
|---------------------|----------|
| `"kubernetes-namespaced"` (default) | SA token + `X-MLFLOW-WORKSPACE: <namespace>` header. Multi-tenant on RHOAI. |
| `"kubernetes"` | SA token only. Single-tenant setups. |
| `"basic"` | HTTP Basic auth via `MLFLOW_TRACKING_USERNAME` / `MLFLOW_TRACKING_PASSWORD` env vars. |
| `"bearer"` | Static bearer token from `MLFLOW_TRACKING_TOKEN` env var. |
| `""` (empty string) | No auth header. Local dev or unprotected MLflow. |

No Kubernetes RoleBinding is needed for MLflow tracking API access. The MLflow server
validates the SA token directly via TokenReview.

---

## Tracking URI resolution order

1. Explicit `trackingUri` in the FeatureStore CR
2. Auto-discovered from MLflow CR `status.address.url` (first Available/Ready CR)
3. `MLFLOW_TRACKING_URI` environment variable (on workbench pods, injected by the MLflow operator)
4. MLflow default (`./mlruns`)

---

## UI URL resolution order

Used for browser hyperlinks in Feast UI lineage panels:

1. Explicit `uiUrl` in the FeatureStore CR
2. `MLFLOW_UI_URL` environment variable
3. Auto-discovered from MLflow CR `status.url` (external gateway route)
4. Falls back to `trackingUri` (works for local dev)

---

## Graceful degradation

| Scenario | Behavior |
|----------|----------|
| MLflow operator not installed | No `mlflow` block in YAML; FeatureStore stays Ready |
| MLflow CR exists but not Ready | Discovery returns empty; MLflow stays off |
| Tracking URI becomes unreachable | SDK logs a warning; feature retrieval is not blocked |
| `spec.mlflow.enabled: false` | MLflow integration explicitly disabled |

---

## Workbench usage

In a RHOAI workbench notebook connected to the FeatureStore:

```python
from feast import FeatureStore

store = FeatureStore(...) # from mounted client config

with store.mlflow.start_run(run_name="training"):
df = store.get_historical_features(
entity_df=entity_df,
features=["driver_stats:conv_rate", "driver_stats:acc_rate"],
).to_df()
model = train(df)
store.mlflow.log_model(model, "model")
```

> **Dependency**: the Feast image must include `feast[mlflow]` (`mlflow` or `mlflow-skinny`).

---

## RBAC permissions

The operator needs `get`, `list`, `watch` on `mlflows` in the `mlflow.opendatahub.io` API
group. This is included in the default operator ClusterRole.

```yaml
- apiGroups:
- mlflow.opendatahub.io
resources:
- mlflows
verbs:
- get
- list
- watch
```

---

## See also

- [API field reference — `MlflowConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md)
- [MLflow DataSource reference](../../reference/mlflow.md)
- [Guide 5 — Security](05-security.md) (RBAC / OIDC auth)
- [Guide 7 — OpenLineage & Materialization](07-openlineage-and-materialization.md)
2 changes: 2 additions & 0 deletions docs/how-to-guides/feast-operator/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ look for store-specific YAML options in the Feast SDK docs.
| 5 | [Security](05-security.md) | Kubernetes RBAC roles vs OIDC auth; TLS for all servers |
| 6 | [Batch Jobs](06-batch-and-jobs.md) | `batchEngine` ConfigMap contract, `cronJob` for scheduled materialization |
| 7 | [OpenLineage & Materialization](07-openlineage-and-materialization.md) | Lineage transports, API key Secret, materialization batch size |
| 8 | [MLflow Integration](08-mlflow-integration.md) | Auto-discovery, experiment tracking, auth, Feast UI lineage |

---

Expand All @@ -42,6 +43,7 @@ look for store-specific YAML options in the Feast SDK docs.
- **"How do I enable Kubernetes RBAC or OIDC auth?"** → [Guide 5](05-security.md)
- **"How do I schedule nightly materialization?"** → [Guide 6](06-batch-and-jobs.md)
- **"How do I send lineage events to Marquez?"** → [Guide 7](07-openlineage-and-materialization.md)
- **"How do I connect Feast to MLflow for experiment tracking?"** → [Guide 8](08-mlflow-integration.md)
- **"What are all valid fields on `ServingConfig`?"** → [API ref](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#servingconfig)

---
Expand Down
59 changes: 59 additions & 0 deletions infra/feast-operator/api/v1/featurestore_types.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,59 @@ type OpenLineageConsumerConfig struct {
NamespaceMapping map[string]string `json:"namespaceMapping,omitempty"`
}

// MlflowConfig enables MLflow experiment tracking integration for Feast.
// When enabled, feature retrieval metadata is automatically logged to MLflow runs
// and the Feast UI displays lineage from feature views to registered models.
// +kubebuilder:validation:XValidation:rule="!has(self.extraConfig) || !('enabled' in self.extraConfig) && !('tracking_uri' in self.extraConfig) && !('ui_url' in self.extraConfig) && !('tracking_auth' in self.extraConfig) && !('auto_log' in self.extraConfig) && !('auto_log_entity_df' in self.extraConfig) && !('entity_df_max_rows' in self.extraConfig) && !('log_operations' in self.extraConfig) && !('ops_experiment_suffix' in self.extraConfig)",message="extraConfig must not contain keys that duplicate typed fields (enabled, tracking_uri, ui_url, tracking_auth, auto_log, auto_log_entity_df, entity_df_max_rows, log_operations, ops_experiment_suffix); use the corresponding spec fields instead."
type MlflowConfig struct {
// Enable MLflow integration.
Enabled bool `json:"enabled"`
// MLflow tracking server URI. When omitted, the operator auto-discovers
// from the cluster MLflow CR (status.address.url). Falls back to
// MLFLOW_TRACKING_URI env var on pods.
// +optional
TrackingUri *string `json:"trackingUri,omitempty"`
// Browser-reachable MLflow UI URL used for hyperlinks in Feast UI lineage.
// When omitted, the operator auto-discovers from the MLflow CR status.url
// (the external gateway route). Falls back to MLFLOW_UI_URL env var, then
// to trackingUri. Only needed when the tracking URI is cluster-internal.
// +optional
UiUrl *string `json:"uiUrl,omitempty"`
// Automatically log feature metadata on every retrieval inside an active MLflow run.
// Defaults to true when enabled.
// +optional
AutoLog *bool `json:"autoLog,omitempty"`
// Save entity DataFrame as MLflow artifact on historical retrieval.
// Defaults to false.
// +optional
AutoLogEntityDf *bool `json:"autoLogEntityDf,omitempty"`
// Maximum number of entity DataFrame rows to save as an MLflow artifact.
// DataFrames exceeding this limit are skipped. Defaults to 100000.
// +kubebuilder:validation:Minimum=1
// +optional
EntityDfMaxRows *int32 `json:"entityDfMaxRows,omitempty"`
// Log feast apply and materialize operations to a separate MLflow experiment.
// Defaults to false.
// +optional
LogOperations *bool `json:"logOperations,omitempty"`
// Suffix appended to the project name for the operations experiment.
// Defaults to "-feast-ops".
// +optional
OpsExperimentSuffix *string `json:"opsExperimentSuffix,omitempty"`
// Authentication method used by Feast pods when calling the MLflow tracking
// server. Common values: "kubernetes-namespaced" (token-based, default on
// OpenShift AI), "basic", "bearer", or "" (no auth for local/dev).
// Defaults to "kubernetes-namespaced".
// +optional
TrackingAuth *string `json:"trackingAuth,omitempty"`
// ExtraConfig holds additional MLflow key-value settings written inline into
// the mlflow block of feature_store.yaml. Boolean and integer string values
// are coerced to native YAML types. Keys must be valid Feast MlflowConfig
// YAML field names.
// +optional
ExtraConfig map[string]string `json:"extraConfig,omitempty"`
}

// FeatureStoreSpec defines the desired state of FeatureStore
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
Expand Down Expand Up @@ -172,6 +225,12 @@ type FeatureStoreSpec struct {
// Written into feature_store.yaml for all service pods.
// +optional
OpenLineage *OpenLineageConfig `json:"openlineage,omitempty"`
// Mlflow enables MLflow experiment tracking integration for Feast.
// Written into feature_store.yaml for all service pods and the client ConfigMap.
// When omitted and a cluster MLflow instance is detected, defaults to enabled
// with the discovered tracking URI.
// +optional
Mlflow *MlflowConfig `json:"mlflow,omitempty"`
}

// FeastProjectDir defines how to create the feast project directory.
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL