feast-dev
@@ -64,6 +64,7 @@
   * [Adding a new online store](how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md)
   * [Adding a custom provider](how-to-guides/customizing-feast/creating-a-custom-provider.md)
 * [Adding or reusing tests](how-to-guides/adding-or-reusing-tests.md)
+* [Starting Feast servers in TLS(SSL) Mode](how-to-guides/starting-feast-servers-tls-mode.md)
 
 ## Reference
 

@@ -1,7 +1,9 @@
 # Starting feast servers in TLS (SSL) mode.
 TLS (Transport Layer Security) and SSL (Secure Sockets Layer) are both protocols encrypts communications between a client and server to provide enhanced security.TLS or SSL words used interchangeably. 
 This article is going to show the sample code to start all the feast servers such as online server, offline server, registry server and UI server in TLS mode. 
-Also show examples related to feast clients to communicate with the feast servers started in TLS mode.
+Also show examples related to feast clients to communicate with the feast servers started in TLS mode. 
+
+We assume you have basic understanding of feast terminology before going through this tutorial, if you are new to feast then we would recommend to go through existing [starter tutorials](./../../examples) of feast.
 
 ## Obtaining a self-signed TLS certificate and key
 In development mode we can generate a self-signed certificate for testing. In an actual production environment it is always recommended to get it from a trusted TLS certificate provider.
@@ -17,15 +19,32 @@ The above command will generate two files
 You can use the public or private keys generated from above command in the rest of the sections in this tutorial.
 
 ## Create the feast demo repo for the rest of the sections.
-create a feast repo using `feast init` command and use this repo as a demo for subsequent sections.
+Create a feast repo and initialize using `feast init` and `feast apply` command and use this repo as a demo for subsequent sections.
 
 ```shell
 feast init feast_repo_ssl_demo
-```
 
-Output is
-```
+#output will be something similar as below
 Creating a new Feast repository in /Documents/Src/feast/feast_repo_ssl_demo.
+
+cd feast_repo_ssl_demo/feature_repo
+feast apply
+
+#output will be something similar as below
+Applying changes for project feast_repo_ssl_demo
+
+Created project feast_repo_ssl_demo
+Created entity driver
+Created feature view driver_hourly_stats
+Created feature view driver_hourly_stats_fresh
+Created on demand feature view transformed_conv_rate
+Created on demand feature view transformed_conv_rate_fresh
+Created feature service driver_activity_v1
+Created feature service driver_activity_v3
+Created feature service driver_activity_v2
+
+Created sqlite table feast_repo_ssl_demo_driver_hourly_stats_fresh
+Created sqlite table feast_repo_ssl_demo_driver_hourly_stats
 ```
 
 You need to execute the feast cli commands from  `feast_repo_ssl_demo/feature_repo` directory created from the above `feast init` command.
@@ -68,7 +87,7 @@ entity_key_serialization_version: 2
 auth:
   type: no_auth
 ```
-{% endcode %}
+
 
 `cert` is an optional configuration to the public certificate path when the online server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`.
 
@@ -106,14 +125,55 @@ entity_key_serialization_version: 2
 auth:
   type: no_auth
 ```
-{% endcode %}
 
 `cert` is an optional configuration to the public certificate path when the registry server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`.
 
 ## Starting feast offline server in TLS mode
 
-TBD
+To start the offline server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast serve_offline` command.
+
+```shell
+feast serve_offline --key /path/to/key.pem --cert /path/to/cert.pem
+```
+You will see the output something similar to as below. Note the server url starts in the `https` mode.
+
+```shell
+11/07/2024 11:10:01 AM feast.offline_server INFO: Found SSL certificates in the args so going to start offline server in TLS(SSL) mode.
+11/07/2024 11:10:01 AM feast.offline_server INFO: Offline store server serving at: grpc+tls://127.0.0.1:8815
+11/07/2024 11:10:01 AM feast.offline_server INFO: offline server starting with pid: [11606]
+```
+
+### Feast client connecting to remote offline sever started in TLS mode.
+
+Sometimes you may need to pass the self-signed public key to connect to the remote registry server started in SSL mode if you have not added the public key to the certificate store.
+You have to add `scheme` to `https`.
+
+feast client example:
 
+```yaml
+project: feast-project
+registry:
+  registry_type: remote
+  path: https://localhost:6570
+  cert: /path/to/cert.pem
+provider: local
+online_store:
+  path: http://localhost:6566
+  type: remote
+  cert: /path/to/cert.pem
+entity_key_serialization_version: 2
+offline_store:
+  type: remote
+  host: localhost
+  port: 8815
+  scheme: https
+  cert: /path/to/cert.pem
+auth:
+  type: no_auth
+```
+
+`cert` is an optional configuration to the public certificate path when the registry server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`.
+`scheme` should be `https`. By default, it will be `http` so you have to explicitly configure to `https` if you are planning to connect to remote offline server which is started in TLS mode.
 
 ## Starting feast UI server (react app) in TLS mode
 To start the feast UI server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast ui` command.

@@ -1114,16 +1114,50 @@ def serve_registry_command(
     default=DEFAULT_OFFLINE_SERVER_PORT,
     help="Specify a port for the server",
 )
+@click.option(
+    "--key",
+    "-k",
+    "tls_key_path",
+    type=click.STRING,
+    default="",
+    show_default=False,
+    help="path to TLS certificate private key. You need to pass --cert as well to start server in TLS mode",
+)
+@click.option(
+    "--cert",
+    "-c",
+    "tls_cert_path",
+    type=click.STRING,
+    default="",
+    show_default=False,
+    help="path to TLS certificate public key. You need to pass --key as well to start server in TLS mode",
+)
+@click.option(
+    "--verify_client",
+    "-v",
+    "tls_verify_client",
+    type=click.BOOL,
+    default="True",
+    show_default=True,
+    help="Verify the client or not for the TLS client certificate.",
+)
 @click.pass_context
 def serve_offline_command(
     ctx: click.Context,
     host: str,
     port: int,
+    tls_key_path: str,
+    tls_cert_path: str,
+    tls_verify_client: bool,
 ):
     """Start a remote server locally on a given host, port."""
+    if (tls_key_path and not tls_cert_path) or (not tls_key_path and tls_cert_path):
+        raise click.BadParameter(
+            "Please pass --cert and --key args to start the offline server in TLS mode."
+        )
     store = create_feature_store(ctx)
 
-    store.serve_offline(host, port)
+    store.serve_offline(host, port, tls_key_path, tls_cert_path, tls_verify_client)
 
 
 @cli.command("validate")

@@ -1963,11 +1963,20 @@ def serve_registry(
             self, port=port, tls_key_path=tls_key_path, tls_cert_path=tls_cert_path
         )
 
-    def serve_offline(self, host: str, port: int) -> None:
+    def serve_offline(
+        self,
+        host: str,
+        port: int,
+        tls_key_path: str = "",
+        tls_cert_path: str = "",
+        tls_verify_client: bool = True,
+    ) -> None:
         """Start offline server locally on a given port."""
         from feast import offline_server
 
-        offline_server.start_server(self, host, port)
+        offline_server.start_server(
+            self, host, port, tls_key_path, tls_cert_path, tls_verify_client
+        )
 
     def serve_transformations(self, port: int) -> None:
         """Start the feature transformation server locally on a given port."""

@@ -70,22 +70,45 @@ def list_actions(self, options: FlightCallOptions = None):
         return super().list_actions(options)
 
 
-def build_arrow_flight_client(host: str, port, auth_config: AuthConfig):
+def build_arrow_flight_client(
+    scheme: str, host: str, port, auth_config: AuthConfig, cert: str = ""
+):
+    arrow_scheme = "grpc+tcp"
+    if cert:
+        logger.info(
+            "Scheme is https so going to connect offline server in SSL(TLS) mode."
+        )
+        arrow_scheme = "grpc+tls"
+
+    kwargs = {}
+    if cert:
+        with open(cert, "rb") as root_certs:
+            kwargs["tls_root_certs"] = root_certs.read()
+
     if auth_config.type != AuthType.NONE.value:
         middlewares = [FlightAuthInterceptorFactory(auth_config)]
-        return FeastFlightClient(f"grpc://{host}:{port}", middleware=middlewares)
+        return FeastFlightClient(
+            f"{arrow_scheme}://{host}:{port}", middleware=middlewares, **kwargs
+        )
 
-    return FeastFlightClient(f"grpc://{host}:{port}")
+    return FeastFlightClient(f"{arrow_scheme}://{host}:{port}", **kwargs)
 
 
 class RemoteOfflineStoreConfig(FeastConfigBaseModel):
     type: Literal["remote"] = "remote"
+
+    scheme: Literal["http", "https"] = "http"
+
     host: StrictStr
     """ str: remote offline store server port, e.g. the host URL for offline store  of arrow flight server. """
 
     port: Optional[StrictInt] = None
     """ str: remote offline store server port."""
 
+    cert: StrictStr = ""
+    """ str: Path to the public certificate when the offline server starts in TLS(SSL) mode. This may be needed if the offline server started with a self-signed certificate, typically this file ends with `*.crt`, `*.cer`, or `*.pem`.
+    If type is 'remote', then this configuration is needed to connect to remote offline server in TLS mode. """
+
 
 class RemoteRetrievalJob(RetrievalJob):
     def __init__(
@@ -178,7 +201,11 @@ def get_historical_features(
         assert isinstance(config.offline_store, RemoteOfflineStoreConfig)
 
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            scheme=config.offline_store.scheme,
+            host=config.offline_store.host,
+            port=config.offline_store.port,
+            auth_config=config.auth_config,
+            cert=config.offline_store.cert,
         )
 
         feature_view_names = [fv.name for fv in feature_views]
@@ -214,7 +241,11 @@ def pull_all_from_table_or_query(
 
         # Initialize the client connection
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            scheme=config.offline_store.scheme,
+            host=config.offline_store.host,
+            port=config.offline_store.port,
+            auth_config=config.auth_config,
+            cert=config.offline_store.cert,
         )
 
         api_parameters = {
@@ -247,7 +278,11 @@ def pull_latest_from_table_or_query(
 
         # Initialize the client connection
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            config.offline_store.scheme,
+            config.offline_store.host,
+            config.offline_store.port,
+            config.auth_config,
+            cert=config.offline_store.cert,
         )
 
         api_parameters = {
@@ -282,7 +317,11 @@ def write_logged_features(
 
         # Initialize the client connection
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            config.offline_store.scheme,
+            config.offline_store.host,
+            config.offline_store.port,
+            config.auth_config,
+            config.offline_store.cert,
         )
 
         api_parameters = {
@@ -308,7 +347,11 @@ def offline_write_batch(
 
         # Initialize the client connection
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            config.offline_store.scheme,
+            config.offline_store.host,
+            config.offline_store.port,
+            config.auth_config,
+            config.offline_store.cert,
         )
 
         feature_view_names = [feature_view.name]
@@ -336,7 +379,11 @@ def validate_data_source(
         assert isinstance(config.offline_store, RemoteOfflineStoreConfig)
 
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            config.offline_store.scheme,
+            config.offline_store.host,
+            config.offline_store.port,
+            config.auth_config,
+            config.offline_store.cert,
         )
 
         api_parameters = {
@@ -357,7 +404,11 @@ def get_table_column_names_and_types_from_data_source(
         assert isinstance(config.offline_store, RemoteOfflineStoreConfig)
 
         client = build_arrow_flight_client(
-            config.offline_store.host, config.offline_store.port, config.auth_config
+            config.offline_store.scheme,
+            config.offline_store.host,
+            config.offline_store.port,
+            config.auth_config,
+            config.offline_store.cert,
         )
 
         api_parameters = {