Repository navigation
Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines #2406
Description
Activity
Will this feature be picked up ？
Will this feature be picked up ？
Yes, but this requires a lot of changes in the Feast architecture thus I have decided to move it to separate repository and create Feast extension: https://github.com/qooba/yummy
Will this feature be picked up ？
Yes, but this requires a lot of changes in the Feast architecture thus I have decided to move it to separate repository and create Feast extension: https://github.com/qooba/yummy
I had a quick look and it seems like the IcebergDataSource is still missing. Are you planning to add it or has anything changed? Do you need help?
Will this feature be picked up ？
Yes, but this requires a lot of changes in the Feast architecture thus I have decided to move it to separate repository and create Feast extension: https://github.com/qooba/yummy
I had a quick link and it seems like the IcebergDataSource is still missing. Are you planning to add it or has anything changed? Do you need help?
I'll add Iceberg support soon :)
Reacted by Leonard AukeaWill this feature be picked up ？
Yes, but this requires a lot of changes in the Feast architecture thus I have decided to move it to separate repository and create Feast extension: https://github.com/qooba/yummy
I had a quick link and it seems like the IcebergDataSource is still missing. Are you planning to add it or has anything changed? Do you need help?
@LeonardAukea - I have finished feast integration with Iceberg here you have video introduction: https://www.youtube.com/watch?v=kv0iWuSf4jw
Reacted by Leonard Aukea and xuren-sunventureJust came across this. Hudi now offers a Python API via hudi-rs. More at https://hudi.apache.org/docs/python-rust-quick-start-guide/
H @qooba would you be open to adding this into Feast by any chance?
CC @HaoXuAI
Reacted by jbauer12I will look into this. One thing is to make the open table format time travel consistent with Feast time travel utility.
Is your feature request related to a problem? Please describe.
Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which was requested by the community) can be added. This feature will be very helpful for the teams which base on Spark but also for integrations with Flink, Spark Streaming eg. when historical features are saved in the data lake formats.
Describe the solution you'd like
The solution assumes adding new data lake sources:
and also support for CSV files.
The data lake sources support will be covered by the Spark engine (which is already in contrib) for users which use Feast on Spark (or Databricks) but also for Dask and Ray.
Additional assumptions:
Delta Lake
The support for Delta Lake for Feast on Spark is already proposed and tested on Databricks and local spark (https://github.com/qooba/feast-pyspark). The solution is based on the plain pyspark rather than on Spark SQL and Jinja thus it is to decide which implementation will be more desirable and maintainable.
The support for Delta Lake for Feast on Dask (Ray) can be implemented using delta python interface:
Apache Iceberg
The Apache Iceberg is covered by the Spark engine but also by the python api which can be used to add Dask/Ray implementation.
Apache Hudi
The Apache Hudi is covered by the Spark. Currently there is no python api (as far as I know).
CSV
The support for csv files will be dedicated for the data scientists which would like to conduct ad-hoc experiments.
Describe alternatives you've considered
N/A
Additional context
N/A