FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: Add HDFS as a feature registry by chimeyrock999 · Pull Request #5655 · feast-dev/feast · GitHub

/ feast Public

feat: Add HDFS as a feature registry - #5655

Merged
ntkathole merged 17 commits into
feast-dev:masterfrom
chimeyrock999:master
Oct 15, 2025
ConversationCommits (17)ChecksFiles changed
Merged

feat: Add HDFS as a feature registry#5655
ntkathole merged 17 commits into
feast-dev:masterfrom
chimeyrock999:master

Conversation

chimeyrock999 commented Oct 10, 2025

Copy link
Copy Markdown
Contributor
  1. Implemented a new feature registry that reads and writes feature data directly on HDFS using pyarrow.fs.HadoopFileSystem.
  2. Added integration tests using two containers (1 Namenode and 1 Datanode) via Testcontainers and bde2020’s Hadoop images.
  3. To use pyarrow.fs.HadoopFileSystem in client or CI environments, you must properly configure the environment by following the PyArrow HDFS documentation.
    .

chimeyrock999 requested a review from a team as a code owner October 10, 2025 23:29
feature(): support hdfs as registry
d1c3270 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
fix(): add type for HDFSRegistryStore
4401b54 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
fix(): reformat code of registry.py file
943b2d3 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
fix(): change hdfs remove api and fix hdfs registry test
b77ea6b 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci: install hadoop dependencies for pyarrow.fs.HDFSFileSystem
ed4fb69 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci: fix install-hadoop-dependencies-ci
751e3d5 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci: typo in install-hadoop-dependencies-ci
abdc263 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
franciscojavierarceo requested changes Oct 11, 2025
View reviewed changes

franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Thank you for this contribution!!

Can we update the registry documentation to include this?

ci: add HADOOP_USER_NAME env var
017e0f5 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
docs: add document for HDFS registry
9ffb856 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
fix: change wait logs of hdfs_registry test to ensure containers are …
c4cd9b5 
…running

Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>

chimeyrock999 commented Oct 11, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Thank you for this contribution!!

Can we update the registry documentation to include this?

Thank @franciscojavierarceo. I’ve fixed the integration test and updated the docs for the HDFS registry.

ntkathole reviewed Oct 12, 2025
View reviewed changes
Comment thread Makefile Outdated
ntkathole reviewed Oct 12, 2025
View reviewed changes
Comment thread docs/reference/registries/hdfs.md
ntkathole reviewed Oct 12, 2025
View reviewed changes
Comment thread docs/reference/registries/hdfs.md
Merge branch 'feast-dev:master' into master
3e0f467
ci: fix typo in install-hadoop-dependencies
2d075d3 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
docs(): Add pre-requisites for hdfs registry
720a9ca 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci(): cache hadoop tarball
d826cd5 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci(): rename hadoop-3.4.2 to hadoop
b6e0018 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ci(): readd install-hadoop-dependencies-ci
9ecb03f 
Signed-off-by: Chimey Rock <trinhvanthoai99@gmail.com>
ntkathole approved these changes Oct 14, 2025
View reviewed changes

ntkathole left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Looks good

ntkathole enabled auto-merge (squash) October 14, 2025 13:43
ntkathole merged commit 4c65872 into feast-dev:master Oct 15, 2025
1 check passed
franciscojavierarceo pushed a commit that referenced this pull request Oct 27, 2025
chore(release): release 0.56.0
90b5086 
# [0.56.0](v0.55.0...v0.56.0) (2025-10-27)

### Bug Fixes

* Add mode field to Transformation proto for proper serialization ([2390d2e](2390d2e))
* Date wise remote offline store historical data retrieval  ([#5686](#5686)) ([949ba3d](949ba3d))
* Fix STRING type handling in on-demand feature views ([#5669](#5669)) ([dfbb743](dfbb743))
* Fixed torch install issue in CI ([366e5a8](366e5a8))
* ODFV not getting counted in resource count ([1d640b6](1d640b6))
* Skip tag updates if user do not have permissions ([#5673](#5673)) ([0a951ce](0a951ce))

### Features

* Add document of Go feature server. ([#5697](#5697)) ([cbd1dde](cbd1dde))
* Add flexible commandArgs support for complete Feast CLI control ([#5678](#5678)) ([6414924](6414924))
* Add HDFS as a feature registry ([#5655](#5655)) ([4c65872](4c65872))
* Add nodeSelector to service config ([#5675](#5675)) ([9728cde](9728cde))
* Add OTEL based observability to the Go Feature Server ([#5685](#5685)) ([f4afdad](f4afdad))
* Added health endpoint for the UI ([#5665](#5665)) ([3aec5d5](3aec5d5))
* Added kuberay support ([e0b698d](e0b698d))
* Added support for filtering multi-projects ([#5688](#5688)) ([eb0a86e](eb0a86e))
* Batch Embedding at scale for RAG with Ray ([cc2a46d](cc2a46d))
* Optimize SQL entity handling without creating temporary tables ([#5695](#5695)) ([aa2c838](aa2c838))
* Support aggregation in odfv ([#5666](#5666)) ([564e965](564e965))
* Support cache_mode for registries ([021e9ea](021e9ea))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

ntkathole ntkathole approved these changes
franciscojavierarceo Awaiting requested review from franciscojavierarceo

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL