FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Generic Input/Output of DataFrames by windoze · Pull Request #475 · feathr-ai/feathr · GitHub

/ feathr Public

Generic Input/Output of DataFrames - #475

Merged
windoze merged 12 commits into
mainfrom
windoze/generic-io
Aug 13, 2022
ConversationCommits (12)ChecksFiles changed
Merged

Generic Input/Output of DataFrames#475
windoze merged 12 commits into
mainfrom
windoze/generic-io

Conversation

windoze commented Jul 15, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

This PR includes:

  1. Extended former InputLocation classes to support both read and write functions, also renamed it to DataLocation to reflect this change.
  2. Added a GenericLocation which supports all Spark confs, modes, and options, so it can be used to operate virtually any connectors supported by Spark
  3. In GenericLocation, I added a format-specific patching mechanism to workaround quirks come with different connectors, e.g. CosmosDb requires rows to have an id column with unique values.
  4. Update FeathrGenJob and FeathrJoinJob, enabling them to use a JSON-encoded string of DataLocation instead of plain path as the input and output target.

Theoretically, Feathr core can support all Spark connectors with this patch, but we still run a series of compatibility tests to confirm the final list.

NOTE: This PR only involves Feathr core, the corresponding Feathr Client changes will be in upcoming PRs.

xiaoyongzhu reviewed Jul 15, 2022
View reviewed changes
Comment thread build.sbt

xiaoyongzhu commented Jul 15, 2022

Copy link
Copy Markdown
Member

This PR looks good to me and is a good way of extending to other sources in the future. Thanks @windoze for the work!

xiaoyongzhu reviewed Jul 15, 2022
View reviewed changes
Comment thread build.sbt
windoze added the safe to test Tag to execute build pipeline for a PR from forked repo label Jul 15, 2022
jaymo001 reviewed Jul 26, 2022
View reviewed changes
Comment thread src/main/scala/com/linkedin/feathr/offline/config/location/DataLocation.scala
Comment thread src/main/scala/com/linkedin/feathr/offline/config/location/GenericLocation.scala
Comment thread src/main/scala/com/linkedin/feathr/offline/config/location/GenericLocation.scala
Comment thread src/main/scala/com/linkedin/feathr/offline/config/location/GenericLocation.scala
Comment thread src/main/scala/com/linkedin/feathr/offline/config/location/GenericLocation.scala
windoze mentioned this pull request Aug 13, 2022
Merged
windoze merged commit 671bae3 into main Aug 13, 2022
xiaoyongzhu deleted the windoze/generic-io branch August 22, 2022 17:15
ahlag pushed a commit to ahlag/feathr that referenced this pull request Aug 26, 2022
Generic Input/Output of DataFrames (feathr-ai#475)
8343ab3 
* GenericLocation for DataFrame read/write

* WIP

* Generate id column

* Fix unit test

* Parse string into DataLocation

* Id column must be string

* Fix auth logic

* Fix unit test

* Fix id column generation

* CosmosDb Sink
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

xiaoyongzhu xiaoyongzhu left review comments
jaymo001 jaymo001 left review comments
blrchen blrchen approved these changes
Yuqing-cat Yuqing-cat approved these changes
blee1234 Awaiting requested review from blee1234
d4ve Awaiting requested review from d4ve
hangfei Awaiting requested review from hangfei

Assignees

No one assigned

Labels

safe to test Tag to execute build pipeline for a PR from forked repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

Back | FazBrowse Home | New Git URL