FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat(v2): add from and to pandas df for documentarray by anna-charlotte · Pull Request #1161 · docarray/docarray · GitHub

/ docarray Public

feat(v2): add from and to pandas df for documentarray - #1161

Merged
anna-charlotte merged 33 commits into
feat-rewrite-v2from
feat-from-to-pandas
Feb 23, 2023
ConversationCommits (33)ChecksFiles changed
Merged

feat(v2): add from and to pandas df for documentarray#1161
anna-charlotte merged 33 commits into
feat-rewrite-v2from
feat-from-to-pandas

Conversation

anna-charlotte commented Feb 22, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Add from and to pandas to DocumentArray IOMixin

  • add DocumentArray[Doc].from_pandas(df)
  • add da.to_pandas()
  • check and update documentation, if required. See guide

anna-charlotte and others added 21 commits February 20, 2023 15:06
feat: load from and to csv
dce8c6b 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: from to csv
37892a9 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
feat: add access path to dict
e1e90cf 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: from to csv
7bd7f1f 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: clean up
05e19ba 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
docs: add docstring and update tmpdir in test
5b0c760 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: merge nested dicts
5babdae 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: clean up
d6b29c5 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: clean up
9149e57 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
test: update test
3d980b7 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: apply samis suggestion from code review
516ffbb 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Merge branch 'feat-rewrite-v2' into feat-from-to-csv
5fac964
fix: apply suggestions from code review wrt access paths
a0d9711 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: apply johannes suggestion
c9005b1 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
fix: apply johannes suggestion
9fe58f5 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
fix: apply suggestions from code review
90867bb 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: apply suggestions from code review
1a395da 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: typos
00a9ea7 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
refactor: move helper functions to helper file
e06e533 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
test: fix fixture
c8e4cf8 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
feat: add to and from pandas df for documentarray
6e3a624 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
chore: add pandas to pyproject.toml
12df134 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
anna-charlotte added 3 commits February 22, 2023 11:50
docs: update docstring
504c13c 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: mypy
4bf8976 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: clean up
ca2bf4f 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Merge remote-tracking branch 'origin/feat-rewrite-v2' into feat-from-…
81ea38d 
…to-pandas

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
github-actions Bot added size/m and removed size/l labels Feb 22, 2023
anna-charlotte marked this pull request as ready for review February 22, 2023 13:38
anna-charlotte mentioned this pull request Feb 23, 2023
Closed
47 tasks
samsja reviewed Feb 23, 2023
View reviewed changes
Comment thread docarray/array/array/io.py Outdated
samsja reviewed Feb 23, 2023
View reviewed changes
Comment thread pyproject.toml Outdated
rich = ">=13.1.0"
lz4 = {version= ">=1.0.0", optional = true}
pydub = {version = "^0.25.1", optional = true }
pandas = ">=1.1.0"

samsja Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

it should be an optional dependecy

anna-charlotte Feb 23, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

add it to common or a as a separate extra pandas ?

samsja Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

separate. we will reorginaze at some point anyway

samsja reviewed Feb 23, 2023
View reviewed changes

samsja left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

look goods, I added some comments

JohannesMessner requested changes Feb 23, 2023
View reviewed changes
Comment thread docarray/array/array/io.py Outdated
)

@staticmethod
def access_path_dict_to_nested_dict(

JohannesMessner Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

didn't you already implement something like this in your previous PR? or am i mixing things up?

anna-charlotte Feb 23, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

yes, the function _access_path_to_dict that is used within _access_path_dict_to_nested_dict that I implemented in the last PR.

  • _access_path_to_dict: transforms one access path to nested dict
  • _ access_path_dict_to_nested_dict: transforms dict with (multiple) access path keys to a joint nested dict by calling the former func

I will move the latter to that other helper file, to keep those functions together yes?

JohannesMessner Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

ok sounds good, yes keeping them in the same helper file would be ideal

Comment thread docarray/array/array/io.py Outdated
Comment thread docarray/array/array/io.py Outdated
Comment thread docarray/array/array/io.py Outdated
Comment thread docarray/array/array/io.py Outdated
return json.dumps([doc.json() for doc in self])

@classmethod
def _check_for_valid_document_type(cls) -> None:

JohannesMessner Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I would prefer returning a bool and rasing at the call site. That way it is clearer where the error actually occurs, and this method can be re-used in contexts where only the information is needed, without wanting to raise an exception

Comment thread docarray/array/array/io.py Outdated
)

@classmethod
def _check_for_valid_access_paths(cls, field_names: Optional[List[str]]) -> None:

JohannesMessner Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

same here

Comment thread docarray/array/array/io.py Outdated

@classmethod
def _check_for_valid_document_type(cls) -> None:
if cls.document_type == AnyDocument:

JohannesMessner Feb 23, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

is there are plan to make this check more sophisticated as part of this PR? otherwise this doesn't need to be a method imo

anna-charlotte Feb 23, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

No, just tried to extract all the duplicate code but maybe I went a bit overboard hehe, moved it back to from_csv and from_pandas again

This was linked to issues Feb 23, 2023
v2: handle load from external data sources #971
Closed
Implement from_to/csv,pandas #1126
Closed
anna-charlotte and others added 6 commits February 23, 2023 11:39
fix: apply suggestions from code review
5d02b35 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: apply suggestion from johannes
687d98a 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
fix: apply suggestion from johannes
2c02bf4 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
fix: apply suggestions from code review
006a186 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
fix: apply suggestion
c68a48c 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Merge branch 'feat-rewrite-v2' into feat-from-to-pandas
fb74c4f
samsja reviewed Feb 23, 2023
View reviewed changes
Comment thread docarray/array/array/io.py Outdated
samsja reviewed Feb 23, 2023
View reviewed changes
Comment thread docarray/array/array/io.py
samsja requested changes Feb 23, 2023
View reviewed changes

samsja left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

we should make pandas install optional. Otherwise looks good

fix: apply suggestions
9d90053 
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

github-actions Bot commented Feb 23, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-from-to-pandas--jina-docs.netlify.app 🎉

anna-charlotte merged commit 4de28ea into feat-rewrite-v2 Feb 23, 2023
anna-charlotte deleted the feat-from-to-pandas branch February 23, 2023 12:48
AsRaNi1 pushed a commit to AsRaNi1/docarray that referenced this pull request Feb 28, 2023
feat(v2): add from and to pandas df for documentarray (docarray#1161)
025fb66 
* feat: load from and to csv

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: from to csv

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* feat: add access path to dict

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: from to csv

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: clean up

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* docs: add docstring and update tmpdir in test

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: merge nested dicts

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: clean up

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: clean up

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* test: update test

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply samis suggestion from code review

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestions from code review wrt access paths

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply johannes suggestion

Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>

* fix: apply johannes suggestion

Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>

* fix: apply suggestions from code review

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestions from code review

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: typos

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* refactor: move helper functions to helper file

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* test: fix fixture

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* feat: add to and from pandas df for documentarray

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* chore: add pandas to pyproject.toml

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* docs: update docstring

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: mypy

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: clean up

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestions from code review

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestion from johannes

Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>

* fix: apply suggestion from johannes

Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>

* fix: apply suggestions from code review

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestion

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

* fix: apply suggestions

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

---------

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Arnav Zutshi <arnzut1324@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

JohannesMessner JohannesMessner approved these changes
samsja samsja approved these changes

Assignees

No one assigned

Labels

area/core area/setup area/testing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement from_to/csv,pandas v2: handle load from external data sources

3 participants

Back | FazBrowse Home | New Git URL