feat: add map function by anna-charlotte · Pull Request #1187 · docarray/docarray · GitHub

Goals:

Add map_docs function and map_batch:

docs = list(map_docs(da, lambda x : x, num_worker=4 ))

This will be different from doing

for doc in da:
    doc = f(doc)

leverage multiprocessing by benchmarking in test, check that using 2 CPUS is faster then using 1
map_docs()
map_docs_batch()
benchmarking tests
check and update documentation, if required. See guide

Is apply_batched also in scope for this issue? And map/map_batched?

Should we have a backend options to switch between multi-processing and multi-threading?

As far as I understood we only wanted to keep multiprocessing, not multithreading @samsja

@JoanFM maybe know more. But to me only multi processing makes sense. Can multi threading really improve performance here ?

Just the keep the PR discussion up to date with what we discussed on Discord: there will be multi threading since it makes sense for IO bound ops and tf/np/torch stuff

Is apply_batched also in scope for this issue? And map/map_batched?

Map is already included, but private right now. Not sure if we want to expose it again, or only one of the two.
The batched one, I can add in this PR, but right now I am still trying to get the benchmark to work correctly. Test is passing in the CI, but only because it is being skipped I think, because there in only 1 CPU available I think, therefore no performance comparison with 1 cpu vs 2 cpus.

Looks good! We will make the map functions public, right?

Looks good! We will make the map functions public, right?

I dont really have an opinion on this. @samsja suggested to keep only one of the two exposed, map or apply. Why would you prefer this @samsja

Looks good! We will make the map functions public, right?

I dont really have an opinion on this. @samsja suggested to keep only one of the two exposed, map or apply. Why would you prefer this @samsja

Why should we keep only one @samsja? I think both make sense, the user might already have an in-place or pure function that they want to use without rewriting

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

📝 Docs are deployed on https://ft-feat-map-apply--jina-docs.netlify.app 🎉

github-actions Bot added size/m area/core area/testing labels Feb 28, 2023

anna-charlotte mentioned this pull request Feb 28, 2023

Meta: DocArray v2 Roadmap #780

Closed

47 tasks

anna-charlotte linked an issue Feb 28, 2023 that may be closed by this pull request

Map and apply with multiprocessing #1173

Closed

anna-charlotte force-pushed the feat-map-apply branch from 0c3bdfd to d72cbf5 Compare March 1, 2023 08:53

github-actions Bot added area/cicd area/housekeeping area/setup labels Mar 1, 2023

anna-charlotte force-pushed the feat-map-apply branch from 93be562 to d71de73 Compare March 1, 2023 12:45

JohannesMessner reviewed Mar 1, 2023

View reviewed changes

anna-charlotte force-pushed the feat-map-apply branch from 3a943a6 to 6c8176a Compare March 2, 2023 08:45

github-actions Bot added component/array size/l and removed size/m labels Mar 2, 2023

anna-charlotte force-pushed the feat-map-apply branch from 92d7a46 to 4c27409 Compare March 2, 2023 14:59

samsja reviewed Mar 2, 2023

View reviewed changes

Comment thread docarray/utils/apply.py Outdated

anna-charlotte force-pushed the feat-map-apply branch from 9263bdb to 14e3cc1 Compare March 2, 2023 16:42

JohannesMessner requested changes Mar 3, 2023

View reviewed changes

Comment thread docarray/utils/apply.py

Comment thread docarray/utils/apply.py

Comment thread docarray/utils/apply.py Outdated

Comment thread docarray/utils/apply.py Outdated

Comment thread docarray/utils/apply.py Outdated

JohannesMessner requested changes Mar 3, 2023

View reviewed changes

Comment thread docarray/utils/apply.py Outdated

Comment thread docarray/utils/apply.py Outdated

anna-charlotte force-pushed the feat-map-apply branch from c7bef8f to a84b56d Compare March 3, 2023 09:33

anna-charlotte marked this pull request as ready for review March 3, 2023 09:38

anna-charlotte requested review from JohannesMessner and samsja March 3, 2023 09:38

samsja reviewed Mar 3, 2023

View reviewed changes

Comment thread .github/workflows/ci.yml

samsja reviewed Mar 3, 2023

View reviewed changes

Comment thread docarray/utils/apply.py

anna-charlotte added 17 commits March 3, 2023 14:30

fix: add tests for func da to doc and da to other len da

18a377b

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: revert last commit

245283f

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: add len assert

76fe8b7

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: add assertions

34b7f9c

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: add test to for da extend in batch apply

c7a968d

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: extend with only one doc

6cf8ed2

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: fix

5dc9e6d

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: test

d3fc203

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: test

45cdc4a

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: set docs in apply

9839602

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: indices

87a93ff

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: indices

eeb7fae

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: indices

72aaf21

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: indices

c0f8029

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix:test

9b83c1f

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: mypy

7638d86

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: type hint

4a3a290

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

anna-charlotte force-pushed the feat-map-apply branch from 6a8b753 to 4a3a290 Compare March 3, 2023 13:30

anna-charlotte changed the title feat: add apply function feat: add map function Mar 3, 2023

anna-charlotte added 2 commits March 3, 2023 15:15

fix: remove apply, only keep map

38aae7a

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

refactor: map to map_docs

01900c9

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

samsja reviewed Mar 3, 2023

View reviewed changes

Comment thread docarray/utils/map.py

samsja reviewed Mar 3, 2023

View reviewed changes

Comment thread docarray/utils/map.py Outdated

anna-charlotte added 2 commits March 3, 2023 15:51

fix: apply suggestion

f6921e0

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

docs: add example usage

c3fb041

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

samsja approved these changes Mar 3, 2023

View reviewed changes

JohannesMessner approved these changes Mar 3, 2023

View reviewed changes

anna-charlotte merged commit 6cd5912 into feat-rewrite-v2 Mar 3, 2023

anna-charlotte deleted the feat-map-apply branch March 3, 2023 15:42

+                  func: Callable[[BaseDocument], BaseDocument],
+                  num_worker: Optional[int] = None,
+                  pool: Optional['Pool'] = None,
+                  show_progress: bool = False,

feat: add map function by anna-charlotte · Pull Request #1187 · docarray/docarray · GitHub

Uh oh!

feat: add map function - #1187

feat: add map function#1187anna-charlotte merged 51 commits intofeat-rewrite-v2docarray/docarray:feat-rewrite-v2from feat-map-applydocarray/docarray:feat-map-applyCopy head branch name to clipboard

Conversation

anna-charlotte commented Feb 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesMessner commented Mar 1, 2023

Uh oh!

JohannesMessner Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

anna-charlotte Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

samsja Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

JohannesMessner Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

anna-charlotte commented Mar 1, 2023

Uh oh!

Uh oh!

JohannesMessner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anna-charlotte commented Mar 3, 2023

Uh oh!

Uh oh!

JohannesMessner commented Mar 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: add map function#1187
anna-charlotte merged 51 commits into
feat-rewrite-v2from
feat-map-apply

anna-charlotte commented Feb 28, 2023 •
edited
Loading

JohannesMessner left a comment •
edited
Loading