FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix: fix search in memory with AnyEmbedding by JoanFM · Pull Request #1696 · docarray/docarray · GitHub

/ docarray Public

fix: fix search in memory with AnyEmbedding - #1696

Merged
JoanFM merged 6 commits into
mainfrom
anytensor-in-memory-search-fix
Jul 12, 2023
ConversationCommits (6)ChecksFiles changed
Merged

fix: fix search in memory with AnyEmbedding#1696
JoanFM merged 6 commits into
mainfrom
anytensor-in-memory-search-fix

Conversation

JoanFM commented Jul 10, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Allows to do easy search in Memory when AnyEmbedding is used as with TextDoc or ImageDoc:

index = InMemoryExactNNIndex[TextDoc]()

docs = DocList[TextDoc](
    [TextDoc(text='hey', embedding=np.random.rand(128)) for _ in range(200)]
)
index.index(docs)
res = index.find_batched(docs[0:10], search_field='embedding', limit=5)

JoanFM force-pushed the anytensor-in-memory-search-fix branch from e484ff0 to f96171f Compare July 10, 2023 17:01
fix: allow find in in memory index with AnyEmbedding
bc0e40d 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the anytensor-in-memory-search-fix branch from f96171f to bc0e40d Compare July 10, 2023 17:11
jupyterjazz reviewed Jul 10, 2023
View reviewed changes

jupyterjazz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

why was AnyEmbedding failing?

JoanFM commented Jul 10, 2023

Copy link
Copy Markdown
Member Author

why was AnyEmbedding failing?

It does not have a computational backend attached, so it cannot stack or do anything on tensors

JoanFM marked this pull request as draft July 10, 2023 22:22
samsja reviewed Jul 11, 2023
View reviewed changes

samsja left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

JoanFM commented Jul 11, 2023

Copy link
Copy Markdown
Member Author

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

and what about the compBackend, is it okey If I return the NumpyBackend for AnyTensor as default?

test: add tests for torch and tf
e7315e8 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the anytensor-in-memory-search-fix branch from d36a0b5 to e7315e8 Compare July 11, 2023 07:24
github-actions Bot added size/m and removed size/s labels Jul 11, 2023
JoanFM force-pushed the anytensor-in-memory-search-fix branch 2 times, most recently from a41e3f7 to 4e013b4 Compare July 11, 2023 10:26
fix: try to pass all mypy checks
2babe51 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4e013b4 to 2babe51 Compare July 11, 2023 10:51
JoanFM requested review from jupyterjazz and samsja July 11, 2023 10:56
JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4d8a20a to 57bf37b Compare July 11, 2023 13:47
test: undo changes in docker compose
bab8099 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the anytensor-in-memory-search-fix branch from 57bf37b to bab8099 Compare July 11, 2023 14:30
JoanFM marked this pull request as ready for review July 11, 2023 14:46

github-actions Bot commented Jul 11, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-anytensor-in-memory-search-fix--jina-docs.netlify.app 🎉

JoanFM merged commit 94a479e into main Jul 12, 2023
JoanFM deleted the anytensor-in-memory-search-fix branch July 12, 2023 07:11
JoanFM linked an issue Jul 13, 2023 that may be closed by this pull request
TextDoc not easily indexable with InMemory and HNSWLib #1695
Closed
6 tasks
JoanFM mentioned this pull request Jul 14, 2023
Merged
8 tasks
jupyterjazz mentioned this pull request Jul 18, 2023
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

samsja samsja approved these changes
JohannesMessner Awaiting requested review from JohannesMessner
jupyterjazz Awaiting requested review from jupyterjazz

Assignees

No one assigned

Labels

area/core area/testing area/typing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TextDoc not easily indexable with InMemory and HNSWLib

3 participants

Back | FazBrowse Home | New Git URL