FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

refactor: hnswlib performance by JoanFM · Pull Request #1727 · docarray/docarray · GitHub

/ docarray Public

refactor: hnswlib performance - #1727

Merged
JoanFM merged 15 commits into
mainfrom
refactor-hnswlib-performance
Jul 31, 2023
ConversationCommits (15)ChecksFiles changed
Merged

refactor: hnswlib performance#1727
JoanFM merged 15 commits into
mainfrom
refactor-hnswlib-performance

Conversation

JoanFM commented Jul 26, 2023

Copy link
Copy Markdown
Member

No description provided.

Joan Fontanals Martinez added 2 commits July 26, 2023 12:44
refactor: hnswlib performance
9aebf2e 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
Merge branch 'main' of https://github.com/docarray/docarray into refa…
02849c4 
…ctor-hnswlib-performance

codecov Bot commented Jul 26, 2023 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Patch coverage: 76.59% and project coverage change: -0.01% ⚠️

Comparison is base (896c20b) 84.46% compared to head (c062c8c) 84.45%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1727      +/-   ##
==========================================
- Coverage   84.46%   84.45%   -0.01%     
==========================================
  Files         134      134              
  Lines        8798     8834      +36     
==========================================
+ Hits         7431     7461      +30     
- Misses       1367     1373       +6
Flag Coverage Δ
docarray 84.45% <76.59%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
docarray/index/backends/hnswlib.py 74.28% <76.59%> (+0.58%) ⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fix: reconstruct on filter
3f56d44 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the refactor-hnswlib-performance branch from f7bd4d0 to 3f56d44 Compare July 27, 2023 01:11
fix: reconstruct from proper type
f8bb5f9 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the refactor-hnswlib-performance branch from 4f6c270 to f8bb5f9 Compare July 27, 2023 03:17
Joan Fontanals Martinez added 3 commits July 27, 2023 05:56
github-actions Bot added size/m and removed size/s labels Jul 27, 2023
Merge branch 'get-current-count-hnswlib' of https://github.com/docarr…
79b0fea 
…ay/docarray into refactor-hnswlib-performance

Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the refactor-hnswlib-performance branch from 3b0922a to 79b0fea Compare July 27, 2023 05:33
JoanFM force-pushed the refactor-hnswlib-performance branch 2 times, most recently from fdfcb56 to 74a187e Compare July 28, 2023 02:27
chore: apply optim
8c0bf6c 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
JoanFM force-pushed the refactor-hnswlib-performance branch from 74a187e to 8c0bf6c Compare July 28, 2023 02:32
JoanFM marked this pull request as ready for review July 28, 2023 18:03

github-actions Bot commented Jul 30, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-refactor-hnswlib-performance--jina-docs.netlify.app 🎉

JoanFM requested a review from jupyterjazz July 30, 2023 13:16
JoanFM requested review from JohannesMessner, jupyterjazz and samsja and removed request for JohannesMessner, jupyterjazz and samsja July 30, 2023 13:16
samsja reviewed Jul 31, 2023
View reviewed changes
Comment thread docarray/index/backends/hnswlib.py
rows = self._execute_filter(filter_query=filter_query, limit=limit)
return DocList[self.out_schema](self._doc_from_bytes(blob) for _, blob in rows) # type: ignore[name-defined]
hashed_ids = [doc_id for doc_id, _ in rows]
embeddings: OrderedDict[str, list] = OrderedDict()

samsja Jul 31, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Why specifically OrderedDict? I think normal dict in python will already be ordered by default

JoanFM Jul 31, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I would still like to be specific about the requirement for order

jupyterjazz reviewed Jul 31, 2023
View reviewed changes

jupyterjazz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

do we have any measure on how much faster hnsw works now?

Comment thread docarray/index/backends/hnswlib.py
self._hnsw_indices[col_name] = self._create_index(col_name, col)
self._logger.info(f'Created a new index for column `{col_name}`')
if self._hnsw_indices[col_name].space == 'cosine':
cosine_metric_index_exist = True

jupyterjazz Jul 31, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

why do we care about this?

JoanFM Jul 31, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Because for cosine, HNSWLib normalizes the vectors, and then if we retrieve, they have chanded, so no consistent API can be provided

JoanFM commented Jul 31, 2023

Copy link
Copy Markdown
Member Author

do we have any measure on how much faster hnsw works now?

We have observed around 10% improvement

JoanFM requested review from jupyterjazz and samsja July 31, 2023 08:36
JoanFM merged commit a643f6a into main Jul 31, 2023
JoanFM deleted the refactor-hnswlib-performance branch July 31, 2023 08:39
jupyterjazz mentioned this pull request Aug 1, 2023
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

JohannesMessner Awaiting requested review from JohannesMessner
samsja Awaiting requested review from samsja
1 more reviewer
jupyterjazz jupyterjazz approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/core area/testing size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL