FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: filtering in hnsw by jupyterjazz · Pull Request #1718 · docarray/docarray · GitHub

/ docarray Public

feat: filtering in hnsw - #1718

Merged
JoanFM merged 6 commits into
mainfrom
feat-hnsw-prefiltering
Jul 26, 2023
ConversationCommits (6)ChecksFiles changed
Merged

feat: filtering in hnsw#1718
JoanFM merged 6 commits into
mainfrom
feat-hnsw-prefiltering

Conversation

jupyterjazz commented Jul 20, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Support Filtering for HnswDocumentIndex, separately as well as inside the query builder

Main changes:

  • Add filtrable fields separately to the sqlite table (so now we store id, doc_blob, and fields that can be filtered)
  • Support filter function which does not touch hnsw and executes filtering on the sqlite table
  • Refactor the query builder (hybrid search) to enable pre+post filtering. Post filtering is done in the same way; Pre filtering first queries the SQL table, gets hashed_ids for the filtered documents, and passes it to hnsw's new feature that can filter vectors based on their ids during search.

Notes:
query builder became quite complex even tho I tried decomposing it as much as possible.
i think codecov has some problems?

feat: filtering in hnsw
cd51624 
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
jupyterjazz linked an issue Jul 20, 2023 that may be closed by this pull request
feat: filtering in HNSW #1717
Closed
2 tasks

codecov Bot commented Jul 20, 2023 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Patch coverage: 39.56% and project coverage change: -1.06% ⚠️

Comparison is base (7ad70bf) 85.54% compared to head (e67dfba) 84.49%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1718      +/-   ##
==========================================
- Coverage   85.54%   84.49%   -1.06%     
==========================================
  Files         133      133              
  Lines        8608     8733     +125     
==========================================
+ Hits         7364     7379      +15     
- Misses       1244     1354     +110
Flag Coverage Δ
docarray 84.49% <39.56%> (-1.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
docarray/index/backends/hnswlib.py 74.34% <39.56%> (-20.99%) ⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

style: mypys the worst
75d0bc3 
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
style: minor changes
ed621ef 
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
jupyterjazz marked this pull request as ready for review July 21, 2023 10:15
chore: bump hnswlib version
1a7eeb9 
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
JoanFM requested changes Jul 25, 2023
View reviewed changes

JoanFM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I like it in general. Do we have an idea on the amount of extra space it takes to index all these into SQLIte? Should we have a DBConfig entry abou tthe fields that are to be filterable?

Comment thread tests/index/hnswlib/test_find.py
Comment thread docarray/index/backends/hnswlib.py
},
},
# `None` is not a Type, but we allow it here anyway
None: {}, # type: ignore

JoanFM Jul 25, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

not really know why it was here, but why removing it?

jupyterjazz Jul 25, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think it's because python_type_to_db_type function was returning None for types other than vector, but I changed it with defaultdict which returns an empty dict by default

Comment thread docarray/index/backends/hnswlib.py
if safe_issubclass(python_type, allowed_type):
return np.ndarray

type_map = {

JoanFM Jul 25, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

so we must be clear on what items are filterable and which are not. To be mindful when documenting

jupyterjazz Jul 25, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'll update docs here #1678

test: add a test with multiple limits
e67dfba 
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
jupyterjazz requested a review from JoanFM July 25, 2023 11:34

github-actions Bot commented Jul 25, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-hnsw-prefiltering--jina-docs.netlify.app 🎉

JoanFM merged commit 00e980d into main Jul 26, 2023
JoanFM deleted the feat-hnsw-prefiltering branch July 26, 2023 02:47
jupyterjazz mentioned this pull request Aug 1, 2023
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

JoanFM JoanFM approved these changes
JohannesMessner Awaiting requested review from JohannesMessner
samsja Awaiting requested review from samsja

Assignees

No one assigned

Labels

area/core area/setup area/testing size/l

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: filtering in HNSW

2 participants

Back | FazBrowse Home | New Git URL