FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

LUCENE-10236: Updated field-weight used in CombinedFieldQuery scoring calculation, and added a test by zacharymorn · Pull Request #444 · apache/lucene · GitHub

/ lucene Public

LUCENE-10236: Updated field-weight used in CombinedFieldQuery scoring calculation, and added a test - #444

Merged
zacharymorn merged 3 commits into
apache:mainfrom
zacharymorn:LUCENE-10236-CombinedFieldQuery-FieldWeight-Fix
Nov 19, 2021
Merged

LUCENE-10236: Updated field-weight used in CombinedFieldQuery scoring calculation, and added a test#444
zacharymorn merged 3 commits into
apache:mainfrom
zacharymorn:LUCENE-10236-CombinedFieldQuery-FieldWeight-Fix

Conversation

Copy link
Copy Markdown
Contributor

Description

Updated field-weight used in CombinedFieldQuery scoring calculation

Tests

  1. Added a new test from LUCENE-10061: Implements dynamic pruning support for CombinedFieldsQuery #418
  2. Run ./gradlew clean; ./gradlew check -Pvalidation.git.failOnModified=false

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
  • I have developed this patch against the main branch.
  • I have run ./gradlew check.
  • I have added tests for my changes.

jimczi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM, good catch @zacharymorn

: "There is duplicated field ["
+ field.field
+ "] used to construct MultiNormsLeafSimScorer";
duplicateCheckingSet.add(field.field);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Could be assert duplicateCheckingSet.add(field.field) == false ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Ah yes. I assume you meant assert duplicateCheckingSet.add(field.field) and have updated it accordingly.

jtibshirani left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Thanks for catching this!

dir.close();
}

public void testSameScoreAndCollectionBetweenCompleteAndTopScores() throws IOException {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Could you explain how this test relates to the fix? Would it make sense to have a more targeted test similar to testCopyField?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This test was actually developed in another related PR #418, and it would generate duplicated field-weight pairs in fields object to trigger the condition. As the existing tests don't currently trigger that condition, I thought this would be a good test to put in first for both PRs.

I guess if we were to have a focused test for this PR, it would still look something very similar to this test (and testCopyField), maybe something like the following, and it just ensures the test run through without assertion exception. The delta between the tests are just I removed some doc content randomizations and result comparison between TOP_SCORE and COMPLETE collection.

 public void testShouldRunWithoutAssertionException() throws IOException {
    int numDocs =
        randomBoolean()
            ? atLeast(1000)
            : atLeast(128 * 8 * 8 * 3); // make sure some terms have skip data
    int numMatchDoc = randomIntBetween(200, 500);
    int numHits = atMost(100);
    int boost1 = Math.max(1, random().nextInt(5));
    int boost2 = Math.max(1, random().nextInt(5));

    Directory dir = newDirectory();
    Similarity similarity = randomCompatibleSimilarity();

    IndexWriterConfig iwc = new IndexWriterConfig();
    iwc.setSimilarity(similarity);
    RandomIndexWriter w = new RandomIndexWriter(random(), dir, iwc);

    // adding potentially matching doc
    for (int i = 0; i < numMatchDoc; i++) {
      Document doc = new Document();

      int freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "foo", Store.NO));
        }
      }

      freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "foo" + j, Store.NO));
        }
      }

      freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "zoo", Store.NO));
        }
      }

      int freqB = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqB; j++) {
          doc.add(new TextField("b", "zoo", Store.NO));
        }
      }

      freqB = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqB; j++) {
          doc.add(new TextField("b", "zoo" + j, Store.NO));
        }
      }

      int freqC = random().nextInt(20) + 1;
      for (int j = 0; j < freqC; j++) {
        doc.add(new TextField("c", "bla" + j, Store.NO));
      }
      w.addDocument(doc);
    }

    IndexReader reader = w.getReader();
    IndexSearcher searcher = newSearcher(reader);
    searcher.setSimilarity(similarity);

    CombinedFieldQuery query =
        new CombinedFieldQuery.Builder()
            .addField("a", (float) boost1)
            .addField("b", (float) boost2)
            .addTerm(new BytesRef("foo"))
            .addTerm(new BytesRef("zoo"))
            .build();

    TopScoreDocCollector completeCollector =
        TopScoreDocCollector.create(numHits, null, Integer.MAX_VALUE); 
    searcher.search(query, completeCollector);

    reader.close();
    w.close();
    dir.close();
  }

I guess I'm fine either way? If you prefer a more focused test, I can replace it with the above one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think starting with this more focused test makes sense for this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Sounds good. I've replaced the test with a more focused one.

Copy link
Copy Markdown
Contributor Author

Thanks @jimczi @jtibshirani for the review and feedback!

jtibshirani left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Looks good to me too!

zacharymorn merged commit 07ee3ba into apache:main Nov 19, 2021

Copy link
Copy Markdown
Contributor Author

Will backport this to 8.11 and 8x.

Copy link
Copy Markdown
Contributor Author

Will backport this to 8.11 and 8x.

Hi @jimczi @jtibshirani @mikemccand, just to confirm as I saw there was a thread on the proper handling of branch_8x, this change should be backported to branches branch_8_11 (version 8.11.2), branch_9_0 (version 9.0.1) & branch_9x (version 9.1.0), but not to branch branch_8x right?

jpountz commented Jan 5, 2022

Copy link
Copy Markdown
Contributor

Correct, changes should no longer be backported to branch_8x.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants


Back | FazBrowse Home | New Git URL