8000 fix: filter out null row ids by BubbleCal · Pull Request #4020 · lancedb/lance · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fix: filter out null row ids #4020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 18, 2025
Merged

fix: filter out null row ids #4020

merged 5 commits into from
Jun 18, 2025

Conversation

BubbleCal
Copy link
Contributor

No description provided.

@BubbleCal
fix: filter out null row ids
011b669 
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@github-actions github-actions bot added the bug Something isn't working label Jun 17, 2025
@BubbleCal BubbleCal requested a review from westonpace June 17, 2025 09:08
@BubbleCal BubbleCal marked this pull request as ready for review June 17, 2025 09:08
@codecov-commenter
Copy link
codecov-commenter commented Jun 17, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 62.50000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 78.61%. Comparing base (f86f2dd) to head (08e581a).

Files with missing lines Patch % Lines
rust/lance/src/io/exec/take.rs 60.00% 2 Missing ⚠️
rust/lance-index/src/scalar/inverted/index.rs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #4020   +/-   ##
=======================================
  Coverage   78.61%   78.61%           
=======================================
  Files         285      285           
  Lines      113097   113101    +4     
  Branches   113097   113101    +4     
=======================================
+ Hits        88916    88920    +4     
+ Misses      20744    20743    -1     
- Partials     3437     3438    +1
Flag Coverage Δ
unittests 78.61% <62.50%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@westonpace
Copy link
Contributor

If we can't handle nulls it might be easier to just filter them out at the source. For example, in scanner.rs in plan_match_query we can change:

            let flat_fts_scan_schema = Arc::new(self.dataset.schema().project(&columns).unwrap());
            let mut scan_node = self.scan_fragments(
                true,
                false,
                /*with_make_deletions_null=*/ true,
                flat_fts_scan_schema,
                Arc::new(unindexed_fragments),
                None,
                false,
            );

to...

            let flat_fts_scan_schema = Arc::new(self.dataset.schema().project(&columns).unwrap());
            let mut scan_node = self.scan_fragments(
                true,
                false,
                /*with_make_deletions_null=*/ false,
                flat_fts_scan_schema,
                Arc::new(unindexed_fragments),
                None,
                false,
            );

Here's a python test to reproduce:

def test_fts_deleted_rows(tmp_path):
    data = pa.table({"text": ["lance is cool", "databases are cool", "search is neat"]})
    ds = lance.write_dataset(data, tmp_path)
    ds.create_scalar_index("text", "INVERTED")
    ds.insert(
        pa.table({"text": ["lance is cool", "databases are cool", "search is neat"]})
    )

    ds.delete("text = 'lance is cool'")
    results = ds.to_table(full_text_query="cool")
    assert results.num_rows == 2

@BubbleCal
add tests
ff06bcd 
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@BubbleCal
remove logs
d954efc 
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
LuQQiu
LuQQiu reviewed Jun 17, 2025
View reviewed changes
BubbleCal added 2 commits June 18, 2025 12:29
@BubbleCal
move comment
1482fe5 
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@BubbleCal
Merge branch 'main' of https://github.com/lancedb/lance into fix-take…
08e581a 
…-nulls
@BubbleCal BubbleCal merged commit a499cfa into lancedb:main Jun 18, 2025
43 of 51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LuQQiu LuQQiu LuQQiu approved these changes

@westonpace westonpace Awaiting requested review from westonpace

Assignees
No one assigned
Labels
bug Something isn't working python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@BubbleCal @codecov-commenter @westonpace @LuQQiu
0