8000 fix: prevent and handle empty fragments by wjones127 · Pull Request #3934 · lancedb/lance · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fix: prevent and handle empty fragments #3934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 4, 2025
Merged

fix: prevent and handle empty fragments #3934

merged 5 commits into from
Jun 4, 2025

Conversation

wjones127
Copy link
Contributor
@wjones127 wjones127 commented Jun 4, 2025 •
edited
Loading

#3933

We found it's possible to create datasets with empty fragments, and that this can break remapping.

This PR:

  1. Makes re-mapping gracefully return early when it detects that the re-map is just all deletions. This avoids the error reported in the issue.
  2. Has the commit step remove any empty fragments. This will fix any broken datasets on any write transaction.
  3. Makes sure that the TransactionRebase will delete full fragments if the merged deletion file covers the whole fragment. This was the source of the empty fragments.

@github-actions github-actions bot added the bug Something isn't working label Jun 4, 2025
@codecov-commenter
Copy link
codecov-commenter commented Jun 4, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 97.20280% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.39%. Comparing base (0cfaf95) to head (a84b908).

Files with missing lines Patch % Lines
rust/lance/src/dataset/write/merge_insert.rs 97.72% 0 Missing and 2 partials ⚠️
rust/lance/src/index.rs 97.29% 1 Missing ⚠️
rust/lance/src/io/commit/conflict_resolver.rs 93.33% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3934      +/-   ##
==========================================
+ Coverage   78.36%   78.39%   +0.02%     
==========================================
  Files         284      284              
  Lines      109925   110060     +135     
  Branches   109925   110060     +135     
==========================================
+ Hits        86146    86280     +134     
- Misses      20402    20403       +1     
  Partials     3377     3377
Flag Coverage Δ
unittests 78.39% <97.20%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wjones127 wjones127 marked this pull request as ready for review June 4, 2025 02:21
@wjones127 wjones127 requested a review from LuQQiu June 4, 2025 02:22
westonpace
westonpace approved these changes Jun 4, 2025
View reviewed changes
Copy link
Contributor
@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find

@wjones127 wjones127 merged commit 4eb2d81 into lancedb:main Jun 4, 2025
30 checks passed
@wjones127 wjones127 deleted the fix/drop-empty-fragments branch June 4, 2025 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@westonpace westonpace westonpace approved these changes

@LuQQiu LuQQiu Awaiting requested review from LuQQiu

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wjones127 @codecov-commenter @westonpace
0