Tags: lancedb/lance
Tags
fix: panic caused by reading empty posting list (#4019) it happens only if all conditions satisfied: - search with new search algo (block max wand), this means it's on the version that new FTS format has been introduced - the FTS index is still in legacy format - some posting list produces a candidate doc, that its doc id is greater than the first posting list's max doc id --------- Signed-off-by: BubbleCal <bubble-cal@outlook.com>
feat: wrap FilterExec in LanceFilterExec (#3939) This wraps usage of Datafusion FilterExec in a LanceFilterExec node that carries the original logical Expr for the filter alongside the wrapped FilterExec. The reason for this is to enable a datafusion optimizer rule to more easily serialize the execution logic of the filter (using substrait) and send the node over the wire.
fix: prevent and handle empty fragments (#3934) #3933 We found it's possible to create datasets with empty fragments, and that this can break remapping. This PR: 1. Makes re-mapping gracefully return early when it detects that the re-map is just all deletions. This avoids the error reported in the issue. 2. Has the commit step remove any empty fragments. This will fix any broken datasets on any write transaction. 3. Makes sure that the `TransactionRebase` will delete full fragments if the merged deletion file covers the whole fragment. This was the source of the empty fragments.
feat: auto conflict resolution for upsert (#3865) Continuation of #3785 with fixes for performance issues. Makes merge_insert and update transactions do row-level conflict resolution checks, to see if we can quickly resolve conflicts by rewriting deletion files. Core changes: - merge_insert and update transactions now produce an affected_rows output, with a map of the affected row addresses. - Create a TransactionRebase struct to handle conflict resolution. This uses the affected_rows output to attempt to rewrite deletion files. Auxiliary changes: - Changed all code paths reading deletion files to go through read_deletion_file_cached(). - Closes #3772 --------- Co-authored-by: Will Jones <willjones127@gmail.com>
PreviousNext