feat(ingestion): support delete ingested doc and garbage collection #4203
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
f3552a2 to
bd6be36
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR introduces support for deleting ingested documents and extends garbage collection to include deletion by source, while also updating the ingestion service and related endpoints.
- Implements two new endpoints for deleting ingested documents (by document id and by source).
- Updates background job runners and garbage collection logic to incorporate ingestion deletion.
- Updates database queries and indexer functionality to support the new deletion features.
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| ee/tabby-webserver/src/service/ingestion.rs | Adds new ingestion service methods for getting and deleting ingested documents. |
| ee/tabby-webserver/src/service/background_job/mod.rs & hourly.rs | Updates background job invocations to pass the ingestion service. |
| ee/tabby-webserver/src/routes/mod.rs & ingestion.rs | Introduces new HTTP DELETE routes for ingestion deletions. |
| ee/tabby-schema/src/schema/ingestion.rs | Updates the IngestionService trait with deletion methods. |
| ee/tabby-db/src/ingestion.rs | Adds SQL queries for deleting ingested documents. |
| crates/tabby-index/* | Adjusts indexer and structured doc garbage collector implementations to accommodate deletion by ingestion status. |
Comments suppressed due to low confidence (2)
ee/tabby-webserver/src/service/ingestion.rs:114
- [nitpick] Consider renaming the parameters 'source' and 'id' to 'source_name' and 'doc_id' (or similar) for clarity, as 'source' is transformed to a source_id via source_id_from_name later in the function.
async fn delete(&self, source: String, id: String) -> Result<()> {
crates/tabby-index/src/indexer.rs:334
- [nitpick] Since iter_ids now returns a tuple (source, id) instead of a single id, consider updating the function name or adding a comment to clearly communicate the new return type to avoid potential confusion in downstream consumers.
pub fn iter_ids(&self) -> impl Stream<Item = (String, String)> + '_ {
wsxiaoys
approved these changes
May 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
API
GC
As for
2., Tantivy can not query ingested doc with streaming return, so we are iterating all structured docsTest
ingested docs:
ingested index:
deleted doc:
after GC:
ingested doc with id
page 123 456 678has gone in index: