8000 feat(storage): implement in memory hnsw builder by wenym1 · Pull Request #21648 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(storage): implement in memory hnsw builder #21648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: yiming/hummock-flat-index
Choose a base branch
from
Open

feat(storage): implement in memory hnsw builder #21648

wants to merge 7 commits into from

Conversation

wenym1
Copy link
Contributor
@wenym1 wenym1 commented Apr 29, 2025

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

This was referenced Apr 29, 2025
Merged
Merged
Merged
@wenym1 Graphite App
Copy link
Contributor Author
wenym1 commented Apr 29, 2025 •
edited
Loading

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@github-actions github-actions bot added the type/feature Type: New feature. label Apr 29, 2025
@wenym1 wenym1 marked this pull request as ready for review April 29, 2025 10:26
@wenym1 wenym1 requested a review from a team as a code owner April 29, 2025 10:26
@wenym1 wenym1 requested review from xiangjinwu and removed request for a team April 29, 2025 10:26
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from 6b6ae10 to 4901f27 Compare April 29, 2025 11:22
@wenym1 wenym1 force-pushed the yiming/in-memory-flat-vector-index branch from 6b09706 to 1cdde0e Compare May 6, 2025 08:50
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch 4 times, most recently from ab5ca20 to f67b272 Compare May 6, 2025 13:16
@wenym1 wenym1 force-pushed the yiming/in-memory-flat-vector-index branch 2 times, most recently from 99acaeb to 624ac0d Compare May 12, 2025 05:47
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from f67b272 to 7b4bed2 Compare May 12, 2025 05:47
@wenym1 wenym1 mentioned this pull request May 12, 2025
Open
8 tasks
@wenym1 wenym1 force-pushed the yiming/in-memory-flat-vector-index branch from 624ac0d to 9301c64 Compare May 14, 2025 03:34
Base automatically changed from yiming/in-memory-flat-vector-index to main May 14, 2025 06:25
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from 7b4bed2 to 929ee4b Compare May 20, 2025 05:20
hzxa21
hzxa21 approved these changes May 22, 2025
View reviewed changes
Copy link
Collaborator
@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

ci/.env
@@ -1 +1 @@
BUILD_ENV_VERSION=v20250509-1
BUILD_ENV_VERSION=FAISS-v20250506
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? related to a new image with the new dependencies libblas-dev liblapack-dev libomp-dev for linear algebra ?

src/storage/src/vector/distance.rs
assert_eq!(len, second.0.len());
let mut sum = 0.0;
let mut start = 0;
let mut end = start + 32;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a check to make sure the length of input vec can be divided by 32?

@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from 929ee4b to 60444ef Compare May 27, 2025 12:01
@wenym1 wenym1 changed the base branch from main to yiming/hummock-flat-index May 27, 2025 12:01
This was referenced May 27, 2025
Merged
Open
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch 3 times, most recently from e871e22 to cdcba01 Compare June 3, 2025 16:38
@wenym1 wenym1 force-pushed the yiming/hummock-flat-index branch from fa520a8 to dc2effc Compare June 4, 2025 08:17
@wenym1 wenym1 mentioned this pull request Jun 4, 2025
Open
8 tasks
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from cdcba01 to e4c169b Compare June 4, 2025 08:17
@wenym1 wenym1 force-pushed the yiming/hummock-flat-index branch from dc2effc to 928b966 Compare June 4, 2025 08:32
@wenym1 wenym1 force-pushed the yiming/in-memory-hnsw-builder branch from e4c169b to a1a555d Compare June 4, 2025 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hzxa21 hzxa21 hzxa21 approved these changes

@xiangjinwu xiangjinwu Awaiting requested review from xiangjinwu xiangjinwu is a code owner automatically assigned from risingwavelabs/cargo-lock

Assignees
No one assigned
Labels
type/feature Type: New feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wenym1 @hzxa21
0