List of encoder supported #1950
-
|
Hi Team, Can I use any embedding model for creating dense index? If not, is there a list of supported models? Best, |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 8 replies
-
|
Yes, start here: https://github.com/castorini/pyserini/blob/master/docs/usage-index.md#building-a-dense-vector-index I would also suggest that you go through our onboarding guide: https://github.com/castorini/onboarding/blob/master/README.md#-foundations-of-retrieval |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Thanks for prompt reply. The link you provided gave the command to generate run on TREC 2019 queries: python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--encoder-class auto \
--encoder BAAI/bge-base-en-v1.5 --l2-norm --query-prefix "Represent this sentence for searching relevant passages:" \
--index msmarco-v1-passage.bge-base-en-v1.5 \
--topics dl19-passage \
--output run.msmarco-v1-passage.bge-base-en-v1.5-pytorch.dl19.txt \
--hits 1000
However, I found that, for creating dense index, there is no https://github.com/castorini/pyserini/blob/master/pyserini/encode/__main__.py |
Beta Was this translation helpful? Give feedback.
-
|
I believe @MXueguang did the encoding using Tevatron [1] - but if you want to add the feature to |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much! One more question: According to the training script in tevatron, does that mean the model trained using default similarity measure used in tevatron is optimized on So using a model trained using tevatron to compute cosine similarity is wrong?
|
Beta Was this translation helpful? Give feedback.
-
|
Oh seems not. if I need a model to produce cosine similarity, I can do that by passing --normalize = True |
Beta Was this translation helpful? Give feedback.
Oh seems not.
if I need a model to produce cosine similarity, I can do that by passing --normalize = True