Anserini is a toolkit for reproducible information retrieval research. By building on Lucene, we aim to bridge the gap between academic information retrieval research and the practice of building real-world search applications. Among other goals, our effort aims to be the opposite of this.* Anserini grew out of a reproducibility study of various open-source retrieval engines in 2016 (Lin et al., ECIR 2016). See Yang et al. (SIGIR 2017) and Yang et al. (JDIQ 2018) for overviews.
Most Anserini features are exposed in the Pyserini Python interface. If you're more comfortable with Python, start there, although Anserini forms an important building block of Pyserini, so it remains worthwhile to learn about Anserini.
You'll need Java 11 and Maven 3.3+ to build Anserini.
Clone our repo with the --recurse-submodules option to make sure the eval/ submodule also gets cloned (alternatively, use git submodule update --init).
Then, build using using Maven:
mvn clean package appassembler:assemble
The tools/ directory, which contains evaluation tools and other scripts, is actually this repo, integrated as a Git submodule (so that it can be shared across related projects).
Build as follows (you might get warnings, but okay to ignore):
cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make && cd ../../..
cd tools/eval/ndeval && make && cd ../../..With that, you should be ready to go. The onboarding path for Anserini starts here!
Windows tips
Note that on Windows, tests may fail due to encoding issues, see #1466.
A simple workaround is to skip tests by adding -Dmaven.test.skip=true to the above mvn command.
See #1121 for additional discussions on debugging Windows build errors.
Anserini is designed to support experiments on various standard IR test collections out of the box.
The following experiments are backed by rigorous end-to-end regression tests with run_regression.py and the Anserini reproducibility promise.
For the most part, these runs are based on default parameter settings.
These pages can also serve as guides to reproduce our results.
See individual pages for details!
MS MARCO V1 Passage Regressions
| dev | DL19 | DL20 | |
|---|---|---|---|
| Unsupervised Sparse Lexical | |||
| BoW baselines | + | + | + |
| Quantized BM25 | ✓ | ✓ | ✓ |
| WP baselines | + | + | + |
| Huggingface WP baselines | + | + | + |
| doc2query | + | ||
| doc2query-T5 | + | + | + |
| Learned Sparse Lexical (uniCOIL family) | |||
| uniCOIL noexp | ✓ | ✓ | ✓ |
| uniCOIL with doc2query-T5 | ✓ | ✓ | ✓ |
| uniCOIL with TILDE | ✓ | ||
| Learned Sparse Lexical (other) | |||
| DeepImpact | ✓ | ||
| SPLADEv2 | ✓ | ||
| SPLADE-distill CoCodenser-medium | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-EnsembleDistil | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-EnsembleDistil (ONNX) | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-SelfDistil | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-SelfDistil (ONNX) | ✓ | ✓ | ✓ |
| Learned Dense | |||
| cosDPR-distil | ✓ | ✓ | ✓ |
| Corpora | Size | Checksum |
|---|---|---|
| Quantized BM25 | 1.2 GB | 0a623e2c97ac6b7e814bf1323a97b435 |
| uniCOIL (noexp) | 2.7 GB | f17ddd8c7c00ff121c3c3b147d2e17d8 |
| uniCOIL (d2q-T5) | 3.4 GB | 78eef752c78c8691f7d61600ceed306f |
| uniCOIL (TILDE) | 3.9 GB | 12a9c289d94e32fd63a7d39c9677d75c |
| DeepImpact | 3.6 GB | 73843885b503af3c8b3ee62e5f5a9900 |
| SPLADEv2 | 9.9 GB | b5d126f5d9a8e1b3ef3f5cb0ba651725 |
| SPLADE-distill CoCodenser-medium | 4.9 GB | f77239a26d08856e6491a34062893b0c |
| SPLADE++ CoCondenser-EnsembleDistil | 4.2 GB | e489133bdc54ee1e7c62a32aa582bc77 |
| SPLADE++ CoCondenser-SelfDistil | 4.8 GB | cb7e264222f2bf2221dd2c9d28190be1 |
| cosDPR-distil | 57 GB | e20ffbc8b5e7f760af31298aefeaebbd |
MS MARCO V1 Document Regressions
| dev | DL19 | DL20 | |
|---|---|---|---|
| Unsupervised Lexical, Complete Doc* | |||
| BoW baselines | + | + | + |
| WP baselines | + | + | + |
| Huggingface WP baselines | + | + | + |
| doc2query-T5 | + | + | + |
| Unsupervised Lexical, Segmented Doc* | |||
| BoW baselines | + | + | + |
| WP baselines | + | + | + |
| doc2query-T5 | + | + | + |
| Learned Sparse Lexical | |||
| uniCOIL noexp | ✓ | ✓ | ✓ |
| uniCOIL with doc2query-T5 | ✓ | ✓ | ✓ |
| Corpora | Size | Checksum |
|---|---|---|
| MS MARCO V1 doc: uniCOIL (noexp) | 11 GB | 11b226e1cacd9c8ae0a660fd14cdd710 |
| MS MARCO V1 doc: uniCOIL (d2q-T5) | 19 GB | 6a00e2c0c375cb1e52c83ae5ac377ebb |
MS MARCO V2 Passage Regressions
| dev | DL21 | DL22 | |
|---|---|---|---|
| Unsupervised Lexical, Original Corpus | |||
| baselines | + | + | + |
| doc2query-T5 | + | + | + |
| Unsupervised Lexical, Augmented Corpus | |||
| baselines | + | + | + |
| doc2query-T5 | + | + | + |
| Learned Sparse Lexical | |||
| uniCOIL noexp zero-shot | ✓ | ✓ | ✓ |
| uniCOIL with doc2query-T5 zero-shot | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-EnsembleDistil | ✓ | ✓ | ✓ |
| SPLADE++ CoCondenser-SelfDistil | ✓ | ✓ | ✓ |