Bootstrap the index with archives and data-dumps #105
Labels
enhancement
New feature or request
Comments
|
Is this still wanted or even possible at this stage to bootstrap a completely new index ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Introduction
The Mwmbl is currently severely lacking a proper index. However, we do know that most of the searched for content is originating from relatively few websites, most of which have first-party or third-party data dumps.
For the purposes of bootstrapping the index, perhaps a manual importer can be written for a common format like ZIM, which can reconstruct the URLs, process the HTML like any other web page, and add all to index?
Candidates
site:reddit.comto their every search, given the degrading quality of the results. But the database is rather large, growing up to terabytes if we include the latest data, might need plenty of filtering, and it might be in a bit of a legal gray zone, though probably lesser so, given it's a search engine.Additionally, there's a great >1TB webtext corpus, Google's Colossal Clean Crawled Corpus, which contains plenty of websites, which text can be simply thrown into the indexing pipeline.
The text was updated successfully, but these errors were encountered: