Closed
Description
There is no information on where this list comes from. We need to get some sources and update the list.
This looks like a good place to start: https://github.com/JayBizzle/Crawler-Detect (other packages rely on this one, so it is probably a very good source)
Others:
- https://github.com/monperrus/crawler-user-agents
- https://github.com/loadkpi/crawler_detect
- https://github.com/nicolasmure/CrawlerDetectBundle
May be useful:
- https://github.com/antoinevastel/bots-zoo and in particular https://github.com/antoinevastel/bots-zoo/tree/main/headers
Query used: https://github.com/search?o=desc&q=crawler+bot&s=stars&type=Repositories