System design and implementation for a web crawler.
This will be the high level flow of the data, initally the api will call DB, which will inturn call the crawler which will fetch html pages and parse information. The crawler also needs URLs to fetch html pages which would be in our URL DB, we would initally feed some pages to the URL DB but it will be self sustaining loop where crawler would provide URLs to URL DB after parsing webpages.