nodejs
npm
yarn
mysql
npm install @types/puppeteer
npm install -g socket.io
yran isntall
yarn start-http-server
// Post request to: http://localhost:3000/url { url: 'https://the.url' }
// Check the test-http-server.html for the sample usage.
yarn start-socket-server
yran shell --headless=false --instances=1 --sites=8 --waitfor=3000 --timeout=60000
"--headless": Boolean,
"--instances": Integer (number of chrome intsances)
"--tabs": Integer, (number of tabs per instance)
"--sites": Integer, (number of sites to crawl)
"--waitfor": Integer (ms after puppeteer page.goto())
"--timeout": Integer (ms puppeteer page.goto() timeout)
To be able to test the extractor api:
- Run the servers above (http, socket)
- Load the
test-http-server.htmlortest-socket-server.htmlfrom theclientsfolder