Generates a report that contains project and contributor data for a specific topic
Check out the post.
- Docker
- Flask
- Celery
- Redis
- Flower
- Pandas
- BeautifulSoup
Required to run this program
- Docker
- Docker Compose
- Scraperapi.com account (free)
Add Scraperapi.com API KEY to docker-compose file
Using the Makefile:
- start (start for the first time)
- rebuild (apply changes)
- teardown (tear down project)
Commandline:
- $ docker-compose up -d --build
Browser:
- Open browser to http://localhost:5004 to view the app
- Open browser to http://localhost:5556 to view the Flower dashboard.
Commandline:
Trigger a new task:
- $ curl http://localhost:5004/tasks -H "Content-Type: application/json" --data '{"topic": "gig"}'
Check the status:
- $ curl http://localhost:5004/tasks/<TASK_ID>/
Reports & Logs
- reports can be found under reports folder created once you run the app
- logs can be found in the logs folder in the project directory
TODO
- complete tests
- add crawlers for GH search page