Project Description: DigiVigi is a 'DNIF Open Source' project which simply tries exhibiting a "How To?" process of analyzing real-time data inside DNIF from start to finish.
DNIF - Open Big Data Analytics Platform (Free Forever Version)
- Virtual Box
- JetBrains: PyCharm Community Edition
- Ubuntu 16.04 or above
- Docker
The execution of project is carried out in two process-phase. Its just a procedural way based on the diagram from issue #1
PROCESS 1: Refer Issue #1
Stage 1:
- Select & understand data-set from a domain of interest.
Stage 2:
- Understand DNIF platform and its capabilities limited to project scope.
- Here's a link to DNIF's complete documentation: https://dnif.it/docs/
Stage 3:
- Get the static data-set inside DNIF platform by following the guidelines mentioned on the website.
- One can use a ready made dataset(json, csv or excel) or create one by scripting it, like writing a web scrapper code.
Stage 4:
- Perform analytics, create dashboards, build alerts(for which one will have to configure SMTP in their container)
- Here's how to configure SMTP: https://dnif.it/docs/guides/tutorials/configuring-smtp-in-docker.html
NOTE:Process 1 will ensure you get a good grasp of executing the project on a more fundametal level, before moving on to advanced level.
PROCESS 2: Refer Issue #1
Stage 1:
- Select & understand data-set from a domain of interest. Dataset has to be dynamic this time. Meaning it can update over a period of min(s), hour(s), day(s), week(s) ... The selection should be careful because in a later stage this might count when it comes down to your machine processing speed (CPU, RAM, Disk Space) - - - Check out the pre-requisites on https://dnif.it/docs/guides/getting-started/prerequisites.html
Stage 2:
- Understand DNIF platform and its capabilities limited to project scope.
- Here's a link to DNIF's complete documentation: https://dnif.it/docs/
Stage 3:
- Fetch the data continuously from the source selected in stage one and store/ update it in a file (ex; mydata.csv)
- For this one can write a code to do so or using an existing one from the repo.
- This file will get updated continuously depending on your scheduler/cron job frequency or any other way of your choice.
Stage 4:
- The data captured needs to be fed to DNIF for it to be analyzed and played with.
- Write a connector/ API which does this job for you or use the existing one from the repo.
Stage5:
- Perform analytics, create dashboards, build alerts(for which one will have to configure SMTP in their container)
- Here's how to configure SMTP: https://dnif.it/docs/guides/tutorials/configuring-smtp-in-docker.html
Process 1:
Process 2:
Additional Credits to: SOC18-Genesis