Reformat the data conversion stages to fit better with pipeline-oriented processing #30
Labels
enhancement
New feature or request
Comments
|
Hello @d-r-scott, thank you for your interest in our work! If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🚀 Feature Request
The data conversion stages (i.e. the conversion from
vcraftfiles tocodif, and then fromcodiftodifx) are structured around the assumption of a single user actively "doing" the processing. This is not ideal for the pipeline-oriented approach the processing has taken on since implementing Nextflow, and a more modular approach would be better.🔈 Motivation
Nextflow handles process parallelization and submission to slurm queues, which are both currently being handled internally to the conversion processes (specifically within
vcraft2obs.pyandaskap2difx.py). This improved efficiency in the old active-user approach, but is now causing the largest bottleneck in the localisation of FRBs. It also makes the consumption of resources and actual time required for the stages more opaque than it could be (e.g. Nextflow's reports know nothing about what the actual slurm jobs are doing, since all Nextflow sees is some jobs being submitted to a queue).🛰 Alternatives
Each of the conversion stages could be separated into their own processes, and their data handling possibly made more granular so that Nextflow can handle the parallelisation and minimise waiting time.
The text was updated successfully, but these errors were encountered: