Open
Description
Description of the bug
When I try to reinstall a subworkflow or install another subworkflow/module after having deleted previously installed subworkflows in a pipeline, it seems like modules_json.py is not able install the missing subworkflows from parsing the data in the modules.json file.
I get errors like
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/bam_sort_stats_samtools' - removing from modules.json modules_json.py:588
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/bam_stats_samtools' - removing from modules.json modules_json.py:588
DEBUG Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/fastq_align_bwa' - removing from modules.json modules_json.py:588
INFO Was unable to reinstall some subworkflows. Removing 'modules.json' entries: 'nf-core/bam_sort_stats_samtools', 'nf-core/bam_stats_samtools', 'nf-core/fastq_align_bwa' modules_json.py:1182
Here it seems like modules_json.py is pointing to the wrong path for subworkflows. It points to modules/<organization>/<subworkflow_name>, when it should be pointing to subworkflows/<organization>/<subworkflow_name>.
I think I identified the cause of this:
When self.reinstall_repo() is called in this line of modules_json.py, we do not carry any information about component_type into the method call.
The self.install_repo() method just uses the self.modules_dir / install_dir as the path to component when calling modules_repo.install_component() see this line. This leads to the behaviour of incorrectly pointing to modules/ directory for subworkflows.
Command used and terminal output
$ nf-core --verbose --log-file test_log.txt subworkflows install fastq_align_bwa
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 3.2.0 - https://nf-co.re
DEBUG Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory') util.py:472
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=True) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-380ad3d9d31216b384383e94b.json utils.py:314
DEBUG Using config file: .nf-core.yml utils.py:1385
subworkflows
{'https://github.com/nf-core/modules.git': {'subworkflows': {'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}}}
{'subworkflows': {'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}}
Found match: subworkflows
{'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}
INFO Reinstalling subworkflows found in 'modules.json' but missing from directory: 'subworkflows/nf-core/bam_sort_stats_samtools', modules_json.py:1161
'subworkflows/nf-core/bam_stats_samtools', 'subworkflows/nf-core/fastq_align_bwa'
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/bam_sort_stats_samtools' - removing from modules.json modules_json.py:588
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/bam_stats_samtools' - removing from modules.json modules_json.py:588
DEBUG Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
ERROR The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git' synced_repo.py:318
WARNING Could not install module 'modules/nf-core/fastq_align_bwa' - removing from modules.json modules_json.py:588
INFO Was unable to reinstall some subworkflows. Removing 'modules.json' entries: 'nf-core/bam_sort_stats_samtools', 'nf-core/bam_stats_samtools', 'nf-core/fastq_align_bwa' modules_json.py:1182
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'cat-file', '--batch-check'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/fastq_align_bwa'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'cat-file', '--batch'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False) cmd.py:1253
INFO Installing 'fastq_align_bwa' install.py:131
DEBUG Installing subworkflows 'fastq_align_bwa' at modules hash a29f18660f5e3748d44d6f716241e70c942c065d from https://github.com/nf-core/modules.git install.py:132
DEBUG Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/bam_sort_stats_samtools'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Installing subworkflows 'bam_sort_stats_samtools' at modules hash 05954dab2ff481bcb999f24455da29a5828af08d from https://github.com/nf-core/modules.git install.py:132
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/bam_stats_samtools'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Installing subworkflows 'bam_stats_samtools' at modules hash 05954dab2ff481bcb999f24455da29a5828af08d from https://github.com/nf-core/modules.git install.py:132
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['bam_stats_samtools'] for module samtools/stats to 'modules.json' without installing the modules.
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['bam_stats_samtools'] for module samtools/idxstats to 'modules.json' without installing the modules.
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['bam_stats_samtools'] for module samtools/flagstat to 'modules.json' without installing the modules.
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['bam_sort_stats_samtools'] for module samtools/sort to 'modules.json' without installing the modules.
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['bam_sort_stats_samtools'] for module samtools/index to 'modules.json' without installing the modules.
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Module is already installed and force is not set. install.py:110
Adding the new installation source ['fastq_align_bwa'] for module bwa/mem to 'modules.json' without installing the modules.
DEBUG The following files were modified by prettier: lint_utils.py:111
modules.json
INFO Use the following statement to include this subworkflow: install.py:155
include { FASTQ_ALIGN_BWA } from '../subworkflows/nf-core/fastq_align_bwa/main'System information
Nextflow version: 24.10.5.5935
Hardware: HPC
Executor: Local
OS: Ubuntu server
Version of nf-core/tools: 3.2.0
Python version: 3.12.5