8000 Error when reinstalling subworkflows from modules.json via resolving missing modules from modules.json · Issue #3526 · nf-core/tools · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Error when reinstalling subworkflows from modules.json via resolving missing modules from modules.json #3526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
thomas-tams opened this issue Apr 2, 2025 · 0 comments
Open

Error when reinstalling subworkflows from modules.json via resolving missing modules from modules.json #3526

thomas-tams opened this issue Apr 2, 2025 · 0 comments
Assignees
@mirpedrol @edmundmiller
Labels
bug Something isn't working

Comments

@thomas-tams
Copy link
thomas-tams commented Apr 2, 2025 •
edited
Loading

Description of the bug

When I try to reinstall a subworkflow or install another subworkflow/module after having deleted previously installed subworkflows in a pipeline, it seems like modules_json.py is not able install the missing subworkflows from parsing the data in the modules.json file.

I get errors like

DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/bam_sort_stats_samtools' - removing from modules.json                                                                         modules_json.py:588
DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/bam_stats_samtools' - removing from modules.json                                                                              modules_json.py:588
DEBUG    Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/fastq_align_bwa' - removing from modules.json                                                                                 modules_json.py:588
INFO     Was unable to reinstall some subworkflows. Removing 'modules.json' entries: 'nf-core/bam_sort_stats_samtools', 'nf-core/bam_stats_samtools', 'nf-core/fastq_align_bwa' modules_json.py:1182

Here it seems like modules_json.py is pointing to the wrong path for subworkflows. It points to modules/<organization>/<subworkflow_name>, when it should be pointing to subworkflows/<organization>/<subworkflow_name>.

I think I identified the cause of this:

When self.reinstall_repo() is called in this line of modules_json.py, we do not carry any information about component_type into the method call.

The self.install_repo() method just uses the self.modules_dir / install_dir as the path to component when calling modules_repo.install_component() see this line. This leads to the behaviour of incorrectly pointing to modules/ directory for subworkflows.

Command used and terminal output

$ nf-core --verbose --log-file test_log.txt subworkflows install fastq_align_bwa


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.2.0 - https://nf-co.re


DEBUG    Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')                                                                                  util.py:472
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                                      cmd.py:1253
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=True)                     cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                       cmd.py:1253
DEBUG    Got '/home/usr_name/.config/nfcore/nf-core/modules' as path        
8000
                                                                                                                 utils.py:280
DEBUG    Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                                                                      utils.py:314
DEBUG    Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml                                                                                                      utils.py:1385
DEBUG    Got '.' as path                                                                                                                                                                utils.py:280
DEBUG    Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-380ad3d9d31216b384383e94b.json                                                                      utils.py:314
DEBUG    Using config file: .nf-core.yml                                                                                                                                               utils.py:1385
subworkflows
{'https://github.com/nf-core/modules.git': {'subworkflows': {'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}}}
{'subworkflows': {'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}}
Found match: subworkflows
{'nf-core': {'bam_sort_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['fastq_align_bwa']}, 'bam_stats_samtools': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['bam_sort_stats_samtools']}, 'fastq_align_bwa': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['subworkflows']}}}
INFO     Reinstalling subworkflows found in 'modules.json' but missing from directory: 'subworkflows/nf-core/bam_sort_stats_samtools',                                          modules_json.py:1161
         'subworkflows/nf-core/bam_stats_samtools', 'subworkflows/nf-core/fastq_align_bwa'                                                                                                          
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                       cmd.py:1253
DEBUG    Got '/home/usr_name/.config/nfcore/nf-core/modules' as path                                                                                                                         utils.py:280
DEBUG    Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                                                                      utils.py:314
DEBUG    Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml                                                                                                      utils.py:1385
DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/bam_sort_stats_samtools' - removing from modules.json                                                                         modules_json.py:588
DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/bam_stats_samtools' - removing from modules.json                                                                              modules_json.py:588
DEBUG    Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
ERROR    The requested module does not exists in the branch 'master' of https://github.com/nf-core/modules.git'                                                                   synced_repo.py:318
WARNING  Could not install module 'modules/nf-core/fastq_align_bwa' - removing from modules.json                                                                                 modules_json.py:588
INFO     Was unable to reinstall some subworkflows. Removing 'modules.json' entries: 'nf-core/bam_sort_stats_samtools', 'nf-core/bam_stats_samtools', 'nf-core/fastq_align_bwa' modules_json.py:1182
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'cat-file', '--batch-check'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False)                          cmd.py:1253
DEBUG    Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/fastq_align_bwa'],                                           cmd.py:1253
         cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                                                                           
DEBUG    Popen(['git', 'cat-file', '--batch'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False)                                cmd.py:1253
INFO     Installing 'fastq_align_bwa'                                                                                                                                                 install.py:131
DEBUG    Installing subworkflows 'fastq_align_bwa' at modules hash a29f18660f5e3748d44d6f716241e70c942c065d from https://github.com/nf-core/modules.git                               install.py:132
DEBUG    Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/bam_sort_stats_samtools'],                                   cmd.py:1253
         cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                                                                           
DEBUG    Installing subworkflows 'bam_sort_stats_samtools' at modules hash 05954dab2ff481bcb999f24455da29a5828af08d from https://github.com/nf-core/modules.git                       install.py:132
DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'rev-list', '--max-count=1', '9fe7867a4f012db581d7bd560228ded1074eb0e5', '--', 'subworkflows/nf-core/bam_stats_samtools'],                                        cmd.py:1253
         cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                                                                           
DEBUG    Installing subworkflows 'bam_stats_samtools' at modules hash 05954dab2ff481bcb999f24455da29a5828af08d from https://github.com/nf-core/modules.git                            install.py:132
DEBUG    Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)         cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['bam_stats_samtools'] for module samtools/stats to 'modules.json' without installing the modules.                                                      
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['bam_stats_samtools'] for module samtools/idxstats to 'modules.json' without installing the modules.                                                   
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['bam_stats_samtools'] for module samtools/flagstat to 'modules.json' without installing the modules.                                                   
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['bam_sort_stats_samtools'] for module samtools/sort to 'modules.json' without installing the modules.                                                  
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['bam_sort_stats_samtools'] for module samtools/index to 'modules.json' without installing the modules.                                                 
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                                           cmd.py:1253
DEBUG    Module is already installed and force is not set.                                                                                                                            install.py:110
         Adding the new installation source ['fastq_align_bwa'] for module bwa/mem to 'modules.json' without installing the modules.                                                                
DEBUG    The following files were modified by prettier:                                                                                                                            lint_utils.py:111
                                                                                                                                                                                                    
         modules.json                                                                                                                                                                               
                                                                                                                                                                                                    
                                                                                                                                                                                                    
INFO     Use the following statement to include this subworkflow:                                                                                                                     install.py:155
                                                                                                                                                                                                    
 include { FASTQ_ALIGN_BWA } from '../subworkflows/nf-core/fastq_align_bwa/main'

System information

Nextflow version: 24.10.5.5935
Hardware: HPC
Executor: Local
OS: Ubuntu server
Version of nf-core/tools: 3.2.0
Python version: 3.12.5
@thomas-tams thomas-tams added the bug Something isn't working label Apr 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mirpedrol mirpedrol

@edmundmiller edmundmiller

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mirpedrol @edmundmiller @thomas-tams
0