8000 Add samplesheet generation for taxprofiler, mag by Joon-Klaps · Pull Request #39 · nf-core/detaxizer · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add samplesheet generation for taxprofiler, mag #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Oct 17, 2024
Merged

Add samplesheet generation for taxprofiler, mag #39

merged 18 commits into from
Oct 17, 2024

Conversation

Joon-Klaps
Copy link
Contributor
@Joon-Klaps Joon-Klaps commented Oct 10, 2024 •
edited
Loading

Closes #38 & #39

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/detaxizer branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@Joon-Klaps
Copy link
Contributor Author
Joon-Klaps commented Oct 10, 2024 •
edited
Loading

Note:

The default test profile outputs empty nanopore files which makes taxprofiler's Porechop crash.

log -profile test | taxprofiler 
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:UNTAR                                                                                       [  0%] 0 of 8
[dc/7bb954] NFCORE_TAXPROFILER:TAXPROFILER:FASTQC (test_paired-end_plus_long-reads_longReads)                                          [100%] 5 of 5 ✔
[0b/72b269] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_SINGLE (test_single-end_short)                [100%] 1 of 1 ✔
[01/7fb7f6] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_PAIRED (test_paired-end_plus_long-reads)      [100%] 2 of 2 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:FASTQC_PROCESSED                                                    -
[6c/c4aa69] NFC…E_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_single-end_long_longReads) [100%] 2 of 2, failed: 2 ✘
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_FILTERING:NANOQ                                             -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:FASTQC_PROCESSED                                                     -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREIL                                                               -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE                                                                   -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_SET                                                                     -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREILCURVESR                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK                                                   -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_BUILD                                                         [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_ALIGN                                                         -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_INDEX                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX                                                         [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_ALIGN                                                         -
Plus 35 more processes waiting for tasks…
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/taxprofiler] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_paired-end_plus_long-reads_longReads)'

Caused by:
  Process `NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_paired-end_plus_long-reads_longReads)` terminated with an error exit status (1)


Command executed:

  porechop_abi \
      --input test_paired-end_plus_long-reads_longReads_filtered.fastq.gz \
      --threads 2 \
       \
      --output test_paired-end_plus_long-reads_longReads_test_paired-end_plus_long-reads_porechop_abi.fastq.gz \
      | tee test_paired-end_plus_long-reads_longReads_test_paired-end_plus_long-reads_porechop_abi.log
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI":
      porechop_abi: $( porechop_abi --version )
  END_VERSIONS

Command exit status:
  1

Command output:
  
  Loading reads
  test_paired-end_plus_long-reads_longReads_filtered.fastq.gz

Command error:
  Traceback (most recent call last):
    File "/usr/local/bin/porechop_abi", line 11, in <module>
  
  Loading reads
  test_paired-end_plus_long-reads_longReads_filtered.fastq.gz
      sys.exit(main())
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/porechop_abi.py", line 63, in main
      reads, check_reads, read_type = load_reads(args.input,
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/porechop_abi.py", line 136, in load_reads
      reads, read_type = load_fasta_or_fastq(input_file_or_directory)
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/misc.py", line 114, in load_fasta_or_fastq
      file_type = get_sequence_file_type(filename)
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/misc.py", line 106, in get_sequence_file_type
      raise ValueError('File is neither FASTA or FASTQ')
  ValueError: File is neither FASTA or FASTQ

Work dir:
  /workspace/detaxizer/testing/work/9e/fe127faf0ee71dc856849d6b73507e

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

detaxizer -profile test_blastn doesn't but crashes at TAXPROFILER:NONPAREIL:NONPAREIL_CURVE

log -profile test_blastn | taxprofiler 
cc/a04a1e] NFCORE_TAXPROFILER:TAXPROFILER:UNTAR (test-db-centrifuge.tar.gz)                                                            [100%] 8 of 8 ✔
[b4/d62ad8] NFCORE_TAXPROFILER:TAXPROFILER:FASTQC (test_paired-end_plus_long-reads_longReads)                                           [100%] 5 of 5 ✔
[0b/6e35f0] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_SINGLE (test_single-end_short)                 [100%] 1 of 1 ✔
[23/b28ca8] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_PAIRED (test_paired-end_plus_long-reads)       [100%] 2 of 2 ✔
[93/f63330] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:FASTQC_PROCESSED (test_single-end_short)                             [100%] 3 of 3 ✔
[d5/75d96c] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_single-end_long_longReads) [100%] 2 of 2 ✔
[84/9469f9] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_FILTERING:NANOQ (test_single-end_long_longReads)             [100%] 2 of 2 ✔
[95/97cbfe] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:FASTQC_PROCESSED (test_single-end_long_longReads)                     [100%] 2 of 2 ✔
[dc/4c58dc] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREIL (test_single-end_short)                                        [100%] 3 of 3 ✔
[25/579fab] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end)                                                  [100%] 2 of 2, failed: 2
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_SET                                                                      [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREILCURVESR                                                         [  0%] 0 of 1
[dc/9bde9f] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK (test_single-end_short)                            [100%] 3 of 3 ✔
[4d/1d618c] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_BUILD (genome.fasta)                                           [100%] 1 of 1 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_ALIGN                                                          [  0%] 0 of 3
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_INDEX                                                         -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS                                                         -
[af/4afeca] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX                                                          [100%] 1 of 1 ✔
[95/9819b6] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_ALIGN (test_single-end_long_longReads)                         [100%] 2 of 2 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:MULTIQC                                                                                      [  0%] 0 of 1
Plus 34 more processes waiting for tasks…
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/taxprofiler] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end_plus_long-reads)'

Caused by:
  Missing output file(s) `*.png` expected by process `NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end_plus_long-reads)`


Command executed:

  #!/usr/bin/env Rscript
  library(Nonpareil)
  
  png(file='test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.png')
  Nonpareil.curve('test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.npo')
  dev.off()
  
  version_file_path <- "versions.yml"
  version_nonpareil <- paste(unlist(packageVersion("Nonpareil")), collapse = ".")
  f <- file(version_file_path, "w")
  writeLines('"NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE":', f)
  writeLines("    nonpareil: ", f, sep = "")
  writeLines(version_nonpareil, f)
  close(f)

Command exit status:
  0

Command output:
  null device 
            1 

Command error:
  Warning message:
  In Nonpareil.curve("test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.npo") :
    Insufficient resolution below 90% coverage, check parameters and re-run (e.g., increase the value of -d in nonpareil).

Work dir:
  /workspace/detaxizer/testing/work/54/03c55ab469331fb1edf02b8b2754c4

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

*Edit, adding logs & different profiles

@Joon-Klaps
Copy link
Contributor Author

Linting with tools 2.14.1 results in 0 errors.

$ nf-core lint 
                                      ,--./,-.
      ___     __   __   __   ___     /,-._.--~\
|\ | |__  __ /  ` /  \ |__) |__         }  {
| \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                      `._,._,'

nf-core/tools version 2.14.1 - https://nf-co.re
There is a new version of nf-core/tools available! (3.0.1)

INFO Testing pipeline: .

╭─ [?] 2 Pipeline Tests Ignored ───────────────────────────────────────────────────╮
│                                                                                  │
│ files_unchanged: File ignored due to lint config: .github/CONTRIBUTING.md        │
│ files_unchanged: File ignored due to lint config:                                │
│ .github/PULL_REQUEST_TEMPLATE.md                                                 │
│                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─ [!] 3 Pipeline Test Warnings ───────────────────────────────────────────────────╮
│                                                                                  │
│ pipeline_todos: TODO string in awsfulltest.yml: You can customise AWS full       │
│ pipeline tests as required                                                       │
│ schema_lint: Schema $schema should be http://json-schema.org/draft-07/schema     │
│ Found https://json-schema.org/draft-07/schema                                    │
│ schema_description: No description provided in schema for parameter:             │
│ saveReference                                                                    │
│                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯

╭─ [!] 8 Module Test Warnings ─────────────────────────────────────────────────────╮
│                             ╷                         ╷                          │
│ Module name                 │ File path               │ Test message             │
│╶────────────────────────────┼─────────────────────────┼─────────────────────────╴│
│ kraken2preparation          │ modules/local/kraken2p… │ 'meta' map not emitted   │
│                             │                         │ in output channel(s)     │
│ bbmap/bbduk                 │ modules/nf-core/bbmap/… │ New version available    │
│ blast/blastn                │ modules/nf-core/blast/… │ New version available    │
│ blast/makeblastdb           │ modules/nf-core/blast/… │ New version available    │
│ fastp                       │ modules/nf-core/fastp   │ New version available    │
│ fastqc                      │ modules/nf-core/fastqc  │ New version available    │
│ kraken2/kraken2             │ modules/nf-core/kraken… │ New version available    │
│ multiqc                     │ modules/nf-core/multiqc │ New version available    │
│                             ╵                         ╵                          │
╰──────────────────────────────────────────────────────────────────────────────────╯

╭─ [!] 3 Subworkflow Test Warnings ────────────────────────────────────────────────╮
│                            ╷                          ╷                          │
│ Subworkflow name           │ File path                │ Test message             │
│╶───────────────────────────┼──────────────────────────┼─────────────────────────╴│
│ utils_nextflow_pipeline    │ subworkflows/nf-core/ut… │ New version available    │
│ utils_nfcore_pipeline      │ subworkflows/nf-core/ut… │ New version available    │
│ utils_nfvalidation_plugin  │ subworkflows/nf-core/ut… │ New version available    │
│                            ╵                          ╵                          │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────╮
│ LINT RESULTS SUMMARY  │
├───────────────────────┤
│ [✔] 487 Tests Passed  │
│ [?]   2 Tests Ignored │
│ [!]  14 Test Warnings │
│ [✗]   0 Tests Failed  │
╰───────────────────────╯

@Joon-Klaps Joon-Klaps linked an issue Oct 10, 2024 that may be closed by this pull request
Add samplesheet generation for Taxprofiler #38
Closed
@jfy133 jfy133 mentioned this pull request Oct 10, 2024
Closed
@Joon-Klaps
Copy link
Contributor Author

✅ It runs with no erros when the input sheet is

sample,short_reads_fastq_1,short_reads_fastq_2,long_reads_fastq_1
ERR5766180,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474936_ERR5766180_1.fastq.gz,,
ERR5766176,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_2.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERR3201952.fastq.gz
ERR5766176_B,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_B_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_B_2.fastq.gz,
ERR3201952,,,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERR3201952.fastq.gz

detaxizer:

nextflow run . -profile docker,test_blastn --filter_trimmed false --outdir ./results --input ./testing/samplesheet.csv -resume

taxprofiler:

mkdir -p testing & cd testing;
nextflow run nf-core/taxprofiler --input ../results/downstream_samplesheet/taxprofiler.csv -profile docker,test --outdir ./results

@Joon-Klaps Joon-Klaps linked an issue Oct 14, 2024 that may be closed by this pull request
Add samplesheet generation for mag #41
Closed
@Joon-Klaps Joon-Klaps changed the title Add samplesheet generation for taxprofiler Add samplesheet generation for taxprofiler, mag Oct 14, 2024
@jannikseidelQBiC
Copy link
Collaborator

I start with the template update 3.0.2. Might be that some merge conflicts arise when it is merged.

jfy133
jfy133 reviewed Oct 14, 2024
View reviewed changes
jfy133
jfy133 reviewed Oct 14, 2024
View reviewed changes
@Joon-Klaps Joon-Klaps requested a review from jfy133 October 15, 2024 07:43
jfy133
jfy133 reviewed Oct 15, 2024
View reviewed changes
Copy link
Member
@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Code and tests look good 👍

I just strongly feel not to produce a 'faulty' samplesheet with the included long-read only lines, and these should be removed.

So I think once this is added (and the minor coulple of other things), we are then good to go :)

@Joon-Klaps Joon-Klaps requested a review from jfy133 October 15, 2024 15:57
jfy133
jfy133 approved these changes Oct 16, 2024
View reviewed changes
Copy link
Member
@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now! Just need the merging in of the latest dev, and I thin kwe are good to go!

@Joon-Klaps Joon-Klaps merged commit d38e101 into dev Oct 17, 2024
9 checks passed
@jfy133
Copy link
Member
jfy133 commented Oct 17, 2024

It might have been nice for @jannikseidelQBiC to check this first as it's his pipeline but I guess he can check on master <- dev :)

@Joon-Klaps
Copy link
Contributor Author

It might have been nice for @jannikseidelQBiC to check this first as it's his pipeline but I guess he can check on master <- dev :)

I checked with him on slack he gave me the 👍

This was referenced Oct 17, 2024
Closed
Closed
@jfy133
Copy link
Member
jfy133 commented Oct 18, 2024

Ah ok, perfect! Also works!

@Joon-Klaps Joon-Klaps mentioned this pull request Oct 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jfy133 jfy133 jfy133 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add samplesheet generation for mag Add samplesheet generation for Taxprofiler
3 participants
@Joon-Klaps @jannikseidelQBiC @jfy133
0