Add 'final reads for profiling' saving mechanism #272
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
TestsFor expecting:
TODO: Documentation 1✅ preprocessing Expect: in analysis_ready_fastqs only run merged files (3 total) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqsPASSED 2✅ preprocessing Expect: in analysis_ready_fastqs only host removed files (5 total) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging falsePASSED 3✅ preprocessing Expect: in analysis_ready_fastqs only complexity or filtlong (_filtered) files (5 files) bbduknextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_runmerging falsePASSED prinseqnextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_runmerging false --shortread_complexityfilter_tool prinseqplusplusPASSED 3✅ preprocessing Expect: in analysis_ready_fastqs only preprocessed reads (5 total - 8 if don't perform read collapsing) fastpExpect 5 nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging false --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter falsePASSED Expect 8(?) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging false --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter false --shortread_qc_mergepairs falsePASSED and adapterremovalExpect 5 nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging false --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter false --shortread_qc_tool 'adapterremoval'PASSED Expect 8 nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging false --perform_shortread_hostremoval false --perform_longread_hostremoval false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter false --shortread_qc_tool 'adapterremoval' --shortread_qc_mergepairs falsePASSED 4✅ preprocessing Expect: in analysis_ready_fastqs host removed reads (*unmapped) (5 files) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_runmerging false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter falsePASSED 5✅ preprocessing Expect: in analysis_ready_fastqs run merged reads (3 files) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter falsePASSED 6✅ preprocessing Expect: in analysis_ready_fastqs run merged reads (3) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter false --perform_shortread_hostremoval false --perform_longread_hostremoval falsePASSED 7❌ preprocessing Expect: in analysis_ready_fastqs run merged reads (5 - 2 samples, 2612 and 2612 unmerged) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_qc false --perform_longread_qc false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter falsePASSED 8❌ preprocessing Expect: in analysis_ready_fastqs run merged reads (4). No nanopore expected as part of preprocessing nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_qc false --perform_longread_qc false --perform_shortread_hostremoval false --perform_longread_hostremoval falsePASSED Missing ERR* from filtlong? 9❌ preprocessing Expect: in analysis_ready_fastqs run merged reads (5 - run-cat but not merged) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_qc false --perform_longread_qc falsePASSED 10❌ preprocessing Expect: in analysis_ready_fastqs run merged reads (expect only run merged reads, all other are all as user provided) nextflow run ../main.nf -profile singularity,test_noprofiling --input data/samplesheet.csv --databases data/database.csv --hostremoval_reference data/genome.fasta --outdir ./results -ansi-log false --publish_dir_mode 'symlink' -resume --save_analysis_ready_fastqs --perform_shortread_qc false --perform_longread_qc false --perform_shortread_complexityfilter false --longread_qc_skipqualityfilter false --perform_shortread_hostremoval false --perform_longread_hostremoval falsePASSED |
|
Note this is maybe not the most optimal implementation. E.g., one could calculate already at the beginning of a pipeline what is teh 'final' step, and assign a numeric ID in meta that is then evaluated by teh However this wasn't working initially when I tried, and would require having the logic in multiple places. So I opted for this more verbose way. |
sofstam
reviewed
Apr 17, 2023
jfy133
commented
Apr 20, 2023
Co-authored-by: Sofia Stamouli <91951607+sofstam@users.noreply.github.com>
|
CAT_FASTQ is not removed, just moved (further up in the config file :) ) |
jfy133
commented
Apr 20, 2023
sofstam
requested changes
Apr 21, 2023
Co-authored-by: Sofia Stamouli <91951607+sofstam@users.noreply.github.com>
sofstam
approved these changes
Apr 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Coming in two stages:
modules.conflogic to select the 'final' FASTQ that goes into profiling for publishing in--outdirCloses #262
PR checklist
nf-core lint).nextflow run . -profile test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).