Benchmarking #118

Fabian-Boehm · 2024-05-08T10:07:02Z

Implementation of benchmarking and statistics based on the comparison of polyA selected RNA and total RNA

modules/local/benchmarking/bed_to_csv/main.nf

modules/local/benchmarking/bed_to_csv/templates/starts_bed.py

github-actions · 2024-06-18T13:11:23Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit c4f4b3c

+| ✅ 248 tests passed       |+
!| ❗   1 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-circrna_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-circrna_logo_light.png
files_exist - File found: docs/images/nf-core-circrna_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-circrna_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowCircrna.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.mirna_min_sample_percentage= 0.2
nextflow_config - Config default value correct: params.mirna_min_reads= 5
nextflow_config - Config default value correct: params.mirna_correlation= pearson
nextflow_config - Config default value correct: params.tools= circexplorer2
nextflow_config - Config default value correct: params.bsj_reads= 1
nextflow_config - Config default value correct: params.tool_filter= 1
nextflow_config - Config default value correct: params.save_intermediates= false
nextflow_config - Config default value correct: params.exon_boundary= 0
nextflow_config - Config default value correct: params.benchmarking= false
nextflow_config - Config default value correct: params.mirna_tools= miranda,targetscan
nextflow_config - Config default value correct: params.mirna_tool_filter= 1
nextflow_config - Config default value correct: params.sjdboverhang= 100
nextflow_config - Config default value correct: params.chimJunctionOverhangMin= 10
nextflow_config - Config default value correct: params.alignSJDBoverhangMin= 10
nextflow_config - Config default value correct: params.limitSjdbInsertNsj= 1000000
nextflow_config - Config default value correct: params.chimSegmentMin= 10
nextflow_config - Config default value correct: params.seglen= 25
nextflow_config - Config default value correct: params.min_intron= 20
nextflow_config - Config default value correct: params.max_intron= 1000000
nextflow_config - Config default value correct: params.min_map_len= 40
nextflow_config - Config default value correct: params.min_fusion_distance= 200
nextflow_config - Config default value correct: params.bootstrap_samples= 30
nextflow_config - Config default value correct: params.save_unaligned= false
nextflow_config - Config default value correct: params.save_reference= true
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.skip_trimming= false
nextflow_config - Config default value correct: params.save_trimmed= false
nextflow_config - Config default value correct: params.skip_fastqc= false
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 50
nextflow_config - Config default value correct: params.max_memory= 300.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (439 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: ci.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
modules_config - CAT_FASTQ found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_FAIDX found in conf/modules.config and Nextflow scripts.
modules_config - CLEAN_FASTA found in conf/modules.config and Nextflow scripts.
modules_config - GTFFILTER found in conf/modules.config and Nextflow scripts.
modules_config - SEQKIT_SPLIT found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - BWA_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - HISAT2_EXTRACTSPLICESITES found in conf/modules.config and Nextflow scripts.
modules_config - HISAT2_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - STAR_GENOMEGENERATE found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_BSJS found in conf/modules.config and Nextflow scripts.
modules_config - MASK_SCORES found in conf/modules.config and Nextflow scripts.
modules_config - CONCAT_TOOLS_PER_SAMPLE found in conf/modules.config and Nextflow scripts.
modules_config - COUNT_TOOLS found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_MIN_TOOLS found in conf/modules.config and Nextflow scripts.
modules_config - CONCAT_SAMPLES found in conf/modules.config and Nextflow scripts.
modules_config - EXTRACT_COUNTS found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_COUNTS_PER_TOOL found in conf/modules.config and Nextflow scripts.
modules_config - UPSET_SAMPLES found in conf/modules.config and Nextflow scripts.
modules_config - UPSET_ALL found in conf/modules.config and Nextflow scripts.
modules_config - FASTA_COMBINED found in conf/modules.config and Nextflow scripts.
modules_config - FASTA_PER_SAMPLE found in conf/modules.config and Nextflow scripts.
modules_config - FASTA_PER_SAMPLE_TOOL found in conf/modules.config and Nextflow scripts.
modules_config - ADD_BACKSPLICE found in conf/modules.config and Nextflow scripts.
modules_config - UNIFY_MIRANDA found in conf/modules.config and Nextflow scripts.
modules_config - UNIFY_TARGETSCAN found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_BINDINGSITES found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_TRANSCRIPTOME_GTFS found in conf/modules.config and Nextflow scripts.
modules_config - EXCLUDE_OVERLONG_TRANSCRIPTS found in conf/modules.config and Nextflow scripts.
modules_config - TRANSCRIPTOME found in conf/modules.config and Nextflow scripts.
modules_config - MARK_CIRCULAR found in conf/modules.config and Nextflow scripts.
modules_config - PSIRC_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - PSIRC_QUANT found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_TX2GENE found in conf/modules.config and Nextflow scripts.
modules_config - TXIMETA_TXIMETA found in conf/modules.config and Nextflow scripts.
modules_config - TXIMETA_TXIMPORT found in conf/modules.config and Nextflow scripts.
modules_config - MERGE_EXPERIMENTS found in conf/modules.config and Nextflow scripts.
modules_config - TARGETSCAN_DATABASE found in conf/modules.config and Nextflow scripts.
modules_config - TARGETSCAN found in conf/modules.config and Nextflow scripts.
modules_config - MIRANDA found in conf/modules.config and Nextflow scripts.
modules_config - MIRNA_TARGETS found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_BINDINGSITES found in conf/modules.config and Nextflow scripts.
modules_config - MAJORITY_VOTE found in conf/modules.config and Nextflow scripts.
modules_config - CIRCTEST_PREPARE found in conf/modules.config and Nextflow scripts.
modules_config - CIRCTEST_CIRCTEST found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-09-16 10:38:28

nictru

Good start :)

modules/local/benchmarking/average_tsv/templates/average.py

modules/local/benchmarking/location_plots/main.nf

modules/local/benchmarking/location_plots/templates/create_plots.py

modules/local/benchmarking/overlap_plot/main.nf

modules/local/benchmarking/average_tsv/main.nf

modules/local/benchmarking/location_plots/main.nf

modules/local/benchmarking/overlap_plot/main.nf

modules/local/benchmarking/png_json/main.nf

modules/local/benchmarking/seq_depth_plot/main.nf

…Pearson-correllation. Implemented collection of correlation values.

…on plots

…tion was calculated from the wrong files

…sometimes apear the wrong way round

…ecessary inputs after the merge

modules/local/benchmarking/location_plots/main.nf

modules/local/benchmarking/overlap_plot/main.nf

nictru · 2024-09-05T15:51:08Z

modules/local/benchmarking/overlap_plot/templates/create_plots.py

+def read_bed_file(file_path):
+    data = {'chromosome': [], 'start': [], 'strand': []}
+    with open(file_path, 'r') as file:
+        reader = csv.reader(file, delimiter='\t')


modules/local/benchmarking/png_json/main.nf

nictru · 2024-09-05T15:53:11Z

modules/local/benchmarking/seq_depth_plot/main.nf

+    conda "bioconda::matplotlib=3.5.1 bioconda::seaborn=0.11.2"
+    container 'https://depot.galaxyproject.org/singularity/bionumpy:0.2.12--pyha8f3691_0'


nextflow_schema.json

subworkflows/local/benchmarking.nf

nf-core-bot · 2024-12-04T18:52:29Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 2.14.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

lpantano · 2025-03-11T15:34:55Z

Hi @Fabian-Boehm , @nictru, is this still relevant? if so, maybe we can add to a project in this coming hackathon.

PS:we are doing spring cleaning , sorry for the pings

nictru · 2025-03-11T16:35:49Z

Fabian left my lab unfortunately, so this was stalled for some time now. But @mweyrich28 will take over soon. So it is not really a hackathon thing, but rather an ongoing project with funding.

lpantano · 2025-03-12T01:45:29Z

Thank you! I will make a note.

Fabian-Boehm requested a review from nictru May 8, 2024 10:07

nictru requested changes May 13, 2024

View reviewed changes

modules/local/benchmarking/bed_to_csv/main.nf Outdated Show resolved Hide resolved

modules/local/benchmarking/bed_to_csv/templates/starts_bed.py Outdated Show resolved Hide resolved

nictru linked an issue Jun 9, 2024 that may be closed by this pull request

Implement polyA-Benchmarking #143

Open

nictru requested changes Jul 15, 2024

View reviewed changes

nictru and others added 21 commits July 18, 2024 13:31

Replicate circRNA discovery for benchmarking

88931b2

Implement Jaccard calculation

56d6bd5

Implement jaccard file collection

31dc5e8

Add benchmarking multiqc configuration

650480c

Group benchmark multiqc sections

d506c8b

implemented location mapping

fac07a4

diagnostics: ch_joined not found, added files

a1a9594

Added Overlap location plots

9c1aae1

prepare seq_depth data, trimgalore reports in benchmarking.nf

31ff400

sequencing depth pearson correllation computation

1d036fb

config files added to git

bda9c59

Postmerge fixes

e0bc4f0

diagnostic: comparison to older version needed

7e6a944

Fixed benchmarking.nf input to be tool specific as previously. Fixed …

4a06426

…Pearson-correllation. Implemented collection of correlation values.

added sequence depth to mulitiqc. Changed labels and layout of locati…

931698d

…on plots

pre-commit run pass. Deleted redundant module

5bb7aa9

Added sections to multiqc, fixed an issue where the Seq-depth-correla…

1e12d20

…tion was calculated from the wrong files

Fixed an issue, where the Strands in the multiqc locationplots would …

0fda391

…sometimes apear the wrong way round

cosmetic cleanup of the benchmarking module.

b7cbd04

Fixed an issue where CIRCRNA_DISCOVERY_BENCHMARKING did not get all n…

3039a80

…ecessary inputs after the merge

readded missing module configs after rebase

4265950

Fabian-Boehm force-pushed the benchmarking branch from 8983c35 to 4265950 Compare July 18, 2024 12:57

Fabian-Boehm added 3 commits August 1, 2024 11:46

Ensured correct syntax, removed unnecessary specifications

1fefca7

Merge branch 'dev' into benchmarking after find_circ fix

9e1717e

removed debugging statements

74a243a

Fabian-Boehm and others added 5 commits August 19, 2024 15:07

added version control as per requested changes (15.07.24) and GENOMECOV

78e8c11

Merge remote-tracking branch 'origin/dev' into benchmarking

3e62a86

Minor adjustments for compatibility

f7bbcbb

Remove outdated module configs

c2b76e6

Remove some unnecessary imports

467e617

nictru requested changes Sep 5, 2024

View reviewed changes

nictru marked this pull request as ready for review September 5, 2024 15:58

Fabian-Boehm and others added 4 commits September 13, 2024 12:44

implemented requested changes from 5.9.24

a06a53a

fixed some issues that pervented termination

c4f4b3c

Template update for nf-core/tools version 3.0.2

8f1c8f8

added rRNA correlation analysis

94859e7

Fabian-Boehm added 2 commits December 4, 2024 21:05

Template update for nf-core/tools version 2.14.1

80c1628

Merged TEMPLATE

3f7b7e2

nictru marked this pull request as draft April 22, 2025 14:39

		conda "bioconda::matplotlib=3.5.1 bioconda::seaborn=0.11.2"
		container 'https://depot.galaxyproject.org/singularity/bionumpy:0.2.12--pyha8f3691_0'

Benchmarking #118

Are you sure you want to change the base?

Benchmarking #118

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

`nf-core lint` overall result: Passed ✅ ⚠️