8000 Initial PR to dev with running pipeline by chris-cheshire · Pull Request #9 · nf-core/cutandrun · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Initial PR to dev with running pipeline #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 153 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
2f0127d
Initial file structuring and cleanup
chris-cheshire Jan 15, 2021
f950830
Pipeline runs
chris-cheshire Jan 15, 2021
e7813ed
sample sheet check implimented
chris-cheshire Jan 15, 2021
0242c40
Added fastq concatenation
chris-cheshire Jan 15, 2021
337403c
fastqc and trim galore running
chris-cheshire Jan 15, 2021
faa2461
Added bowtie
chris-cheshire Jan 16, 2021
9eeec41
added bowtie2 indexing and align
chris-cheshire Jan 18, 2021
fbafe01
Added more alignment framework
chris-cheshire Jan 18, 2021
c388726
preapre genome indexing complete
chris-cheshire Jan 19, 2021
e30ceb6
reenabled pipeline
chris-cheshire Jan 19, 2021
3ad8cda
aligner complete
chris-cheshire Jan 19, 2021
b6ba24d
Added metadata annotation for bt2
chris-cheshire Jan 20, 2021
601c49d
dedup via piccard added
chris-cheshire Jan 20, 2021
7d57231
Added qscore filtering
chris-cheshire Jan 21, 2021
96f7146
options update
chris-cheshire Jan 21, 2021
348ff66
Added scaled bedgraph creation
chris-cheshire Jan 21, 2021
417c737
SEACR peak calling implimented
chris-cheshire Jan 22, 2021
65635be
added bigwig conversion
chris-cheshire Jan 22, 2021
cb1eb79
Added igv prep
chris-cheshire Jan 22, 2021
7c5d05f
Corrected gtf issue
chris-cheshire Jan 22, 2021
c11b606
igv session complete
chris-cheshire Jan 22, 2021
78f3d14
tidy up
chris-cheshire Feb 1, 2021
c3ca3e8
Fixed channel name error
chris-cheshire Feb 1, 2021
5b1628b
Added multiqc support
chris-cheshire Feb 1, 2021
c6a9712
Added metadata export to csv file
chris-cheshire Feb 1, 2021
b75ce53
added test reporting module
chris-cheshire Feb 1, 2021
7d8629f
Added first plot gen
chris-cheshire Feb 2, 2021
a5fca7f
Added pdf export
chris-cheshire Feb 2, 2021
ae1ae13
Added dash support
chris-cheshire Feb 2, 2021
b2db491
Restructured python directories
chris-cheshire Mar 1, 2021
dae6fe5
Commenting
chris-cheshire Mar 1, 2021
f3f5386
Move dash to separate container
chris-cheshire Mar 1, 2021
ef36e1b
Added deseq2 dev
chris-cheshire Mar 1, 2021
96b8a32
Config changes
chris-cheshire Mar 1, 2021
9391c9c
ignore file update
chris-cheshire Mar 1, 2021
6984e24
add full path of python file now that its been moved
charlotte-west Mar 1, 2021
0f7c7ed
add full paths for where python files have been moved to
charlotte-west Mar 1, 2021
327500d
Merge remote-tracking branch 'origin/char-dev' into dev
chris-cheshire Mar 4, 2021
6368a79
Added reporting dev env
chris-cheshire Mar 5, 2021
35d7193
Ignoring generated test output
chris-cheshire Mar 5, 2021
ae312c6
Logging now working
chris-cheshire Mar 8, 2021
aaad92a
Added piccard metrics meta data parsing
chris-cheshire Mar 10, 2021
b7c7459
ignore updates
chris-cheshire Mar 16, 2021
dfc415d
updated ignore
chris-cheshire Mar 16, 2021
4b19981
Improved annotate meta
chris-cheshire Mar 16, 2021
653b01e
Improved export meta to deal with missing and out of order metadata i…
chris-cheshire Mar 16, 2021
5dadc47
Updated test data
chris-cheshire Mar 16, 2021
6cada60
Workflow minor changes
chris-cheshire Mar 16, 2021
51b1123
Duplication summary now renders
chris-cheshire Mar 16, 2021
a1aceab
Added fragment size analysis
chris-cheshire Mar 18, 2021
412aa2e
Added bamtobed conversion
chris-cheshire Mar 19, 2021
6e2554d
add blank line to end of file
charlotte-west Mar 22, 2021
99486f2
fix file_list bug
charlotte-west Mar 23, 2021
3d11ddc
remove merge error
charlotte-west Mar 23, 2021
ab38707
run deseq2 now formatted like nfcore pipelines
charlotte-west Mar 23, 2021
8dfaf71
add plotting section copied from nfcore
charlotte-west Mar 24, 2021
a616e90
Updated deseq 2 to working env
chris-cheshire Mar 24, 2021
e48f7a6
volcano plot added
charlotte-west Mar 24, 2021
a7903d0
tidy up
charlotte-west Mar 24, 2021
b8bc5bf
working conda env for deseq2
charlotte-west Mar 24, 2021
2d46bee
initialise new deseq2 module
charlotte-west Mar 24, 2021
67abdf8
add conda env and module skeleton
charlotte-west Mar 24, 2021
14124d4
view channel
charlotte-west Mar 24, 2021
0585c4c
deseq2 module progress
charlotte-west Mar 24, 2021
19b610a
port in atacseq pairwise comparisons
charlotte-west Mar 24, 2021
88f6cae
changing param input names and accounting for include/exclude lists
charlotte-west Mar 24, 2021
d32b793
edits to deseq
charlotte-west Mar 24, 2021
aafdee8
setting up deseq in pipeline
charlotte-west Mar 24, 2021
b7f6ad7
working deseq2 script
charlotte-west Mar 25, 2021
2925282
Merge pull request #1 from luslab/deseq2-update
charlotte-west Mar 25, 2021
79e5294
updates module for new deseq2 script parse format
charlotte-west Mar 25, 2021
982b59d
start organising channels for deseq2 input
charlotte-west Mar 25, 2021
8cfa20c
input structure change
charlotte-west Mar 25, 2021
4209357
include module in mainscript and view input channels
charlotte-west Mar 26, 2021
7b6c8fa
correct syntax
charlotte-west Mar 26, 2021
1e34dc5
correct syntax
charlotte-west Mar 26, 2021
9d2246c
renaming
charlotte-west Mar 26, 2021
f9cb85b
channels setup for deseq2
charlotte-west Mar 26, 2021
e0c51fa
extracting values may be this mapping
charlotte-west Mar 26, 2021
a4cf611
only have paths for files currently being emitted
charlotte-west Mar 26, 2021
e2bd77c
versioning for deseq added
charlotte-west Mar 26, 2021
b76e394
comment out length check, now redundant
charlotte-west Mar 26, 2021
e70aae0
update to deseq2
charlotte-west Apr 14, 2021
51a553a
deseq2 now working and producing plots
charlotte-west Apr 19, 2021
7b83e6c
typo
charlotte-west Apr 20, 2021
3d3bf7c
add file name to bin500 frags count files for sample identification
charlotte-west Apr 20, 2021
c4706f3
progress with adding reproducibility plot
charlotte-west Apr 20, 2021
4bc576e
more prog
charlotte-west Apr 20, 2021
7f197da
place holder for heatmap function
charlotte-west Apr 20, 2021
3fd7bfe
now have merged frag log2 counts dataframe
charlotte-west Apr 21, 2021
ccce2af
replicate reproducibility plot done
charlotte-west Apr 21, 2021
d170f5f
scale factor plot in
charlotte-west Apr 21, 2021
c213967
structure data load
charlotte-west Apr 22, 2021
cf56d12
add seacr_bed to required input
charlotte-west Apr 22, 2021
75cd6dd
added seacr_bed
charlotte-west Apr 22, 2021
44ba39c
made dataframe of seacr bed of all groups and replicates
charlotte-west Apr 22, 2021
ff90a86
first peak plot done
charlotte-west Apr 23, 2021
46b1dc5
added pyranges to env
charlotte-west Apr 26, 2021
c36b378
start and some fill to peaks reproduced plot
charlotte-west Apr 26, 2021
c1f31fe
7d peak reproduced rate figure done
charlotte-west Apr 27, 2021
38eb9d2
add bam reader to env
charlotte-west Apr 27, 2021
5c7201b
figure 7d of percentage frags in peaks done
charlotte-west Apr 27, 2021
36935d3
add new inputs and glob patterns for input files
charlotte-west Apr 28, 2021
f884095
add bams to required input
charlotte-west Apr 28, 2021
0c261b8
add channels to reporting module in main script
charlotte-west Apr 28, 2021
b2d161b
charting reports done and working in pipeline
charlotte-west Apr 28, 2021
1277617
Merge remote-tracking branch 'origin/dev' into char-deseq2
charlotte-west Apr 28, 2021
918919c
Merge pull request #2 from luslab/char-deseq2
charlotte-west Apr 28, 2021
5c8b2f0
indent syntax
charlotte-west Apr 28, 2021
9594eb9
added nf-core dt module tools computematrix and plotheatmap
charlotte-west Apr 28, 2021
1ddd2c1
setup for deeptools heatmap processes
charlotte-west Apr 28, 2021
fee6ae7
setup channels for matching samples in dt heatmap
charlotte-west Apr 29, 2021
7372f4f
set singularity_pull_docker_container to true
charlotte-west Apr 29, 2021
e573162
checks
charlotte-west Apr 29, 2021
6b957e0
update docker image
charlotte-west Apr 29, 2021
1ceb5e4
Merge remote-tracking branch 'origin/dev' into char-dt-heatmaps
charlotte-west Apr 29, 2021
966ac01
try different row mapping
charlotte-west Apr 29, 2021
c25316a
meta data and bw file mapping
charlotte-west Apr 29, 2021
2176d93
can't state meta.id variable in modules config
charlotte-west Apr 29, 2021
5bdb4c2
remove channel views
charlotte-west Apr 29, 2021
53045ed
edit profile configs
charlotte-west Apr 29, 2021
5244c9a
need to pass both bam and bai
charlotte-west Apr 29, 2021
8256549
cleanup comments
charlotte-west Apr 29, 2021
76d7041
move data manipulation into dataload instead of figure functions
charlotte-west Apr 30, 2021
a24e3c0
cover case for only 1 replicate
charlotte-west Apr 30, 2021
cecbc76
fixing charting plot1 titles
charlotte-west May 4, 2021
058cf63
edit frag violin df in order to edit plot
charlotte-west May 4, 2021
9772050
working syntax for pyranges count_overlaps and adding titles to plots
charlotte-west May 4, 2021
3137016
update how I run python reports script in container
charlotte-west May 5, 2021
447a7a6
Merge pull request #3 from luslab/char-dt-heatmaps
charlotte-west May 5, 2021
0f36a5d
Merge remote-tracking branch 'origin/dev' into char-dev
charlotte-west May 5, 2021
1fa3185
add process specific resources to overwrite nf-core label for process
charlotte-west May 5, 2021
07713fb
Added with name for deeptools
chris-cheshire May 6, 2021
683edf9
Lowered resources for local.config
chris-cheshire May 6, 2021
0a1322c
Added samtools custom view
chris-cheshire May 6, 2021
373d9d6
Changed histogram calculation method (unfinished)
chris-cheshire May 6, 2021
62212b5
intermediate work on charting
charlotte-west May 6, 2021
2cae543
Merge remote-tracking branch 'origin/chris-dev' into char-dev
charlotte-west May 7, 2021
beed2f2
typo txt should be tsv
charlotte-west May 7, 2021
2df7e6d
histogram (+violin) now working with new frag data
charlotte-west May 7, 2021
ff32783
coloured plots
charlotte-west May 7, 2021
bf414a0
include multiqc and get software versions
charlotte-west May 10, 2021
25b9457
set skip_multiqc to false as default
charlotte-west May 10, 2021
3949f28
add samtools sort process specific options
charlotte-west May 10, 2021
75d32c2
sort bam by query name
charlotte-west May 10, 2021
dfb95e0
use pysam for constructing frags
charlotte-west May 11, 2021
edc0797
fragments in peaks plot fixed
charlotte-west May 11, 2021
25adf41
ignore test data and outputs
charlotte-west May 11, 2021
81975ee
change from samtools sort workflow to just samtools sort process as b…
charlotte-west May 11, 2021
659ee04
add sorted suffix
charlotte-west May 11, 2021
d68b396
restructured with cutandrun.nf in workflows folder - working
charlotte-west May 11, 2021
7e15168
edit header to same as rnaseq
charlotte-west May 11, 2021
cdd1853
Merge pull request #4 from luslab/char-dev
charlotte-west May 11, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,29 @@ tests/
testing/
testing*
*.pyc

.local
.config
.bash_history

R_sessionInfo.log

dev/docker/static_reports/test_output/alignment_summary.csv
dev/docker/static_reports/test_output/alignment_summary.png
dev/docker/static_reports/test_output/report.pdf
dev/docker/static_reports/test_output/log.txt
dev/docker/static_reports/test_output/duplication_summary.csv

dev/docker/static_reports/test_output/duplication_summary.png

dev/docker/static_reports/test_output/frag_hist.csv

dev/docker/static_reports/test_output/frag_hist.png

dev/docker/static_reports/test_output/frag_violin.csv

dev/docker/static_reports/test_output/frag_violin.png

dev/docker/static_reports/test_data/tmp_dir/*

dev/docker/static_reports/test_output/*
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"python.pythonPath": "/usr/local/Caskroom/miniconda/base/envs/cutrun/bin/python"
}
35 changes: 35 additions & 0 deletions assets/awk/bt2_report_to_csv.awk
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/awk -f

# This awk script parses a bowtie2 alignment summary to a csv file for easier downstream processing

BEGIN {
cols="bt2_total_reads:bt2_align1:bt2_align_gt1:bt2_non_aligned:bt2_total_aligned"
FS="\n"
RS="\n\n"
col_count=split(cols, col_arr, ":");
for(i=1; i<=col_count; i++) printf col_arr[i] ((i==col_count) ? "\n" : ",");
}
{
for (i=1; i<=NF; i++) {
if(index($i,"reads; of these:") != 0) {
split($i, line_split, " ");
data["bt2_total_reads"]=line_split[1];
}
if(index($i,"aligned concordantly exactly 1 time") != 0) {
split($i, line_split, " ");
data["bt2_align1"]=line_split[1];
}
if(index($i,"aligned concordantly >1 times") != 0) {
split($i, line_split, " ");
data["bt2_align_gt1"]=line_split[1];
}
if(index($i,"aligned concordantly 0 times") != 0) {
split($i, line_split, " ");
data["bt2_non_aligned"]=line_split[1];
}
}
}
END {
data["bt2_total_aligned"] = data["bt2_align1"] + data["bt2_align_gt1"]
for (i=1; i<=col_count; i++) printf data[col_arr[i]] ((i==col_count) ? "\n" : ",");
}
28 changes: 28 additions & 0 deletions assets/awk/dt_frag_report_to_csv.awk
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/awk -f

BEGIN {
cols="dt_frag_sampled:dt_frag_mean_len:dt_frag_min_len:dt_frag_max_len"
FS="\t"
RS="\n"
col_count=split(cols, col_arr, ":");
for(i=1; i<=col_count; i++) printf col_arr[i] ((i==col_count) ? "\n" : ",");
}
{
for (i=1; i<=NF; i++) {
if(index($i,"Frag. Sampled") !=0) {
data["dt_frag_sampled"]=i;
}
if(index($i,"Frag. Len. Mean") !=0) {
data["dt_frag_mean_len"]=i;
}
if(index($i,"Frag. Len. Min") !=0) {
data["dt_frag_min_len"]=i;
}
if(index($i,"Frag. Len. Max") !=0) {
data["dt_frag_max_len"]=i;
}
}
}
END {
for (j=1; j<=col_count; j++) printf $(data[col_arr[j]]) ((j==col_count) ? "\n" : ",");
}
Loading
0