8000 [FEATURE] speed improvement in OpenSwathAssayGenerator and OpenSwathDecoyGenerator by hroest · Pull Request #5611 · OpenMS/OpenMS · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[FEATURE] speed improvement in OpenSwathAssayGenerator and OpenSwathDecoyGenerator #5611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Oct 25, 2021
Merged

[FEATURE] speed improvement in OpenSwathAssayGenerator and OpenSwathDecoyGenerator #5611

merged 9 commits into from
Oct 25, 2021

Conversation

hroest
Copy link
Contributor
@hroest hroest commented Oct 23, 2021 •
edited
Loading

Substantial improvement in speed of the OpenSwathAssayGenerator

using the PanHuman library, before:

OpenSwathAssayGenerator took 10:55 m (wall), 10:52 m (CPU), 6.61 s (system), 10:45 m (user); Peak Memory Usage: 3502 MB.

after:

Loading /tmp/PanHuman_IM_negative1_targets.tsv  
Progress of 'Reading TSV File':  
done [took 10.37 s (CPU), 10.37 s (Wall)] -- 
Annotating transitions  
Progress of 'Annotating transitions': 
done [took 28.22 s (CPU), 28.22 s (Wall)] --
Annotating detecting transitions  
Progress of 'Restricting transitions':
done [took 2.
8000
90 s (CPU), 2.91 s (Wall)] --      
Progress of 'Select detecting transitions': 
-- done [took 2.08 s (CPU), 2.07 s (Wall)] --                                                                                                                                  Writing assays /tmp/test.pqp 
Progress of 'Writing out PQP file':
done [took 23.38 s (CPU), 23.63 s (Wall)] --
OpenSwathAssayGenerator took 01:16 m (wall), 01:16 m (CPU), 3.01 s (system), 01:13 m (user); Peak Memory Usage: 2847 MB.

Specifically writing of the PQP file is now 5x faster (23 seconds instead of 110 seconds) and selection of transitions is now 250x faster (2 seconds instead of 500 seconds). Overall it is 10x faster using the panhuman library.

@hroest hroest changed the title [FEATURE] speed improvement in OpenSwathAssayGenerator [FEATURE] speed improvement in OpenSwathAssayGenerator and OpenSwathDecoyGenerator Oct 23, 2021
@hroest
Copy link
Contributor Author
hroest commented Oct 23, 2021

the additional changesets also improve OpenSwathDecoyGenerator by about 2x and furthermore decrease the memory footprint of storing PQP files substantially

./bin/OpenSwathDecoyGenerator -in /tmp/PanHuman_IM_negative1_targets.tsv -out /tmp/PanHuman_IM_negative1_decoy_PRwsKR.pqp -
method pseudo-reverse -switchKR
OpenSwathDecoyGenerator took 02:12 m (wall), 02:11 m (CPU), 5.86 s (system), 02:05 m (user); Peak Memory Usage: 7936 MB.   

./bin/OpenSwathDecoyGenerator -in /tmp/PanHuman_IM_negative1_targets.tsv -out /tmp/PanHuman_IM_negative1_decoy_PRwsKR.pqp -method pseudo-reverse -switchKR true -enable_detection_specific_losses  -enable_detection_unspecific_losses
OpenSwathDecoyGenerator took 06:00 m (wall), 06:00 m (CPU), 6.05 s (system), 05:54 m (user); Peak Memory Usage: 7930 MB.

after this changeset:

/bin/OpenSwathDecoyGenerator -in /tmp/PanHuman_IM_negative1_targets.tsv -out /tmp/PanHuman_IM_negative1_decoy_PRwsKR.pqp -method pseudo-reverse -switchKR
OpenSwathDecoyGenerator took 01:38 m (wall), 01:36 m (CPU), 3.22 s (system), 01:33 m (user); Peak Memory Usage: 3092 MB.

./bin/OpenSwathDecoyGenerator -in /tmp/PanHuman_IM_negative1_targets.tsv -out /tmp/PanHuman_IM_negative1_decoy_PRwsKR.pqp -method pseudo-reverse -switchKR true -enable_detection_specific_losses  -enable_detection_unspecific_losses   
OpenSwathDecoyGenerator took 03:48 m (wall), 03:47 m (CPU), 3.34 s (system), 03:43 m (user); Peak Memory Usage: 3092 MB.

so, a substantial (2x) speedup and an almost 3x reduction in the memory footprint.

@hroest hroest force-pushed the feature/assay_generator branch from 604a0d7 to cb3244e Compare October 23, 2021 03:44
@timosachsenberg
Copy link
Contributor

rebuild jenkins

timosachsenberg
timosachsenberg approved these changes Oct 25, 2021
View reviewed changes
Copy link
Contributor
@timosachsenberg timosachsenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. Maybe the second part of the ifdef is not needed and could be removed

@timosachsenberg
Copy link
Contributor

failed test unrelated

@timosachsenberg timosachsenberg merged commit ad698b0 into OpenMS:develop Oct 25, 2021
@hroest hroest deleted the feature/assay_generator branch October 25, 2021 16:31
@hroest
Copy link
Contributor Author
hroest commented Oct 25, 2021

@timosachsenberg is there a good place to have these "helper" functions in OpenMS, eg in CONCEPT ? I tested both versions and the one we use was better in terms of memory consumption, but I should add the commented out reserve again...

@timosachsenberg
Copy link
Contributor

Yes maybe in concept would be good but I agree that it is hard to find good places for these kind of templated free functions. ContainerHelper or similar would work I guess.

@hroest hroest mentioned this pull request Nov 25, 2021
Merged
@timosachsenberg timosachsenberg mentioned this pull request Nov 25, 2021
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timosachsenberg timosachsenberg timosachsenberg approved these changes

Assignees
No one assigned
Labels
OpenSWATH Performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull 3961 request may close these issues.
2 participants
@hroest @timosachsenberg
0