Speed up the best-case scenario of dependency resolution. #37
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3a1ac2b to
a8841a4
Compare
A few small touchups that improve the speed of dependency resolution when
many packages are already in the local cache.
- when dispatching multiple crawling threads:
- make each thread a bit more reactive by reducing the polling timeout from
100ms to 10ms
- do not wait for each of the workers threads to complet, just wait for them to complete their
workload.
Without this change, dependency resolution is guaranteed to take longer than 100ms per
dependency, which is a large amount of time for just checking a local zipfile's content.
- cache the result of a couple calls that are repeated many times:
- Link.from_filename
- Package.from_href
Each of this call is performed for each file in the cache, for each dependency that is resolved.
While both these calls are not especially expensive, when we repeat them n^2 times in a largish
local cache * set of dependencies they do add up.
Somewhat unscientific benchmarking on my system show that the average time for resolving a single
dependency (namely 'pytz==2013b') goes down from 150ms to 30ms.
Running the modified code on the urbancompass codebase produced similarly desireable timings.
a8841a4 to
b660f1f
Compare
|
Only minor nit is that the package memoizer should be flushed each time we call .register, but this is purely pedantic. Will merge as-is. |
wickman
added a commit
that referenced
this pull request
Jan 30, 2015
Speed up the best-case scenario of dependency resolution.
|
This is awesome! Thanks @ugodiggi !! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A few small touchups that improve the speed of dependency resolution when
many packages are already in the local cache.
100ms to 10ms
workload.
Without this change, dependency resolution is guaranteed to take longer than 100ms per
dependency, which is a large amount of time for just checking a local zipfile's content.
Each of this call is performed for each file in the cache, for each dependency that is resolved.
While both these calls are not especially expensive, when we repeat them n^2 times in a largish
local cache * set of dependencies they do add up.
Somewhat unscientific benchmarking on my system show that the average time for resolving a single
dependency (namely 'pytz==2013b') goes down from 150ms to 30ms.
Running the modified code on the urbancompass codebase produced similarly desireable timings.