vine: make throttling worker transfer a tunable parameter #4101
Labels
Comments
|
Would it make sense to instead ignore transfer failures if they correspond to pruned files? If the correct behavior depends on finely tuning a parameter then we need to reconsider the approach. |
|
I previously didn't observe this error because the worker immediately crashes under that condition, only one error message was sent back. |
|
Got it! In general we shouldn't add a parameter to make something work. The general rule is to add a parameter for something that works acceptably well to make it work really well once we know something about the workflow. We do not want to add a parameter that would lead us to ignore a problem. |
|
That's a good rule to remember! Sounds like we need to find a proper solution, not a workaround! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In
@vine_current_transfers.c:set_throttle(link), we check whether the number of consecutive failures exceeds a threshold or if total failures outweigh successes. If either condition is met, we assume the worker is misbehaving and remove it.However, with file pruning enabled, workers transferring files may receive excessively frequent pruning requests, leading to a lot of transfer failures returning back to the manager, but these are as expected.
Therefore, I propose making the throttling behavior a tunable parameter, as it might be unnecessary in this scenario.
The text was updated successfully, but these errors were encountered: