feature: hybrid mode with ring allreduce #170
Merged
+150
−72
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hybrid mode of combining distributed learning and federated learning is implemented. This implementation is based on PR cisco-open#131. The current implementation has a bug thta can cause a deadlock when trainers arrive late. Addressing the issue will be handled as a separate PR as it requires quite changes in backend, channel, channel manager, etc.
GaoxiangLuo
reviewed
Jul 8, 2022
| # | ||
| # non-committers send a dummy message so that the aggregator won't | ||
| # be blocked. | ||
| # TODO: figure out a way not to send a dummy message |
There was a problem hiding this comment.
Isn't line 105 already a way to send the dummy message?
There was a problem hiding this comment.
It's better if there is a way to skip a dummy message.
GaoxiangLuo
approved these changes
Jul 8, 2022
There was a problem hiding this comment.
lgtm. The implementation is consistent with the previous PR.
When I was testing it:
- If a trainer of
us/orgis already running with the aggregator, then when a new trainer ofus/XXXjoins, it will cause a deadlock. - However, when a new trainer of
eu/XXXjoins, it won't cause a deadlock if there are not existingeu/XXXtrainers.
In a word, a new trainer will only be hanging if there is an existing trainer of the same realm who may potentially form a ring with the new trainer. The new trainer is blocked on the stage of fetching weights from the aggregator. Hence, it may have something to do with how to distinguish aggregator-trainer channel and trainer-trainer channel. This will be resolved as a separate PR.
dhruvsgarg
pushed a commit
to dhruvsgarg/flame
that referenced
this pull request
Oct 18, 2024
Hybrid mode of combining distributed learning and federated learning is implemented. This implementation is based on PR cisco-open#131. The current implementation has a bug thta can cause a deadlock when trainers arrive late. Addressing the issue will be handled as a separate PR as it requires quite changes in backend, channel, channel manager, etc. Co-authored-by: Gaoxiang Luo <luo00042@umn.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hybrid mode of combining distributed learning and federated learning
is implemented. This implementation is based on PR #131.
The current implementation has a bug thta can cause a deadlock when
trainers arrive late. Addressing the issue will be handled as a
separate PR as it requires quite changes in backend, channel, channel
manager, etc.