fix: zero weight upload from trainer in hybrid mode #389
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
myungjin
reviewed
Apr 3, 2023
| @@ -109,8 +109,6 @@ def _upload_weights(self, tag: str) -> None: | |||
| # one aggregator is sufficient | |||
| end = channel.one_end() | |||
|
|
|||
| self._update_weights() | |||
There was a problem hiding this comment.
This is a nice catch! Thanks!
But this needs some additional change: I think it should be moved to the statement "if not self.can_ring_allreduce():".
If all reduce was executed, this self._update_weights() is problematic because it was already called.
If all reduce was not executed, removing this part means that prev_weights was not updated. To handle this case, I think self._update_weights() needs to be inside the if-statement.
There was a problem hiding this comment.
I see, you are correct! Should have catched this before I make a PR. Just updated the PR!
remove _update_weights() call from _upload_weights() function before calculating the delta weights, as such a function call makes self.prev_weights and self.weights equal, resulting in delta_weights to be all zero
171bfde to
8a57690
Compare
dhruvsgarg
pushed a commit
to dhruvsgarg/flame
that referenced
this pull request
Oct 18, 2024
remove _update_weights() call from _upload_weights() function before calculating the delta weights, as such a function call makes self.prev_weights and self.weights equal, resulting in delta_weights to be all zero
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
remove _update_weights() call from _upload_weights() function before calculating the delta weights, as such a function call makes self.prev_weights and self.weights equal, resulting in delta_weights to be all zero
Description
Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.
Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.
Type of Change
Checklist