Resize T5 Vocab #5497
Resize T5 Vocab #5497
Conversation
|
Thank you! @dirkgr |
|
|
||
| # resize lm head | ||
| old_size = self.lm_head.out_features | ||
| if old_size == new_size: |
There was a problem hiding this comment.
Minor thing: Maybe we can do this check first thing? It'll avoid the 2 empty calls.
There was a problem hiding this comment.
Or perhaps not? We want to resize the embedding dim and the output dim to the same new_size. Will the old embedding_dim and the old lm_head.out_features be different?
There was a problem hiding this comment.
It's a bit of defensive programming. It should never happen, but if the three places where the size of the token embedding matters get out of sync, this call will sync them all back up.
|
|
||
| # resize lm head | ||
| old_size = self.lm_head.out_features | ||
| if old_size == new_size: |
There was a problem hiding this comment.
It's a bit of defensive programming. It should never happen, but if the three places where the size of the token embedding matters get out of sync, this call will sync them all back up.
|
I also want to point out that I managed to push this and it passed all CI checks the first time 😁. |
CC @HarshTrivedi