Open
Description
At this line, the comment seems to suggest that the intention is to exclude the class token (text_target_token_idx) from the text tokens. Since the class token is also excluded from the image tokens, it would be natural to exclude it from the text tokens as well for consistency.
However, based on the actual implementation in line 106: text_features = text_features[:, self.text_target_token_idx:, :]
def forward_text_feature(self, input_ids, attention_mask):
text_features = self.text_encoder(
input_ids=input_ids, attention_mask=attention_mask
)
# pre-pooling
if self.cfg.model.pool.name == "identity":
# cls token
text_features = text_features[:, self.text_target_token_idx, :].squeeze(dim=1)
else:
# word tokens <- is this right ?????
text_features = text_features[:, self.text_target_token_idx:, :]
return text_featuresit appears that the class token is not being excluded, since [: , :, :] is same with [:, 0:, :]
Could you clarify which behavior is intended?
Metadata
Metadata
Assignees
Labels
No labels