8000 My SID and ASV results of the data2vec-base model are different from the benchmark · Issue #469 · s3prl/s3prl · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

My SID and ASV results of the data2vec-base model are different from the benchmark #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
qijieshao opened this issue Mar 14, 2023 · 8 comments
Open

My SID and ASV results of the data2vec-base model are different from the benchmark #469

qijieshao opened this issue Mar 14, 2023 · 8 comments

Comments

@qijieshao
Copy link

I trained a data2vec-base model myself, and then used the parameters in s3prl/s3prl/downstream/voxceleb1/config.yaml to reproduce the ASV and SID task in superb benchmark, but the results are very different. I trained the downstream task with single GPU.

Benchmark results:
SID: 70.21
ASV: 5.77

My results:
SID: 56.57
ASV: 6.77

Even if I use the official model from torch.hub, the downstream SID task results are not reproducible (only 54.85)

May I ask if you changed the parameters in the config.yaml file when training the downstream tasks of different pre-train models? If yes, where can I find the list of parameters you actually used?
@leo19941227
Copy link
Member

Hi,

Hmm this is really different.
While since the data2vec-base results are summited from the community (not from the original author or us), so I also don't have the exact hypers for that.
For SID, you might need to tune the learning rate hyper.
For ASV, I think the default config should work find.
So the inconsistency might come from that the current data2vec-base usage is different from the usage of the previous submitter.
How do you run the data2vec-base experiment? By -u data2vec?

@qijieshao
Copy link
Author

I have tried -u data2vec (result is 54.85) and -u data2vec_local (result is 56.57)

@leo19941227
Copy link
Member

I believe the default learning rate of SID is too small, please try 1.0e-2 and 1.0e-3, and the results can be closer.
But I guess they still won't be the same, since there are implementation inconsistency between ours and the people submitted to the leaderboard.
I might need more time to make sure which result is more reliable.
So for now, to be honest I not yet have a good answer, sorry for that.
I will revisit this issue when I have an answer later.

@qijieshao
Copy link
Author

Ok, I will try your suggestion later, thanks a lot!

I have another question, if I use DDP with multi-GPU, could I adjust some hyperparameters? To consistent with the benchmark, which hyperparameters are allowed to be adjusted in DDP training? Lr, total_steps, or any other?

@leo19941227
Copy link
Member

Sure,

I think only these two are adjustable.
Please note that, the batch size in the config file is for single process, so you need to adjust the batch size accordingly for multi-gpu setting.
I will suggest to let single process batch size * gpu num == default batch size under single gpu setting, since by default to align the results with the benchmark, the batch size is not recommended to be changed.

@qijieshao
Copy link
Author

Ok, I know, thank you!

@gancx
Copy link
gancx commented Dec 14, 2023

Hi,

Hmm this is really different. While since the data2vec-base results are summited from the community (not from the original author or us), so I also don't have the exact hypers for that. For SID, you might need to tune the learning rate hyper. For ASV, I think the default config should work find. So the inconsistency might come from that the current data2vec-base usage is different from the usage of the previous submitter. How do you run the data2vec-base experiment? By -u data2vec?

Hello. Thanks for your work. I mainly work on ASV. Actually, I have two questions. The results of EER reported in the paper is 6.02% by using a pre-trained wav2vec2 base model. I wonder if the training data is voxceleb1 dev or voxceleb 2 dev. Another question is that when I tried to reproduce the results (wav2vec2 base + x-vector), I found the accuracy was quite low, after several epochs, it is still below 10%. Can you share the training log for us to refer to if possible? Thank you.

@qijieshao
Copy link
Author
qijieshao commented Dec 14, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@leo19941227 @gancx @qijieshao
0