8000 Evaluation performance on GEBD validation set · Issue #5 · MCG-NJU/DDM · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Evaluation performance on GEBD validation set #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHu 8000 b”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
SJTUwxz opened this issue Sep 26, 2022 · 7 comments
Open

Evaluation performance on GEBD validation set #5

SJTUwxz opened this issue Sep 26, 2022 · 7 comments

Comments

@SJTUwxz
Copy link
SJTUwxz commented Sep 26, 2022

Hi, this project is great and thanks for releasing the code!
I've re-trained DMM and the evaluation result on GEBD val set is as follows, which is around 2% lower than the reported result.

+GEBD Performance on Kinetics-GEBD----+--------+--------+--------+--------+--------+--------+--------+--------+
| Rel.Dis. | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.45 | 0.50 | Avg |
+----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
| F1 | 0.7447 | 0.8252 | 0.8496 | 0.8615 | 0.8679 | 0.8722 | 0.8750 | 0.8774 | 0.8796 | 0.8817 | 0.8535 |
+----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+

I've also tried loading the trained weights you've released and run the evaluation again, the result is still around 2% lower, which is,

+GEBD Performance on Kinetics-GEBD----+--------+--------+--------+--------+--------+--------+--------+--------+
| Rel.Dis. | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.45 | 0.50 | Avg |
+----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
| F1 | 0.7462 | 0.8234 | 0.8462 | 0.8578 | 0.8642 | 0.8684 | 0.8715 | 0.8739 | 0.8758 | 0.8776 | 0.8505 |
+----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+.

I would really appreciate it if you could provide any insights on possible reasons of this. Thanks a lot!
@JackyTown
Copy link
Collaborator
JackyTown commented Sep 26, 2022 •
edited
Loading

Maybe you can follow https://github.com/MCG-NJU/DDM/blob/main/GUIDE.md and find out the reason.

I guess the problem may be frame extraction or pickle preparation.

@SJTUwxz
Copy link
Author
SJTUwxz commented Sep 27, 2022

Thanks for quick response! Yeah I followed that guide and ignore videos that I couldn't download from Youtube anymore. I will take a closer look and check again.
I have another two questions:

  1. Are the "multi-frames-GEBD-train-5.pkl" file and also val file released? I know that these pickle files are generated in data loading, so I just want to see if I can compare mine with yours and see if that's causing the trouble.
  2. It seems that in the data loading, the number of frames of the clip centering query frame is only 10 (T=10), 5 frames before the center frame and 5 frames after the center frame, where the center frame is not loaded in the sequence. This is causing inconsistency because the positional embeddings are initialized with num_locations = 11. I've changed this part of the original code to put center frame back to the sequence because in the paper it says 11 frames are loaded for each clip.

Thanks again!

@Jo-won
Copy link
Jo-won commented Feb 20, 2023

Hi. I'm also experiencing the same problem with @SJTUwxz regarding reimplementation, so could you (@JackyTown) release the "multi-frames-GEBD-train-5.pkl" and "multi-frames-GEBD-val-5.pkl" you used for troubleshooting?

@JackyTown
Copy link
Collaborator
JackyTown commented Mar 6, 2023 •
edited
Loading

Have you tried this in generate_pickle.py?

-shift = np.arange(-frame_per_side, frame_per_side)
-shift[shift >= 0] += 1
+shift = np.arange(-frame_per_side, frame_per_side + 1)

@Ziwei-Zheng
Copy link

Hi. I also have trouble with the reimplementation. Same to @SJTUwxz, I got a 2% performance drop on the validation set with both the model trained by myself and the provided pre-trained one. I wonder if you can release the "multi-frames-GEBD-val-5.pkl" so that I could check out the possible reasons during validation.

Thanks!

@ForeverPs
Copy link
ForeverPs commented Feb 28, 2024 •
edited
Loading

Hello, I encounter some difficulties in preparing the Kinetics-GEBD data, could you give me some detailed help? And thanks for your excellent work!

Solved, thanks!

@Loewen-Hob
Copy link
Loewen-Hob commented Mar 19, 2024 •
edited
Loading

Thanks for quick response! Yeah I followed that guide and ignore videos that I couldn't download from Youtube anymore. I will take a closer look and check again. I have another two questions:

  1. Are the "multi-frames-GEBD-train-5.pkl" file and also val file released? I know that these pickle files are generated in data loading, so I just want to see if I can compare mine with yours and see if that's causing the trouble.
  2. It seems that in the data loading, the number of frames of the clip centering query frame is only 10 (T=10), 5 frames before the center frame and 5 frames after the center frame, where the center frame is not loaded in the sequence. This is causing inconsistency because the positional embeddings are initialized with num_locations = 11. I've changed this part of the original code to put center frame back to the sequence because in the paper it says 11 frames are loaded for each clip.

Thanks again!

Hello! @SJTUwxz
I also encountered the second issue you mentioned, but I haven't figured out how to put the center frame back to the sequence. Can you tell me which folder it is in

你在generate_pickle.py中尝试过这个吗？
-shift = np.arange（-frame_per_side，frame_per_side）
-shift[shift >= 0] += 1
+shift = np.arange（-frame_per_side，frame_per_side + 1）

I solved this problem. I found that the above modifications were not made in the file DDM-Net/datasets/MultiFDataset.py. my question!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@SJTUwxz @JackyTown @Ziwei-Zheng @Jo-won @ForeverPs @Loewen-Hob
0