Pre-trained models do not reproduce paper results #6
Comments
|
Hey! Sorry for bothering again. I've also made some movies with these models and they definitely do not correspond to what is shown in the official video of the paper -- maybe you didn't upload the final final models? |
|
The final models are here: https://drive.google.com/open?id=0B7lfjqylzqmMZlI3TUNUUEFQMXc (same link as above). The numbers is Table 1 are Euler angle errors, and not exponential map error. I think you are outputting exponential map errors. The model are trained on exponential map representation of joints, the output is then converted to Euler angle representation for visualization and quantitative comparison. |
|
You should look into the Utils directory. It has some Matlab scripts that the do the conversion for you (Sorry, Utils is not documented yet). |
|
Thanks a lot. I looked into the utils directory and found this motionGenerationError.m file that computes error with the conversion expmap->rotmat->euler. When I run this on the generated motion of pre-trained models, I get the following errors: erd walking [0.93 1.18 1.59 1.97 2.24 ]
lstm3lr walking [0.77 1.00 1.29 1.74 1.84 ]
srnn walking [0.81 0.94 1.16 1.48 1.78 ]
erd eating [1.27 1.45 1.66 1.95 2.02 ]
lstm3lr eating [0.89 1.09 1.35 1.66 1.97 ]
srnn eating [0.97 1.14 1.35 1.62 2.09 ]
erd smoking [1.66 1.95 2.35 2.63 3.61 ]
lstm3lr smoking [1.34 1.65 2.04 2.30 2.59 ]
srnn smoking [1.45 1.68 1.94 2.24 2.64 ]
erd discussion [2.27 2.47 2.68 2.92 3.16 ]
lstm3lr discussion [1.88 2.12 2.25 2.33 2.45 ]
srnn discussion [1.22 1.49 1.83 2.07 2.24 ]These results are a bit better than those reported in Table 1 :) -- Do you have an idea of what could be causing the discrepancy? I've noticed that the code ignores the global rotation and translation (e.g. motionGenerationError.m#L35 sets them to zero); I experimented with setting only the global rotation to zeros and I get slightly worse results, but still better than those in Table 1. However, if I completely comment that line (e.g., I add global rotation and global rotation), I get the following results: erd walking [4.69 11.71 33.38 62.81 106.05 ]
lstm3lr walking [4.30 10.24 29.06 51.01 83.57 ]
srnn walking [6.94 15.12 32.33 57.26 89.95 ]
erd eating [4.87 10.07 17.09 27.64 38.08 ]
lstm3lr eating [6.24 12.47 22.04 40.35 82.21 ]
srnn eating [5.05 9.32 14.75 23.29 35.91 ]
erd smoking [4.20 7.77 15.42 31.38 51.39 ]
lstm3lr smoking [3.75 7.26 14.21 21.88 31.83 ]
srnn smoking [4.44 8.15 14.34 22.08 31.76 ]
erd discussion [5.95 13.99 33.42 59.98 111.36 ]
lstm3lr discussion [9.55 20.88 42.32 62.46 82.69 ]
srnn discussion [9.40 19.81 39.03 59.83 99.46 ]Which are definitely much worse. Thanks again for getting back to me; we seem to be getting closer to reproducing the results in the paper. |
|
As a side note, I'm assuming that everything is at 25fps, right? Since you have 8x less data than what can be downloaded from human3.6m, and that is sampled at 200fps. Hence, in the error vector I'm using the indices |
|
The errors reported in table 1 only include the Euler angles, and does not include the global translation and rotation errors. This similar to Fragkiadaki et al. ICCV'15. We used mocap data at 100hz (down sampled by 2) and not at 25Hz. The reason you see less data is because we don't use all the data from human3.6. The details on the sequences we used can be found in the experiment section of the paper. Just to reiterate, our experiment settings (to the best of our effort) are very similar to Fragkiadaki et al. |
|
Hello @asheshjain399 , Thank you very much for releasing code and pre-trained models. I'm trying to reproduce the your results but i couldn't manage it. I see that you are normalizing the data, so prediction is also normalised. are you computing error over normalized data? or are you unnormalizing your prediction with data statistics. I used the motionGenerationError.m file to generate error, but it seems it is expecting the prediction vector should be 99 dimensional, but on the other hand code produces 54 dimensional vector. I modify motionGenerationError.m file to handle this but i'm not sure if that is correct way or not. Another thing is that i see that you are computing direct 2d L2 loss between each angles, shouldn't it be 3d loss between joints with this your error will be considerebly less. |
|
@una-dinosauria I am trying to reproduce your result by modifying motionGenerationError.m to calculate error from forecast_N_n and ground_truth_forecast_N_n, n in [0, 23]. lstm3lr walking [0.77 1.00 1.29 1.74 1.84 ] Note : I believe that this is the same value that appears in your paper (cvpr2017; on human motion prediction using RNN). However, I got the number that is very different from your result and the result in srrn paper. lstm3lr walking [7.7294, 8.7923, 8.7971, 9.2380, 9.1237] The only modification I done on motionGenerationError.m is
And other differences could be
Am I missing something here? e.g. unnormalized that data, consider only n in [0,7]. @Seleucia So you manage to solve your issue of reproducing the results from srnn paper using a pre-trained model? Thank you. |
|
I did manage to get the numbers that I reported, and I remember them being reasonably close to what the SRNN paper reports. Have you made movies for your predictions? If I remember correctly, the movie for discussion was exactly was is shown in the official SRNN movie, but I never managed to get the other ones. I'm currently away at a conference but if you make your branch public I can look at the code once I get back to the lab (and make a diff with my code to see if there's something noticeably different). |
|
Yes, i manage to get exactly same number given at the srnn paper with pre-trained models. I tthink confusion here is related with the subsampling. @una-dinosauria is right that given data here is 25ps not 100fps. |
|
@una-dinosauria No, I haven't made movies for the predictions yet. Let's me try a couple things on my own. If it is still not working still, I will ask you a favor for a diff. @Seleucia Do you make any change to the source code more than motionGenerationError.m as mentioned earlier (99->54)? Can you also elaborate more on the subsampling issue? Thank you. |
|
I did not make any changes except that I mention here. Subsampling issue was related with the selected frames, SRNN paper is reporting the frames: [8, 16, 32, 56, 100], not the one @una-dinosauria said: [2,4,8,14,25]. SRNN paper assumes that they subsampled by 2, on the other @una-dinosauria paper assuming that subsampled by 8. I think @una-dinosauria is right, given time at SRNN paper is wrong. |
|
@una-dinosauria I tried to duplicate your result as mentioned, but without success. Thank you. Here is what I did and my result: 1.) I use both srnn branch of both RNNexp (@3ba986b) and NeuralModels (@fb02335). 2.) Changes I made is to make the program run, and they can be found in patch_srnn.txt 3.) I download the data and pre-trained model, then forecase the motion with 4.) I calculate the error using matlab script (I actually use octave here, also I need to download H3.6m visualize code version 1.1 and extract it under RNNexp/structural_rnn/CRFProblems/H3.6m/h36devkit folder). lstm3lr walking Below is the error value for a 100 predicted frames.
|
|
@una-dinosauria I think I got it already. The output of the prediction from generateMotionForecast.py needed to be unnormalised before calculating the error with motionGenerationError.m. There is a unnormalised method provide in unNormalizeData.py, but you would need to modify the source code to do it yourself. For those who is following the thread, I will provide the patch once I clean my code. Thank you. |
|
As promised, please replace the following patch srnn_patch.txt in step 2.) I provided above. With this, you should be able to reproduce the same/similar result as the Structural-RNN for lstm3lr and erd case. lstm3lr walking erd walking Please note that if I used merr([2,4,8,14,25]) the different is slightly better than the one report in the baseline paper (cvpr2017; on human motion prediction using RNN). lstm3lr walking erd walking UPDATE (9 August 2017): |
Hi, I also try to reproduce the code. And I use motionGenerationError.m to convert the data expmap->rotmat->euler. But the result what I got is like that: srnn walking [4.57 5.12 5.95 6.04 7.43 ] (skel_err)
[0.10 0.11 0.12 0.13 0.15 ] (err_per_dof) It's different from your result srnn walking [0.81 0.94 1.16 1.48 1.78 ]And it is also different from results in the paper. So maybe I have a look at your reproduce code? I appreciate you so much. |
Uh oh!
There was an error while loading. Please reload this page.
Hi!
I'm using the pre-trained models available at https://drive.google.com/open?id=0B7lfjqylzqmMZlI3TUNUUEFQMXc and running generateMotionForecast.py. This produces motion predictions for different activities and models, but I've found that these do not correspond to what is reported in the paper. For reference, here's the figure from the paper that I'm talking about:
But, for example, using the lstm3lr_walking model, the
checkpoint.pikforecast_errorfile contains the following values:where the left and right columns correspond to
skel_erranderr_per_dofas computed in forecastTrajectories.py#L124I find one value to be much worse, and the other to be about 1 order of magnitude better. Do you have any pointers as to what I could be doing wrong?
The text was updated successfully, but these errors were encountered: