some questions about the results #23

Fruit0218 · 2024-03-18T09:07:37Z

thanks for ur code!These days i tryed to run ur code with Non-iid MNIST dataset. However,my ultimate results didn't match that presented in ur paper.I'm wondering why this happened.It seems that FedAvg did better than pFedAvg.I did try different model to train that one,apparently,it didn't work.Maybe i need some suggestions for that.Sorry for bothering!

kwunhang · 2024-03-27T09:00:46Z

May I know more about the details?

Fruit0218 · 2024-08-28T15:28:46Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

kwunhang · 2024-09-03T03:07:02Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

I remember the niid data generation of CIFAR10 contains some problems. But I remember the MNIST dataset should be ok. Cause I also run the PFedMe code in MNIST dataset and get similar result to the paper.

May I know what changes you have made to code? Or you just self implemented the PFedMe code once?

Fruit0218 · 2024-09-07T08:55:59Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

I remember the niid data generation of CIFAR10 contains some problems. But I remember the MNIST dataset should be ok. Cause I also run the PFedMe code in MNIST dataset and get similar result to the paper.

May I know what changes you have made to code? Or you just self implemented the PFedMe code once?

Hi, I'm pleased to receive your response. I haven't made any changes to the algorithmic process of the author's pFedMe code, such as the custom pFedMeOptimizer used by the author (originally, my own code added an ME regularization term directly after the loss function). The hyperparameter settings are as follows: the learning rate for the first layer is set to 0.01, for the second layer it is 0.03, and lambda is set to 15, with the number of local model gradient descent iterations equal to the number of mini-batches. Currently, my task involves using federated learning for CV tasks, and I'm using the Cityscapes dataset, so each user's local dataset only contains 150 samples. When running the code, I noticed that after each round of testing, the accuracy of FedAvg rises quickly, reaching near-convergence levels almost immediately, whereas pFedMe converges more slowly. I'm not sure why this is happening. I've reviewed the algorithm logic, and there can't be any issues there; perhaps there's still a problem with how I've set the hyperparameters. I've switched the network from the author's CNN to a more complex model like BiSeNetV2, which has relatively strong expressive power

kwunhang · 2024-09-07T09:53:34Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

I remember the niid data generation of CIFAR10 contains some problems. But I remember the MNIST dataset should be ok. Cause I also run the PFedMe code in MNIST dataset and get similar result to the paper.
May I know what changes you have made to code? Or you just self implemented the PFedMe code once?

Hi, I'm pleased to receive your response. I haven't made any changes to the algorithmic process of the author's pFedMe code, such as the custom pFedMeOptimizer used by the author (originally, my own code added an ME regularization term directly after the loss function). The hyperparameter settings are as follows: the learning rate for the first layer is set to 0.01, for the second layer it is 0.03, and lambda is set to 15, with the number of local model gradient descent iterations equal to the number of mini-batches. Currently, my task involves using federated learning for CV tasks, and I'm using the Cityscapes dataset, so each user's local dataset only contains 150 samples. When running the code, I noticed that after each round of testing, the accuracy of FedAvg rises quickly, reaching near-convergence levels almost immediately, whereas pFedMe converges more slowly. I'm not sure why this is happening. I've reviewed the algorithm logic, and there can't be any issues there; perhaps there's still a problem with how I've set the hyperparameters. I've switched the network from the author's CNN to a more complex model like BiSeNetV2, which has relatively strong expressive power

I am not sure. But the reason may due to your model. You mentioned that you are using a strong model and take MNIST and CIFAR10 as testing dataset. The dataset mission maybe too easy for the model and lead to high accuracy for both PM and GM.

Fruit0218 · 2024-09-07T16:08:52Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

I remember the niid data generation of CIFAR10 contains some problems. But I remember the MNIST dataset should be ok. Cause I also run the PFedMe code in MNIST dataset and get similar result to the paper.
May I know what changes you have made to code? Or you just self implemented the PFedMe code once?

Hi, I'm pleased to receive your response. I haven't made any changes to the algorithmic process of the author's pFedMe code, such as the custom pFedMeOptimizer used by the author (originally, my own code added an ME regularization term directly after the loss function). The hyperparameter settings are as follows: the learning rate for the first layer is set to 0.01, for the second layer it is 0.03, and lambda is set to 15, with the number of local model gradient descent iterations equal to the number of mini-batches. Currently, my task involves using federated learning for CV tasks, and I'm using the Cityscapes dataset, so each user's local dataset only contains 150 samples. When running the code, I noticed that after each round of testing, the accuracy of FedAvg rises quickly, reaching near-convergence levels almost immediately, whereas pFedMe converges more slowly. I'm not sure why this is happening. I've reviewed the algorithm logic, and there can't be any issues there; perhaps there's still a problem with how I've set the hyperparameters. I've switched the network from the author's CNN to a more complex model like BiSeNetV2, which has relatively strong expressive power

I am not sure. But the reason may due to your model. You mentioned that you are using a strong model and take MNIST and CIFAR10 as testing dataset. The dataset mission maybe too easy for the model and lead to high accuracy for both PM and GM.

I totally agree with what you said. By the way, I am wondering whether I could use Global Test Data to test my Personalized model of pFedMe? Or only Local Test Data can do test Personalized model of pFedMe? Thanks for your response again!

Fruit0218 · 2024-09-08T03:52:48Z

我可以了解更多细节吗？

I'm sorry it took so long to see your reply. Well, I tested pFedMe and FedAvg on MNIST dataset, but the result of the test is that FedAvg obviously shows a high acc from the beginning, on the contrary, the GM and PM accuracy of pFedMe are almost the same. I replaced dataset with CIFAR10 and the results were similar. I'm wondering if there's something wrong with the niid setup. Have you ever encountered a similar problem? Because I have checked the logic of the algorithm, the pFedMe algorithm is consistent with the author's algorithm, that is, using the rewritten pFedMeOptimizer as the first layer, and the second layer is the update of local_model. The logic is not wrong. But I don't think I would make such a silly mistake, because other people have not encountered similar situation), if you can see my reply, I can provide you with other specific information about me, I really need help. Thank you for replying to me in your spare time

I remember the niid data generation of CIFAR10 contains some problems. But I remember the MNIST dataset should be ok. Cause I also run the PFedMe code in MNIST dataset and get similar result to the paper.
May I know what changes you have made to code? Or you just self implemented the PFedMe code once?

Hi, I'm pleased to receive your response. I haven't made any changes to the algorithmic process of the author's pFedMe code, such as the custom pFedMeOptimizer used by the author (originally, my own code added an ME regularization term directly after the loss function). The hyperparameter settings are as follows: the learning rate for the first layer is set to 0.01, for the second layer it is 0.03, and lambda is set to 15, with the number of local model gradient descent iterations equal to the number of mini-batches. Currently, my task involves using federated learning for CV tasks, and I'm using the Cityscapes dataset, so each user's local dataset only contains 150 samples. When running the code, I noticed that after each round of testing, the accuracy of FedAvg rises quickly, reaching near-convergence levels almost immediately, whereas pFedMe converges more slowly. I'm not sure why this is happening. I've reviewed the algorithm logic, and there can't be any issues there; perhaps there's still a problem with how I've set the hyperparameters. I've switched the network from the author's CNN to a more complex model like BiSeNetV2, which has relatively strong expressive power

I am not sure. But the reason may due to your model. You mentioned that you are using a strong model and take MNIST and CIFAR10 as testing dataset. The dataset mission maybe too easy for the model and lead to high accuracy for both PM and GM.

As well as I'm not quite sure why the author's code for pFedMe is to train all users, test all users and aggregate the selected users, and for FedAvg is to train the selected users, test all users and aggregate the selected users. So FedAvg is not self training all users, so why test all users? Here is the biggest question I have. Perhaps I missed something? Sorry to bother you again.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

some questions about the results #23

some questions about the results #23

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

some questions about the results #23

some questions about the results #23

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!