Memory issue using Islands/Archipelagos #168
Replies: 1 comment 8 replies
-
|
@davide-roversi what is the memory consumption when you are using the BFE on a single population and how many individuals do you have in the population? |
Beta Was this translation helpful? Give feedback.
-
|
@davide-roversi The default BFE behaviour is to use multiprocessing as well, just like the archipelago, thus also in the case of BFE your problem is being copied around to remote processes. The difference I think is that in the archipelago, in addition to the copies being made and sent around for multiprocessing, you also have one problem copy per island, so the overall memory consumption is increased. Is there any internal problem state that grows over time as you perform more function evaluations? |
Beta Was this translation helpful? Give feedback.
-
Conceivably you could rent some compute from AWS or some other cloud provider and set up your cluster there, but it may start to be a bit on the expensive side. Having access to an institutional cluster would certainly be cheaper :) |
Beta Was this translation helpful? Give feedback.
-
|
I do have a few arrays that are increased throughout the evolutions, such as an array containing all best fitnesses, and one containing all best individuals. However, let's say I have 250 max iterations and a decision variables vector of length 8, those to arrays I mention would end up being 250-by-1 and 250-by-8. It would surprise me if that was the cause for memory filling up. Moreover, once again, using BFE on GACO or PSO does not lead me to memory issues, despite above mentioned evolutions-increasing arrays being present. Additionally, I have gone back on pygmo API to see if I was overlooking something. I noticed that, when creating the archipelago, the default udi is set to |
Beta Was this translation helpful? Give feedback.
-
|
In theory using the pool can be more efficient because you are avoiding creating ex-novo a process every time an island evolves (which can have a large overhead), but it also means that the resources used during evolution are not necessarily cleaned up when the evolution concludes. It looks like there is some memory leakage, but it is hard to say with certainty what is going on exactly without seeing the code or having at least a reproducer to investigate. It could be just Python holding onto references to your problems in the pool processes instead of garbage collecting them, but again, I would need to take a look at the code in order to investigate. |
Beta Was this translation helpful? Give feedback.
-
|
I understand, thank you for the insights. The code is quite extensive, so I might try to reduce it to a minimum working example before sending it over. For now, having run some simulations in the past week with |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello everyone, I am posting this here and also on Gitter (link: https://app.gitter.im/#/room/#pagmo2_Lobby:gitter.im)
I am using Pygmo for the single-objective optimization of a problem of size 8 (i.e. 8 decision variables, and 1 fitness value per-evolution). The fitness function takes about 1~2 seconds to run on single-core. To speed things up, I implemented both multi-processing via bfe and multi-core using the island/archipelago setup.
Bfe works like a charm, significantly speeds up the process, and does not use a lot of RAM. The problem is, for single-objective optimization, it only works on GACO, PSO, PSO_GEN amd CMAES, all of which do not perform well with the problem I am trying to solve. SADE is a far better option, but does not support “set_bfe()”.
So I switched to the coarse-grained archipelago setup. I have 20 CPU cores (virtual + physical), and 32GB RAM at my disposal. Using SADE with a population of just 7 individuals per-island, times 20 islands, ram gets full at 99.9% after a few evolutions. If I increase the islands sizes, I sometimes get out-of-memory errors. Looking at the resource monitor as the optimization progresses, I see that the RAM usage continuously rises, and never gets reset throughout the evolutions.
My questions are two:
Other info:
Beta Was this translation helpful? Give feedback.
All reactions