Open
Description
The way multiprocessing.pool works seems to conflict with OpenMP. The resulting behavior is that the program gets stuck.
As @noambernstein suggested:
We could try to detect this and give a warning, or investigate whether there's some way to modify the multiprocessing.pool calls to allow for OpenMP in the subprocesses.
Related github-issue:
dmlc/xgboost#6617 (comment)
Concrete example:
import os
from quippy.potential import Potential
from wfl.calculators import generic
from wfl.autoparallelize.autoparainfo import AutoparaInfo
from wfl.configset import ConfigSet, OutputSpec
configset = ConfigSet('./train.xyz')
outputspec = OutputSpec("train_out.xyz")
calculator = (Potential, [], {'param_filename': 'GAP.xml'}) # (calculator_constructor_function, arguments, keyword_arguments)
generic.run(
inputs = configset,
outputs = outputspec,
calculator = calculator,
output_prefix = "GAP_",
autopara_info = AutoparaInfo(
num_python_subprocesses=2,
num_inputs_per_python_subprocess=40
)
)
Tried to execute this piece of code, but it gets stuck when it reaches the point where it is trying to perform the actual calculations with the GAP.
The code works as expected when setting OMP_NUM_THREADS=1.
Interestingly, setting it to values greater than 1 again causes the script to get stuck.
Metadata
Metadata
Assignees
Labels
No labels