Dyablo is a framework to develop Computational Fluid Dynamics (CFD) simulations using Adaptive Mesh Refinement (AMR) on large scale supercomputers.
It's an attempt to modernize de software stack, initially for numerical simulations for astrophysics. Dyablo is written in C++ with performance portability in mind and uses an MPI+Kokkos hybrid approach to parallelism.
The MPI Library is used for distributed parallelism and compute kernels using shared-memory parallelism use the Kokkos performance portability library. MPI is used to distribute the AMR mesh across multiple compute nodes, while Kokkos allows us to write a single code that can be executed on multithreaded CPUs, GPUs and other parallel architectures supported by Kokkos.
Dyablo is also build with modularity and ease of use in mind to allow physicists to easily add new kernels written with abstract interfaces to access and modify the AMR mesh.
Modularity is also key to use state-of-the-art libraries interchangeably, reuse existing work and allow compatibility with external tools for vizualization or post-processing for instance. In Dyablo, the AMR mesh can be managed by the PABLO external library or by the custom implementation in written in Kokkos. This modularity enables us to plug in other external libraries to manage the AMR mesh or to perform IO, vizualization or post-processing operations. For now vizualisation outputs are handled by the HDF5 library but other backends can be integrated to Dyablo through plug-ins.
Dyablo includes Kokkos and other libraries as git submodules. Make sure to clone this repository recursively, this will also download Kokkos and other dependencies.
git clone --recurse-submodules git@drf-gitlab.cea.fr:dyablo/dyablo.gitor if you already cloned the repository without --recurse-submodules :
git submodule update --initFor the latest version of Dyablo, we recommend that you use the dev (default) branch .
NOTE : If you don't have access to github from the machine (e.g at TGCC), you will need to populate the external/ folder manually with kokkos and backward-cpp. If you want to build unit-tests you also need gtest. If you want to enable the legacy PABLO backend for AMR, you also need bitpit.
The CMake superbuild should automatically find dependencies and warn you if any dependency is missing. CMake version > 3.16 is needed to compile Kokkos.
You will need a recent C++ compiler compatible with C++17 and capable of compiling Kokkos. Recommended compiler versions for Dyablo are :
g++> 12icc> 19.0.5clang> 11nvcc> 12- ...
Other dependencies include :
- MPI
- HDF5 in parallel mode (apt install libhdf5-mpi-dev)
- libxml2
To compile for GPU (Nvidia), a CUDA installation is needed, preferably newer than CUDA 11.2. Dyablo supports both CUDA-Aware and non CUDA-Aware MPI implementations when compiling for GPU, make sure that cuda-aware support has been correctly detected in the CMake logs.
Kokkos automatically detects and sets the CUDA compiler when Kokkos_ENABLE_CUDA is ON :
- When the C++ compiler is not compatible with CUDA, Kokkos uses NVCC to compile device code. NVCC version must be >= 11.2
- When the C++ compiler is compatible with CUDA (e.g clang), Kokkos uses this compiler For more details, see the Kokkos documentation
AMD and Intel GPUs are also supported, see "Build on specific systems" for details.
GPU-Aware auto-detection only works for OpenMPI on Nvidia, use -DDYABLO_USE_MPI_CUDA_AWARE_ENFORCED=ON if you know your MPI implementation is GPU-Aware but it was not detected automatically.
The top-level CMakeLists.txt uses the the super-build pattern to build Dyablo and its depencies (here Kokkos) using cmake command ExternalProject_Add. Using the superbuild is the recommended way to compile Dyablo because it ensures that the Kokkos compilaton configuration (Architecture, enabled backends, etc...) is compatible with how Dyablo is configured.
To build Kokkos and dyablo (for Kokkos/OpenMP backend which is the default)
mkdir build_openmp; cd build_openmp
cmake -DCMAKE_BUILD_TYPE=Release ..
makeThe same for Kokkos/CUDA:
mkdir build_cuda; cd build_cuda
cmake -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_CUDA=ON ..
makeBuild commands for some linux distributions and supercomputers can be found here to help you find the right packages or modules as well as the command line to use to compile Dyablo.
Configuration options for Dyablo ( cmake -D arguments ) include :
-DCMAKE_BUILD_TYPE=<buildtype>:Release(recommended for performance),Debug(enables asserts and debug symbols),RelWithDebINfo-DKokkos_ENABLE_CUDA=ON/OFF: enable CUDA (use Kokkos_ENABLE_HIP for AMD, ...)-DKokkos_ARCH="<arch1>,...": target specific architectures (i.e. set architecture-specific optimization flags) listed in the Kokkos documentation. You may target multiple architectures : for example to compile for a machine with Intel Skylake CPU + V100 GPUs, you might want to set-DKokkos_ARCH="SKX;VOLTA70". CUDA architecture is auto-detected ifKokkos_ARCHis not present. Default CPU architecture might not fully use vectorization features, don't forget to set a CPU architecture for OpenMP compilation.-DDYABLO_ENABLE_UNIT_TESTING=ON/OFF: enable/disable unit tests. If enabled, runmake dyablo-testfrom your build directory to run all tests
Compilation may take a long time, we recommend you use parallel compilation with make -j <number of cores>.
The main executable dyablo is in build/dyablo/bin/. The directory also contains some *.ini files that can passed to the executable on the command line. For instance, to run a 2d Sedov-blast on the block-AMR backend, the command will be ./dyablo test_blast_2D_block.ini. Other .ini files are available in the settings/ directory at the root of the project
Beware, when recompiling, the .ini files in bin may be reset to their original state.
run for instance ./dyablo test_blast_3D_block.ini to run the 3D block-based blast test case. This executable accepts Kokkos command-line parameters.
For the best performance, you should follow the global advice for any Kokkos program :
- Configure OpenMP to bind threads by setting the environment variable OMP_PROC_BIND=true
- Use the Kokkos command line arguments to correcty bind GPUs : e.g.
--num-devices=4when there are 4 GPUs/node - Of course, learn to use your job manager (slurm) efficiently for hybrid codes (
-c,--hint, etc...).
Dyablo uses .ini files for configuration. The format is described here. key=value lists are arranged into [sections] :
[mesh]
# 2D or 3D
ndim=2
xmin=0.0
xmax=1.0
ymin=0.0
ymax=1.0
[amr]
level_min=5
level_max=7
[...]Dyablo generates a last.ini file listing the values used for all the variables needed for the last simulation. Values in last.ini are annotated with :
Default: the value was not listed in the original.inifile and the value shown inlast.iniis the defaut value chosen by DyabloUnused: The value was listed in the original.inibut was not used by Dyablo
Unfortunately no documentation about the content of the .ini file is available yet, you will need to read the code (sorry). For now you can grep -r configMap.get core/src to see what variables are fetched from the .ini.
If you used the XDMF+HDF5 output backend, Dyablo should produce *.h5 and *.xmf files that contain unstructured mesh data. You can open *_main.xmf with paraview to display the output of your simulation.
For now, just visit the wiki page https://drf-gitlab.cea.fr/dyablo/dyablo/-/wikis/home