The OpenFOAM Benchmark Runner (OBR) is an experimental workflow manager for simple provisioning of complex parameter studies and reproducible simulations. A typical OpenFOAM workflow of setting up a case, performing various parameter manipulations, and serial or parallel execution, can be defined using the yaml markup language. OBR is build on top of signac.
As of now, we recommend to clone the repository and to perform a symlink install of OBR into your virtual environment via pip:
pip install -e .
The benchmark runner is split into several layers:
- case definition via yaml files
- case generation via
obr initandobr run -o generate - run or submit solver execution via
obr run -o runParallelSolverorobr submit -o runParallelSolver - postprocessing cases Experimental via,
obr applyandobr archive
The micro_benchmarks repository provides a good point to start learning from. After cloning the repository, cd into the LidDrivenCavity3D directory, where an example yaml file can be found in the case assets folder. Another example of a workflow is shown next
case:
type: CaseOnDisk
solver: pimpleFoam
origin: ${{yaml.location}}/../basicSetup
variation:
- operation: fvSolution
schema: "linear_solver/{solver}{preconditioner}{executor}"
values:
- set: solvers/p
preconditioner: none
solver: GKOCG
forceHostBuffer: 1
verbose: 1
executor: ${{env.GINKGO_EXECUTOR}}
- set: solvers/p
preconditioner: IC
solver: GKOCG
forceHostBuffer: 1
verbose: 1
executor: ${{env.GINKGO_EXECUTOR}}
The workflow copies an existing case on disk and creates a Workspace.
In general, to create a tree of case variations run
obr init --folder [path] --config path-to-config.yaml
Within the context of the micro_benchmarks example, simply run
obr init --folder . --config assets/scaling.yaml
OBR should now print some output, followed by INFO: successfully initialized.
Finally, operations on a tree can be run with the obr run command-line option, for example fetchCase, which is responsible for copying the base case into the workspace:
obr run -o fetchCase --folder path-to-tree
Or, in this specific example (the default of --folder is .):
obr run -o fetchCase
Within LidDrivenCavity3D/workspace should now have appeared a multitude of directories (=jobs), which are in the form of a UID eg. 78e2de3e6205144311d04282001fe21f. Each job represents a distinct operation such as modifying the blockMeshDict and call blockMesh including its dependencies. The order in which operations can be applied are defined by the config.json, however, running
obr run -o generate
also runs all defined operations in the appropriate order.
On HPC cluster OBR can submit operations via the job queue. For example
obr submit -o blockMesh
will submit the blockMesh operation to the cluster manager for every job that is eligible. OBR detects the installed job queuing system, eg. slurm, pbs, etc. A jobs ubmission script will be generated automatically. For fine grained control over the submission script the --template argument allows to specify the location of a submission script template. Since OBR uses signac for job submission more details on how to write job submission templates can be found here. To avoid submitting numereous jobs individually, the --bundling-key argument can be used to bundle all jobs for which the bundling key has the same value into the same job.
Since OBR aims at performing parameter studies containing a larger number of individual cases, postprocessing cases manually should be avoided. Its is recommended to use obr apply instead.
obr apply --file script.py --campaign ogl_170
The passed script.py file must implement a call(jobs: list[Job], kwargs={}) function. On execution this gets a list of jobs which allow access to the case paths.
The mandatory --campaign argument is used to separate (families of) experiments. The optional argument --tag <str> can be used to further differentiate between experiments within a campaign. Internally, this will create subfolders as such:
fd52708db5c296d1fa52b056701be4ee
├── campaign1
│ ├── tag1
│ │ ├── decomposePar_2024-01-05_17:38:44.log
│ │ ├── instrumentedPimpleFoam_2024-01-05_17:44:32.log
│ │ └── solverExitCode.log
│ └── tag2
│ ├── decomposePar_2024-01-05_17:38:44.log
│ ├── instrumentedPimpleFoam_2024-01-05_17:44:32.log
│ └── solverExitCode.log
└── campaign2
OBR workflows often rely on environmental variables to adapt a workflow to specific node or cluster properties. In workflow.yaml files for example ${{env.HOST}} is replaced by
$HOST. Additionally, OBR_RUN_CMD defines the exact command to execute for parallel runs and OBR_PREFLIGHT can call a script to verify your environment just before the solver execution.
export OBR_RUN_CMD="mpirun --bind-to core --map-by core -np {np} {solver} -parallel -case {path}/case > {path}/case/{solver}_{timestamp}.log 2>&1"
export OBR_PREFLIGHT="python3 $HOME/data/code/exasim_project/micro_benchmarks/common/preflight.py"
Additionally, OBR_SKIP_COMPLETE defines if a already complete run should be repeated.
To contribute to this project, we suggest creating an issue. From then on, we can discuss further development.