SEML is the missing link between the open-source workload scheduling system Slurm, the experiment management tool sacred, and a MongoDB experiment database. It is lightweight, hackable, written in pure Python, and scales to thousands of experiments.
Keeping track of computational experiments can be annoying and failure to do so can lead to lost results, duplicate running of the same experiments, and lots of headaches.
While workload scheduling systems such as Slurm make it easy to run many experiments in parallel on a cluster, it can be hard to keep track of which parameter configurations are running, failed, or completed.
sacred is a great tool to collect and manage experiments and their results, especially when used with a MongoDB. However, it is lacking integration with workload schedulers.
SEML enables you to
- very easily define hyperparameter search spaces using YAML files,
- run these hyperparameter configurations on a compute cluster using
Slurm, - and to track the experimental results using
sacredandMongoDB.
In addition, SEML offers many more features to make your life easier, such as
- automatically saving and loading your source code for reproducibility,
- easy debugging on Slurm or locally,
- automatically checking your experiment configurations,
- extending Slurm with local workers,
- and keeping track of resource usage (experiment runtime, RAM, etc.).
To get started, install SEML either via pip:
pip install semlor conda:
conda install -c conda-forge semlThen configure your MongoDB via:
seml configure --mongodb # provide your MongoDB credentialsSee our simple example to get familiar with how SEML works.
SEML supports (rudimentary) command line completion. To install this feature run:
seml configure --argcompleteThere are three options available, we recommend to install the completion as user:
user: Creates~/.bash_completion, which will be automatically sourced when a shell is openedglobal: Creates a similar file at a suitable location depending on the shell being used. You need sudo rights to install argument completion globally.path: Queries for a specific path to which to install the argument completion script. This script needs to be sourced in every shell session in which argument completion is to be used. In the background, this feature relies on the argcomplete module. To disable and purge the feature you should uninstall this module and delete (the relevant parts of) the file created byseml configure --argcomplete(which depends on the installation mode you chose).
SEML should work with Slurm 18.08 and above out of the box. Version 17.11 and earlier do not have a SIGNALING job state, which you have to remove from the SLURM_STATES defined in SEML's settings (seml/settings.py). Earlier versions have not been tested and might have other issues.
Contact us at zuegnerd@in.tum.de, johannes.gasteiger@tum.de, or n.gao@tum.de for any questions.
When you use SEML in your own work, please cite the software along the lines of the following bibtex:
@software{seml_2022,
author = {Z{\"u}gner, Daniel and Gasteiger, Johannes and Gao, Nicholas},
title = {{SEML: Slurm Experiment Management Library}},
url = {https://github.com/TUM-DAML/seml},
version = {0.3.7},
year = {2022}
}
Copyright (C) 2023
Daniel Zügner, Johannes Gasteiger, and Nicholas Gao
Technical University of Munich