Kinetic models of cellular metabolism rely on accurate enzyme-specific parameters, but data scarcity and high-dimensional spaces hinder traditional methods. We present RL-Renaissance, an extension of the NES-based RENAISSANCE framework, that formulates kinetic parameterization as a sequential decision process. Using Proximal Policy Optimization (PPO), it iteratively refines a 384-dimensional parameter vector, guided by rewards based on the model's Jacobian eigenvalue. Compared to RENAISSANCE, RL-Renaissance achieves a 91% success rate (vs. 95%) on E. coli metabolism, while being almost 2x more sample efficient. Reward and eigenvalue analyses show effective learning, though PPO’s sensitivity to hyperparameters suggests avenues for refinement. These results demonstrate the potential of policy-gradient methods for efficient and scalable systems-biology modeling.
🔮 Future work
For those who are interested in the future of this project, here are some ideas:
- Try using more predictions per step to obtain better estimate of the distribution of the next state, and stabilise the training
- Run for more than 100 episodes, e.g. 150, 200
- Try integrating timestep information into the model
- Try using more complex model such as LSTM that can take into account temporal information
- Try using non-stochastic policy, however stochastic input
In order to run the code, you need to set up the docker container which is due to the dependency on skimpy package. To make your life easier, and not having to build the container yourself, it is publicly available on Docker Hub.
First, pull the image from the Docker registry using apptainer (takes a couple of minutes):
mkdir -p /scratch/izar/$USER/images
apptainer pull /scratch/izar/$USER/images/renaissance_with_ml.sif docker://ludekcizinsky/renaissance_with_ml:latestThen, you can run the image with the following command:
mkdir -p /scratch/izar/$USER/rl-for-kinetics/output
apptainer shell --nv --bind "$(pwd)":/home/renaissance/work --bind "/scratch/izar/$USER/rl-for-kinetics/output:/home/renaissance/output" /scratch/izar/$USER/images/renaissance_with_ml.sifFor some random reason, when inside the container, you also need to run:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8If things go well, you should be able to execute the following command to check if the image is working:
python -c "import skimpy; import torch;print('Success')"First, pull the image from the Docker registry (this should be already done, but just in case):
sudo docker pull ludekcizinsky/renaissance_with_ml:latestNext, in your home directory, create a directory for the output as well as installing the dependencies. For instance:
mkdir -p /home/rl_team/ludek/output
mkdir -p /home/rl_team/python_packagesMake sure that the folders are writable by the user:
chmod -R 777 /home/rl_team/ludek/output
chmod -R 777 /home/rl_team/python_packagesNext, visit your w&b user setting, and copy your API key. Then add it to your .netrc as follows:
machine api.wandb.ai
login user
password <your_api_key>For instance, I save mine at /home/rl_team/ludek/.netrc.
Assuming you have cloned our rl repo and you are currently in it, you can start the docker container using the below command. Make sure to change the output directory to the one you created earlier,
and the python packages directory to the one you created earlier (do not change the paths within the container). Finally, double check that you also mount the .netrc file (I include mine in the command below):
sudo docker run --rm -it -v "$(pwd)":/home/renaissance/work -v "/home/rl_team/ludek/output:/home/renaissance/output" -v "/home/rl_team/python_packages:/home/renaissance/.local" -v "/home/rl_team/ludek/.netrc:/home/renaissance/.netrc" ludekcizinsky/renaissance_with_mlFinally, since we have to work with python 3.6, we have to downgrade weights and biases:
pip install wandb==0.15.11If things go well, you should be able to execute the following command to check if the image is working:
python -c "import skimpy; import torch;print('Success')"We save all our final results in the wandb platform which is publicly accessible. You can take a look there to download any trained model using the available tags, please see how to do so in this notebook. However, if you just want to use the best model, we have saved all neccesary files in the inference directory under best model folder.
Finally, to run it and play around with the generated kinetic models, and inspect the rewards, you can use the inference notebook.
Assuming you are now inside the docker container and followed the instructions above, you can now run the code in the following way.
In order to reproduce the results of the main method reported in the report, you can run the following command:
python train.pyIn order to reproduce the results of the ablation studies reported in the report, you can schedule a SLURM job as follows:
sbatch train1.slurm <your_username>
sbatch train2.slurm <your_username>Or just look at the train1.slurm and train2.slurm files for which configurations you want to run on whatever system you are on.
We would like to thank Ljubisa Miskovic and Ilias Toumpe for giving us useful guidance and feedback on the project.