This is the official implementation of "DoF: A Diffusion Factorization Framework for Offline Multi-Agent Decision Making" published in ICLR 2025.
DoF is a novel Diffusion Factorization Framework for tackling the challenges of offline Multi-Agent Decision Making (MADM). While diffusion models have shown success in image and language generation, their application in cooperative multi-agent decision-making remains limited due to issues with scalability and cooperation. To address this, we extend the Individual-Global-Max (IGM) principle to the Individual-Global-Identically Distributed (IGD) principle, ensuring that outcomes from a centralized diffusion model are identically distributed with those from multiple decentralized models, enhancing both scalability and cooperative efficiency.
-
🔍 Introduction of the IGD Principle
A novel extension of the traditional IGM principle that ensures multi-agent diffusion outcomes are identically distributed across centralized and decentralized models. -
🏗️ DoF Framework
A powerful factorization framework that decomposes centralized diffusion models into multiple agent-specific models through:- Noise Factorization Function: Factorizes the centralized diffusion noise into decentralized noise components while adhering to the IGD principle.
- Data Factorization Function: Models complex inter-agent data relationships, improving coordination and learning efficiency.
-
📈 Theoretical Validation
We provide formal proof demonstrating that our noise factorization approach satisfies the IGD principle, ensuring both scalability and improved cooperative learning. -
🧪 Extensive Empirical Evaluation
Our experiments across various MADM benchmarks (including SMAC, MPE, and MA-Mujoco) demonstrate DoF's effectiveness and superior scalability compared to existing methods.
sudo apt-get update
sudo apt-get install libssl-dev libcurl4-openssl-dev swig
conda create -n dof python=3.8
conda activate dof
pip install torch==1.12.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txtDatasets for SMAC, sourced from off-the-grid MARL, and for MPE, provided byOMAR, are available at the following links:
Run SMAC Environment
For the 3m scenario in SMAC, use the following command:
python pyrun.py -c cfg/smac/mad_smac_3m_ctde_history_good.yamlThe evaluation results will be presented in logs/smac.
Run MPE Environment
For the Spread scenario in MPE, use the following command:
python pyrun.py -c cfg/mpe/mad_mpe_spread_ctde_exp.yaml SMACv2 Environment Run Terran 5v5 Maps:
python pyrun.py -c cfg/smacv2/mad_smac_terran_5_vs_5_ctde_replay_history_DoF.yaml Run Terran 10v10 Maps:
python pyrun.py -c cfg/smacv2/mad_smac_terran_10_vs_10_ctde_replay_history_DoF_weight.yamlRun Zerg Maps
python pyrun.py -c cfg/smacv2/mad_smac_zerg_ctde_replay_history_DoF_weight.yaml Evaluate To evaluate the trained model:
python pyrun.py -c cfg/eval_inv.yaml
- MPE Environment:
pip install -e third_party/multiagent-particle-envs
pip install -e third_party/ddpg-agent- MA-Mujoco Environment:
pip install -e third_party/multiagent_mujocoDatasets for MPE different tasks can be found at the following link OMAR
Datasets for MA-Mujoco from from off-the-grid MARL can be found at the following links:
Please follow the below commands to replicate the results in the paper.
python train_policy_main.py --config <DoF-P_YAML_DIR>/<ENVIRONMENT>_<SUB_ENVIRONMENT>_<DATASET_TYPE>_dof_<DATA_MODE>_<NOISE_MODE>_<SEED>.yamlModify the YAML configuration for data and noise factorization:
data_factorization_mode: "concat" # "concat" "w-concat" "default"
noise_factorization_mode: "concat" # "concat" "w-concat"Refer to the official MADIFF repository for installation and configuration.
To train:
# For Multi-Agent Particle Environment
python run_experiment.py -e exp_specs/mpe/<task>/mad_mpe_<task>_attn_<dataset>.yaml
Follow the official OG-MARL repository for setup and experiments.
Clone and install:
git clone https://github.com/instadeepai/og-marl.git
pip install -e .[tf2_baselines]
@inproceedings{lidof2025,
title={DoF: A Diffusion Factorization Framework for Offline Multi-Agent Decision Making},
author={Li, Chao and Deng, Ziwei and Lin, Chenxing and Chen, Wenqi and Fu, Yongquan and Liu, Weiquan and Wen, Chenglu and Wang, Cheng and Shen, Siqi},
booktitle={International Conference on Learning Representations},
year={2025}
}
Our codebase is inspired by the following repositories: