😃 This repository contains the implementation of "BlobCtrl: A Unified and Flexible Framework for Element-level Image Generation and Editing".
Keywords: Image Generation, Image Editing, Diffusion Models, Element-level
TL;DR: BlobCtrl enables precise, user-friendly multi-round element-level visual manipulation.
Main Features: 🦉Element-level Add/Remove/Move/Replace/Enlarge/Shrink.
Yaowei Li 1, Lingen Li 3, Zhaoyang Zhang 2‡, Xiaoyu Li 2, Guangzhi Wang 2, Hongxiang Li 1, Xiaodong Cun 2, Ying Shan 2, Yuexian Zou 1✉
1Peking University 2ARC Lab, Tencent PCG 3The Chinese University of Hong Kong ‡Project Lead ✉Corresponding Author
🌐Project Page | 📜Arxiv | 📹Video | 🤗Hugging Face Demo | 🤗Hugging Model
🤗Hugging Data (TBD) | 🤗Hugging Benchmark (TBD)
compress_BlobCtrl-video.mp4
Youtube Introduction Video: Youtube.
📖 Table of Contents
- [TBD] Release the data preprocessing code.
- [TBD] Release the BlobData and BlobBench.
- [TBD] Release the training code
- [20/03/2025] Release the inference code.
- [17/03/2025] Release the paper, webpage and gradio demo.
We introduce BlobCtrl, a framework that unifies element-level generation and editing using a probabilistic blob-based representation. By employing blobs as visual primitives, our approach effectively decouples and represents spatial location, semantic content, and identity information, enabling precise element-level manipulation. Our key contributions include: 1) a dual-branch diffusion architecture with hierarchical feature fusion for seamless foreground-background integration; 2) a self-supervised training paradigm with tailored data augmentation and score functions; and 3) controllable dropout strategies to balance fidelity and diversity. To support further research, we introduce BlobData for large-scale training and BlobBench for systematic evaluation. Experiments show that BlobCtrl excels in various element-level manipulation tasks, offering a practical solution for precise and flexible visual content creation.
Environment Requirement 🌍
BlobCtrl has been implemented and tested on CUDA121, Pytorch 2.2.0, python 3.10.15.
Clone the repo:
git clone git@github.com:TencentARC/BlobCtrl.git
We recommend you first use conda to create virtual environment, and install needed libraries. For example:
conda create -n blobctrl python=3.10.15 -y
conda activate blobctrl
python -m pip install --upgrade pip
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121
pip install xformers torch==2.2.0 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
Then, you can install diffusers (implemented in this repo) with:
pip install -e .
Download Model Checkpoints 💾
Download the corresponding checkpoints of BlobCtrl.
sh examples/blobctrl/scripts/download_models.sh
The ckpt folder contains
- Our provided BlobCtrl checkpoints (
UNet LoRA+BlobNet). - Pretrained SD-v1.5 checkpoint.
- Pretrained DINOv2 checkpoint.
- Pretrained SAM checkpoint.
The checkpoint structure should be like:
|-- models
|-- blobnet
|-- config.json
|-- diffusion_pytorch_model.safetensors
|-- dinov2-large
|-- config.json
|-- model.safetensors
...
|-- sam
|-- sam_vit_h_4b8939.pth
|-- unet_lora
|-- pytorch_lora_weights.safetensors
BlobCtrl demo 🤗
You can run the demo using the script:
sh examples/blobctrl/scripts/run_app.sh
BlobCtrl Inference 🌠
You can run the inference using the script:
examples/blobctrl/scripts/inference.sh
@misc{li2024brushedit,
title={BlobCtrl: A Unified and Flexible Framework for Element-level Image Generation and Editing},
author={Yaowei Li, Lingen Li, Zhaoyang Zhang, Xiaoyu Li, Guangzhi Wang, Hongxiang Li, Xiaodong Cun, Ying Shan, Yuexian Zou},
year={2025},
eprint={2503.13434},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Our implementation builds upon the diffusers library. We extend our sincere gratitude to all the contributors of the diffusers project!
We also acknowledge the BlobGAN project for providing valuable insights and inspiration for our blob-based representation approach.
For any question, feel free to email liyaowei01@gmail.com.