Fusing multidimensional data together, with a focus on microscopy. This repo allows you to fuse time-resolved multispectral single-pixel camera data (4D) with structured illumination microscopy (SIM) (3D) data acquired with CMOS or CCD cameras to reconstruct high-resolution fluorescence lifetime multispectral data (5D).
Fused.mp4
- ⚙️ Instructions
- 💾 Data
- ℹ️ How does it work?
- ⚡️ How to use it?
- 🙌 Acknowledgements
Use conda to create an environment with the required dependencies.
# Clone it
git clone https://github.com/FluoLab/datafusion
# Enter
cd datafusion
# Create environment
conda env create -f environment.yml
# Activate environment
conda activate datafusionYou can find the data used in this repository on Zenodo:
Data fusion algorithms in this repo solve the following convex optimization problem:
where:
-
$\mathbf{X}$ is the 5D fluorescence time-resolved multispectral volume we are reconstructing -
$\mathbf{Y}_{cmos}$ is the structured illumination microscopy 3D data -
$\mathbf{Y}_{spc}$ is the single-pixel camera 4D data -
$T$ ,$S$ and$D$ are the summation operators over different dimensions, temporal decay, spectrum, and depth dimensions, respectively -
$R$ is the downsampling operator in 2D. -
$w_1$ and$w_2$ are the weights for the fidelity terms.
fusion.py provides two ways to solve the optimization problem:
- Linear Conjugate Gradient: best for convergence speed of this problem
- Adam + Backpropagation: great for inclusion of learned reconstruction techniques
from datafusion.fusion import FusionCG
from datafusion.utils import (
download_url,
load_data,
ZENODO_URL,
RESOURCES_PATH,
)
# Download the example data.
download_url(
ZENODO_URL,
RESOURCES_PATH / "acquisitions.zip",
chunk_size=2048,
unzip=True,
)
# Load the data.
spc, cmos, _, _ = load_data(
path=RESOURCES_PATH / "acquisitions" / "cells" / "cells_0.50cr.npz",
max_xy_size=128,
)
# Set up the penalties for the fidelity terms.
weights = {
"spatial": 0.5,
"spectro_temporal": 0.5,
}
# Fuse the data.
fuse_with_cg = FusionCG(
spc,
cmos,
weights=weights,
init_type="baseline",
device="cpu", # "cuda" or "mps" for GPU acceleration.
)
x, _, _ = fuse_with_cg(max_iterations=20)
# x is now the reconstructed 5D fluorescence time-resolved multispectral volume.This work is supported by the European Union (GA 101072354) 🇪🇺 and the UK Research and Innovation (EP/X030733/1) 🇬🇧. It comes from the effort and synergy of multiple people from different fields: Federico Simoni, Serban Cristian Tudosie, Shivaprasad Varakkoth, Valerio Gandolfi, Simon Arridge, Andrea Farina, and Cosimo D'Andrea.