Ali Farhadi2,3, Abhishek Gupta2, Shenlong Wang1, Wei-Chiu Ma4
Our code repo needs to setup three environments.
-
For SDF Reconstruction, please follow the instructions Here.
-
For Gaussian Splat Reconstruction, please follow the instructions Here.
-
For Isaac Sim, please follow the instructions Here.
-
We provide the data we collected in UIUC and UW, which can be downloaded from Here.
-
We provide our door tracking evaluation data Here.
We provide the download links of checkpoints of foundation models we used Here.
DRAWER is a system with multiple stages. We provide a guidance for the usage of our code stage-by-stage (SDF Reconstruction, 3D Based Perception, Isaac Sim Simulation, and Gaussian Splat Reconstruction) and the whole system altogether Here.
We also provide the scripts for the usage of our dual representation reconstruction. Please directly check Here.
For the whole system usage, here is a brief explanation:
Stage 1: SDF Reconstruction:
-
Use diffusion-e2e-ft to generate monocular normal and depth priors.
-
Use an improved version of BakedSDF based on SDFStudio to run sdf reconstruction.
Stage 2: 3D Based Perception:
-
Use Grounded SAM to recognize drawers and cabinet doors in the scene.
-
Use graph based method and GPT-4o to filter and finalize the 2d locations of doors. (We suggest double checking with the perception results, since it's not perfect and could lead to failure cases, and please refer to note Here.)
-
Use 3DOI to infer the articulation information.
-
Fit drawer and cabinet doors to correct 3d location.
Stage 3: Isaac Sim Simulation:
-
Transform the scene into USD file format.
-
Simulate the scene dynamics and attain the trajectory of doors.
Stage 4: Gaussian Splat Reconstruction:
-
Run our Gaussian on Mesh reconstruction developed based on NeRFStudio.
-
Use Matfuse to generate textures.
-
Merge Gaussian Splats of the scene and the cabinets.
DRAWER has applications including:
-
Use scripts Here to export Gaussians of the scene and each doors, which could be loaded into Unreal Engine with Lama AI Plugin.
-
Real-to-Sim-to-Real Training: Please use the scripts Here to generate usd file for isaac lab and refer to the code Here.
- SDF Reconstruction
- Perception
- Isaac Sim Simulation
- Gaussian Splat on Mesh
- Data Release
- Real-to-Sim-to-Real Training
Contact Hongchi Xia if you have any further questions.
Our codebase builds heavily on SDFStudio, NeRFStudio, diffusion-e2e-ft, Grounded SAM, 3DOI, and Matfuse. Thanks for open-sourcing!