This is the official code release of the CVPR 2024 paper NeuRAD: Neural Rendering for Autonomous Driving and CVPR 2025 paper SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving building on top of Nerfstudio. Besides releasing the code for NeuRAD and SplatAD, we hope that this can lay the ground-work for research on applying neural rendering methods in autonomous driving.
In line with Nerfstudio's mission, this is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.
Do you have feature requests or want to add your new AD-NeRF model? Or maybe provide structures for a new dataset? We welcome contributions!
[June 2025] Code for SplatAD is now released in neurad-studio. The code uses our custom fork of gsplat, found here, for handling rolling shutter and lidar rendering. For Apptainer users we provide a recipe that lets you build an image with all the needed dependencies.
The quickstart will help you get started with the NeuRAD model on a PandaSet sequence. For more complex changes (e.g., running with your own data/setting up a new NeRF graph), please refer to our references.
Our installation steps largely follow Nerfstudio, with some added dataset-specific dependencies. You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDA here.
The models require python >= 3.10. We recommend using conda to manage dependencies. Make sure to install Conda before proceeding.
conda create --name neurad -y python=3.10
conda activate neurad
pip install --upgrade pipInstall PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and tiny-cuda-nn.
cuda-toolkit is required for building tiny-cuda-nn.
For CUDA 11.8:
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
# Some need to upgrade dill prior to tiny-cuda-nn install
pip install dill --upgrade
pip install --upgrade pip "setuptools<70.0"
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torchFor support of Waymo-Open-Dataset v2 (requires python3.10, also dependencies from this package are very strict so cannot add it to pyproject.toml and need install first):
pip install waymo-open-dataset-tf-2-11-0==1.6.1We refer to Nerfstudio for more installation support.
git clone https://github.com/georghess/neurad-studio.git
cd neurad-studio
pip install -e .If you want to use SplatAD, our 3DGS-based method for camera and lidar rendering (paper here), you will need to also install our custom fork of gsplat.
OR if you want to skip all installation steps and directly start using NeuRAD, use the provided docker image or apptainer recipe:
Dockerfile or Apptainer recipe.
The following will train a NeuRAD model. However, training SplatAD instead is as easy as calling a different method name.
Begin by downloading PandaSet and unzip it under data/pandaset.
The dataset is no longer hosted by Scale but can be downloaded from the provided huggingface link.
Training models is done the same way as in nerfstudio, i.e.,
# Train NeuRAD
python nerfstudio/scripts/train.py neurad pandaset-dataIf everything works, you should see training progress like the following:
Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).
To train SplatAD instead, you simply call
# Train SplatAD
python nerfstudio/scripts/train.py splatad pandaset-dataIf you run into issues, it could be due to the training taking up too much memory. You can try to adjust the model parameters according to the neurad-tiny vscode launch config.
It is possible to load a pretrained model by running
pyhton nerfstudio/scripts/train.py neurad pandaset-data --load-dir {outputs/.../nerfstudio_models}Given a pretrained model checkpoint, you can start the viewer by running
python nerfstudio/scripts/viewer/run_viewer.py --load-config {outputs/.../config.yml}Once you have a NeRF model you can render its output. There are multiple different renders, more info available using
python nerfstudio/scripts/render.py --helpWe currently provide implementations for the following models:
splatad: 3DGS-based. Official implementation for SplatAD. This is currently our fastest and best performing model on AD scenes.neurad: NeRF-based. Official implementation for NeuRAD.unisim: NeRF-based. This is an unofficial of UniSim. Available as plugin, see https://github.com/carlinds/unisim for more info. Beside NeuRAD, we also provide our 3DGS-based model SplatAD.
Any of these models can be trained as described above
# Train model
python nerfstudio/scripts/train.py <model name> pandaset-dataFurther, as we build on top of nerfstudio, models such as nerfacto or splatfacto are available as well, see nerfstudio for details. However, note that these are made for static scenes.
For a full list of included models run python nerfstudio/scripts/train.py --help.
Each model contains many parameters that can be changed, too many to list here. Use the --help command to see the full list of configuration options.
python nerfstudio/scripts/train.py neurad --helpThere are four different methods to track training progress, using the viewer, tensorboard, Weights and Biases, and Comet. You can specify which visualizer to use by appending --vis {viewer, tensorboard, wandb, comet viewer+wandb, viewer+tensorboard, viewer+comet} to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps.
And that's it for getting started with the basics of NeuRAD. If you are missing some features, have a look at Planned Features to see if we have plans on implementing this. Otherwise, feel free to open an issue, or even better implement it yourself and open a PR!
If you want to add a dataset, look here. If you want to add a method, have a look here.
We have provided dataparsers for multiple autonomous driving dataset, see below for a complete list. However, your favorite AD dataset might still be missing.
To add a dataset, create nerfstudio/data/dataparsers/mydataset.py containing one dataparsers config class MyADDataParserConfig and one dataparser class MyADData. Preferrably, these inherit from ADDataParserConfig and ADDataParser, as these provide common functionality and streamline the expected format of AD data. For most datasets, it should then be sufficient to overwrite _get_cameras, _get_lidars, _read_lidars, _get_actor_trajectories, and _generate_dataparser_outputs.
| Data | Cameras | Lidars |
|---|---|---|
| 🚗 nuScenes | 6 cameras | 32-beam lidar |
| 🚗 ZOD (Annotations) | 1 camera | 128-beam + 2 x 16-beam lidars |
| 🚗 Argoverse 2 | 7 ring cameras + 2 stereo cameras | 2 x 32-beam lidars |
| 🚗 PandaSet (huggingface download) | 6 cameras | 64-beam lidar |
| 🚗 KITTIMOT (Timestamps) | 2 stereo cameras | 64-beam lidar |
| 🚗 Waymo v2 | 5 cameras | 64-beam lidar |
A brief introduction about Waymo dataparser for NeuRAD can be found in waymo_dataparser.md
Nerfstudio has made it easy to add new methods, see here for details. We have added our UniSim reimplementation as a plugin, which can be run as any other method using the ns-train command:
ns-train unisim pandaset-data --data data/pandasetand follow the instructions in the terminal.
See our UniSim repo for reference on how to add a new method as a plugin.
- Dataparser for multiple autonomous driving datasets including
- Dataparsing of lidar data (3D+intensity+time)
- Dataparsing of annotations
- Datamanager for lidar+image data
- Rolling shutter handling for ray generation
- Viewer improvements
- Lidar rendering
- Dynamic actor modifications
- NeuRAD - SOTA NeRF-based rendering method for dynamic AD scenes
- SplatAD - SOTA splatting-based rendering method for dynamic AD scenes
- 3DGS implementation supporting dynamic objects
- UniSim plug-in
- Release code
- Collaboration friendly studio for NeRFs
You can find our papers for NeuRAD and SplatAD on arXiv. You can also head over to our research blog for project pages and more papers on AD.
If you use this code or find our paper useful, please consider citing:
@inproceedings{tonderski2024neurad,
title={{NeuRAD}: Neural rendering for autonomous driving},
author={Tonderski, Adam and Lindstr{\"o}m, Carl and Hess, Georg and Ljungbergh, William and Svensson, Lennart and Petersson, Christoffer},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={14895--14904},
year={2024}
}
@inproceedings{hess2024splatad,
title={{SplatAD}: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving},
author={Hess, Georg and Lindstr{\"o}m, Carl and Fatemi, Maryam and Petersson, Christoffer and Svensson, Lennart},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2025}
}