English| 简体中文
RDK Model Zoo S is developed based on RDK S Series, providing deployment routines for most mainstream algorithms. These routines include exporting D-Robotics *.bin models, and using APIs such as Python to infer D-Robotics *.bin models. Some models also include data collection, model training, exportation, transformation, and deployment processes.
Currently, RDK Model Zoo offers reference models of the following types.
samples/
|-- LLM
| `-- DeepSeek_R1_Distill_Qwen
|-- Multi-model
| `-- InternVL2
|-- Planning
| `-- LeRobot_ACTpolicy
| `-- walk-these-ways-go2
| `-- ASAP
|-- Speech
| |-- ASR
| |-- KWS
| `-- TTS
`-- Vision
|-- ResNet
|-- YOLOv5_Detect
|-- ultralytics_YOLO_Detect
|-- ultralytics_YOLO_Pose
`-- ultralytics_YOLO_SegRDK Model Zoo S supports the following platforms.
- Supports RDK S100 / RDK S100P platform (Nash)
RDK Model Zoo supports the following platforms.
GitHub: https://github.com/d-Robotics/rdk_model_zoo
Recommended System Version
- RDK S100: RDK OS 4.0.0, Based on Ubuntu 22.04 aarch64, TROS-Humble.
Refer to the RDK User Manual to ensure that the board can normally access the internet, ensuring one of the following conditions is met.
- Use SSH to connect to the RDK board, input commands through Terminal knowing the IP address of the RDK board. Including but not limited to MobaXtern, Windows Terminal, etc.
- Utilize VSCode Remote SSH plugin to remotely connect to the RDK board, use VSCode normally, or use other IDEs.
- Access the board via VNC, operate the board through the xfce graphical interface.
- Connect to the board via HDMI, operate the board through the xfce graphical interface.
- Please check if both the Docker version of the OpenExplore toolchain and the version of libdnn.so on the board are the latest released versions.
- When exporting models, ensure that you follow the requirements specified in the README files within the corresponding examples folders.
- Ensure that the cosine similarity of each output node reaches above 0.999 (a minimum of 0.99).
- The inference performance using Python APIs is weaker; please test performance using C/C++ APIs.
- Performance data does not include preprocessing and postprocessing steps, which differ from the total time taken by complete demos. Generally, models with nv12 input can achieve end-to-end throughput equal to BPU throughput.
- Ensure that the board has been set to the highest frequency specified in the corresponding README.
- Check if other applications are occupying CPU/BPU and DDR bandwidth resources, as this can degrade inference performance.
- According to the platform version, first refer to the relevant documentation for the specific platform, particularly the PTQ section on precision debugging, to debug precision issues.
- If int8 quantization leads to a loss of precision due to model structure characteristics or weight distribution, consider using hybrid quantization or QAT quantization.
Please modify the resolution in the preprocess.py file located in the same directory to match the resolution of the ONNX model you intend to convert. Delete all calibration datasets and rerun script 02 to regenerate the calibration dataset. Currently, the calibration dataset for this example comes from the ../../../01common/calibration data/coco directory and is generated in the ./calibration_data_rgb_f32 directory.
Hello, no.
- Due to project scheduling constraints, we have selected frequently asked models as demo examples to cater to the needs of most D-Robotics developers. If you have better model recommendations, feel free to provide feedback on the D-Robotics Developer Community.
- Additionally, BPU and algorithm toolchain-related resources have already been released on the developer community, allowing custom models to be converted independently.
- Ultralytics measures mAP using dynamic shape models, whereas BPU uses fixed shape models, resulting in lower mAP test results compared to dynamic shapes.
- It's normal for the precision calculated by RDK Solutions using pycocotools to be slightly lower than that calculated by Ultralytics due to minor differences in calculation methods. We focus on using the same calculation method to test the precision of quantized and floating-point models to assess precision loss during quantization.
- After quantization and converting NCHW-RGB888 input to YUV420SP(nv12) input, BPU models also experience some precision loss.
Yes, but it's not recommended.
- For public model structures or self-designed output head structures, you need to design the post-processing code yourself.
- The adjustments made to model structures in the RDK Solutions repository have been considered for both precision and performance. Other modifications have not been tested yet, but we welcome explorations of more high-performance and high-precision modification methods.
Under the PTQ scheme, models need to be exported to ONNX or Caffe first, then converted to bin models. Under the QAT scheme, a new Torch model needs to be built for quantization-aware training, converting from pt models to hbm models.
During training, follow the public version entirely and only modify when exporting. This ensures that the training loss function calculations align, and deployment will match the post-processing code running on the board.
During model conversion, operators that cannot be quantized or do not meet BPU constraints will fallback to CPU computation. Specifically, for a fully BPU operator-based bin model, there will be quantization and dequantization nodes before and after the model, responsible for converting float to int and vice versa, which are computed by the CPU.