Version Select: ROS-Noetic | ROS2-Foxy/Humble
Important
This repository continuously maintains the ROS1 C++ version, while the ROS2 and Python versions have not been updated in sync, which may result in missing or inconsistent features. Therefore, it is recommended to use the ROS1 C++ version to obtain the latest features and bug fixes.
This repository provides a framework for simulation verification and physical deployment of robot reinforcement learning algorithms, suitable for quadruped robots, wheeled robots, and humanoid robots. "sar" stands for "simulation and real"
feature:
- Built-in pre-trained models for multiple robot simulations, including
Unitree-A1,Unitree-Go2,Unitree-Go2W,Unitree-B2,Unitree-B2W,Unitree-G1,FFTAI-GR1T1,FFTAI-GR1T2,GoldenRetriever-L4W0,GoldenRetriever-L4W4; - The training framework supports IsaacGym and IsaacSim, distinguished by
framework; - The code has ROS-Noetic and ROS2-Foxy/Humble versions;
- The code has cpp and python versions, with the python version located in
src/rl_sar/scripts;
Note
If you want to train policy using IsaacLab(IsaacSim), please use robot_lab project.
The order of joints in robot_lab cfg file joint_names is the same as that defined in xxx/robot_lab/config.yaml in this project.
Discuss in Github Discussion or Discord.
Clone the code
git clone https://github.com/fan-ziqi/rl_sar.gitThis project uses ros-noetic (Ubuntu 20.04) and requires the installation of the following ROS dependency packages:
sudo apt install ros-noetic-teleop-twist-keyboard ros-noetic-controller-interface ros-noetic-gazebo-ros-control ros-noetic-joint-state-controller ros-noetic-effort-controllers ros-noetic-joint-trajectory-controller ros-noetic-joy ros-noetic-ros-control ros-noetic-ros-controllers ros-noetic-controller-managerDownload and deploy libtorch at any location (Please modify <YOUR_PATH> below to the actual path)
cd <YOUR_PATH>/libtorch
wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.0.1%2Bcpu.zip
unzip libtorch-cxx11-abi-shared-with-deps-2.0.1+cpu.zip -d ./
echo 'export Torch_DIR=<YOUR_PATH>/libtorch' >> ~/.bashrc
source ~/.bashrcInstall yaml-cpp and lcm. If you are using Ubuntu, you can directly use the package manager for installation:
sudo apt install liblcm-dev libyaml-cpp-devThis project uses the Intel TBB (Threading Building Blocks) library to implement data exchange between different threads. If you use Ubuntu, you can directly use the package manager to install it
sudo apt install libtbb-devYou can also use source code installation (Click to expand)
Install yaml-cpp
git clone https://github.com/jbeder/yaml-cpp.git
cd yaml-cpp && mkdir build && cd build
cmake -DYAML_BUILD_SHARED_LIBS=on .. && make
sudo make install
sudo ldconfigInstall lcm
git clone https://github.com/lcm-proj/lcm.git
cd lcm && mkdir build && cd build
cmake .. && make
sudo make install
sudo ldconfigCompile in the root directory of the project
cd ..
catkin buildIf you need to use ROS, you need to manually open the macro definition in rl_xxx.hpp
Note
If catkin build report errors: Unable to find either executable 'empy' or Python module 'em', run catkin config -DPYTHON_EXECUTABLE=/usr/bin/python3 before catkin build
If you don't need simulation, just run on the robot, you can compile without catkin and disable ROS:
cmake src/rl_sar/ -B cmake_build -DUSE_CATKIN=OFF
cmake --build cmake_build -j4The executable file generated by compilation is in cmake_build/bin, and the library is in cmake_build/lib.
In the following text, <ROBOT>/<CONFIG> is used to represent different environments, such as a1/isaacgym and go2/himloco.
Before running, copy the trained pt model file to rl_sar/src/rl_sar/models/<ROBOT>/<CONFIG>, and configure the parameters in <ROBOT>/<CONFIG>/config.yaml and <ROBOT>/base.yaml.
Open a terminal, launch the gazebo simulation environment
source devel/setup.bash
roslaunch rl_sar gazebo_<ROBOT>.launch cfg:=<CONFIG>Open a new terminal, launch the control program
source devel/setup.bash
(for cpp version) rosrun rl_sar rl_sim
(for python version) rosrun rl_sar rl_sim.pyKeyboard Control:
- Press to toggle the simulator between running and stopped.
- Press 0 to move the robot from the initial simulation pose to the
default_dof_posdefined in the YAML file using position control interpolation. - Press p to switch to Reinforcement Learning mode.
- Use W/S to move forward/backward, J/L to move left/right, and A/D to turn. Press to reset all control commands to zero.
- Press n to switch to Navigation mode, disabling gamepad commands and receiving commands from the
cmd_veltopic. - If the robot falls down, press R to reset the Gazebo environment.
- Press 1 to move the robot from its current position back to the initial simulation pose using position control interpolation.
Gamepad Control:
- Press LB to toggle the simulator between running and stopped.
- Press RB + Y to move the robot from the initial simulation pose to the
default_dof_posdefined in the YAML file using position control interpolation. - Press RB + B to switch to Reinforcement Learning mode.
- Use LY to move forward/backward, LX to move left/right, and RX to turn.
- Press the down button on the left to switch to Navigation mode, disabling gamepad commands and receiving commands from the
cmd_veltopic. - If the robot falls down, press RB + X to reset the Gazebo environment.
- Press RB + A to move the robot from its current position back to the initial simulation pose using position control interpolation.
Unitree A1 (Click to expand)
Unitree A1 can be connected using both wireless and wired methods:
- Wireless: Connect to the Unitree starting with WIFI broadcasted by the robot (Note: Wireless connection may lead to packet loss, disconnection, or even loss of control, please ensure safety)
- Wired: Use an Ethernet cable to connect any port on the computer and the robot, configure the computer IP as 192.168.123.162, and the netmask as 255.255.255.0
Open a new terminal and start the control program
source devel/setup.bash
rosrun rl_sar rl_real_a1Press R2 on the gamepad to switch the robot to the default standing pose, R1 to switch to Reinforcement Learning (RL) control mode, and L2 in any state to return to the initial lying-down pose. The left joystick up/down controls movement along the x-axis, left joystick left/right controls yaw, and right joystick left/right controls movement along the y-axis.
Alternatively, press 0 on the keyboard to switch the robot to the default standing pose, P to switch to RL control mode, and 1 in any state to return to the initial lying-down pose. Use W/S to control the x-axis, A/D to control yaw, and J/L to control the y-axis.
Unitree Go2/Go2W (Click to expand)
Connect one end of the Ethernet cable to the Go2/Go2W robot and the other end to your computer. Then, enable USB Ethernet on the computer and configure it. The IP address of the onboard computer on the Go2 robot is 192.168.123.161, so the computer's USB Ethernet address should be set to the same network segment as the robot. For example, enter 192.168.123.222 in the "Address" field (you can replace 222 with another number).
Use the ifconfig command to find the name of the network interface for the 123 network segment, such as enxf8e43b808e06. In the following steps, replace <YOUR_NETWORK_INTERFACE> with the actual network interface name.
Open a new terminal and start the control program. If you are controlling Go2W, you need to add wheel after the command, otherwise leave it blank.
source devel/setup.bash
rosrun rl_sar rl_real_go2 <YOUR_NETWORK_INTERFACE> [wheel]Go2/Go2W supports both joy and keyboard control, following the same method as described for A1.
Connect your computer to the robot using the Ethernet cable and log into the Jetson onboard computer. The default password is 123:
ssh unitree@192.168.123.18Check the JetPack version:
sudo pip install jetson-stats
sudo jtopDownload and install the PyTorch version that matches your JetPack:
wget https://developer.download.nvidia.cn/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
sudo apt install python-is-python3 python3.9-dev
pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whlCheck the installed torch path:
python -c "import torch; print(torch.__file__)"Manually create the libtorch directory and copy necessary files
mkdir ~/libtorch
cp -r /home/unitree/.local/lib/python3.8/site-packages/torch/{bin,include,lib,share} ~/libtorch
echo 'export Torch_DIR=~/libtorch' >> ~/.bashrc
source ~/.bashrcClone the source code and compile it as described in the previous sections.
Take A1 as an example below
- Uncomment
#define CSV_LOGGERin the top ofrl_real_a1.hpp. You can also modify the corresponding part in the simulation program to collect simulation data for testing the training process. - Run the control program, and the program will log all data after execution.
- Stop the control program and start training the actuator network. Note that
rl_sar/src/rl_sar/models/is omitted before the following paths.rosrun rl_sar actuator_net.py --mode train --data a1/motor.csv --output a1/motor.pt
- Verify the trained actuator network.
rosrun rl_sar actuator_net.py --mode play --data a1/motor.csv --output a1/motor.pt
In the following text, <ROBOT>/<CONFIG> is used to represent your robot environment.
- Create a model package named
<ROBOT>_descriptionin therl_sar/src/robotsdirectory. Place the robot's URDF file in therl_sar/src/robots/<ROBOT>_description/urdfdirectory and name it<ROBOT>.urdf. Additionally, create a joint configuration file with the namespace<ROBOT>_gazeboin therl_sar/src/robots/<ROBOT>_description/configdirectory. - Place the trained RL model files in the
rl_sar/src/rl_sar/models/<ROBOT>/<CONFIG>directory, and create a newconfig.yamlfile in this path. Refer to therl_sar/src/rl_sar/models/a1/isaacgym/config.yamlfile to modify the parameters. Then, create abase.yamlfile in the upper-level directory and refer torl_sar/src/rl_sar/models/a1/base.yamlto modify the parameters. - Modify the
forward()function in the code as needed to adapt to the specific robot model. - If simulation is required, refer to the launch files in
rl_sar/src/rl_sar/launchand modify them accordingly. - If running on the physical robot, refer to and modify the
rl_sar/src/rl_sar/src/rl_real_a1.cppfile as needed.
Wholeheartedly welcome contributions from the community to make this framework mature and useful for everyone. These may happen as bug reports, feature requests, or code contributions.
Please cite the following if you use this code or parts of it:
@software{fan-ziqi2024rl_sar,
author = {fan-ziqi},
title = {rl_sar: Simulation Verification and Physical Deployment of Robot Reinforcement Learning Algorithm.},
url = {https://github.com/fan-ziqi/rl_sar},
year = {2024}
}
The project uses some code from the following open-source code repositories:
- unitreerobotics/unitree_guide
- mertgungor/unitree_model_control
- The code in src/rl_sar/scripts/actuator_net.py is modified from scripts/actuator_net in the Improbable-AI/walk-these-ways repository.