Reliable FogROS2 is a cloud robotics platform on FogROS2-SGC for connecting disjoint ROS2 networks across different physical locations, networks, and Data Distribution Services.
Table of Contents
The certificates can be generated by
cd sgc_launch/configs
./generate_crypto.sh
You need to make sure all the machines that you intend to connect share the same certificate. You can check if the certificates match by running
cat /fog_ws/install/sgc_launch/share/sgc_launch/configs/crypto/test_cert/test_cert-private.pem
The default certificate is named test_cert.
docker build . -t keplerc/fogros2-rt-router:latest -f release.Dockerfile
docker run --net=host -it keplerc/fogros2-rt-router:latest bash -c "source ./install/setup.sh && RMW_IMPLEMENTATION=rmw_cyclonedds_cpp ros2 run sgc_launch sgc_router --ros-args -p config_file_name:=service-sam.yaml -p whoami:=machine_server"
you need to change the config file name and whoami associated with your config file.
You can add your custom message types to examples/shared_srvs or any subfolder in examples.
Run signaling server by
docker run --net=host -it keplerc/fogros2-rt-router:latest bash -c "cd rt-fogros2/fog_rs/fogrs-signaling && RUST_LOG=info cargo run"
The server will be automatically hosted at port 8000. By default, the routing records are cleaned up by 90 seconds.
The certificates can be generated by
cd sgc_launch/configs
./generate_crypto.sh
You need to make sure all the machines that you intend to connect share the same certificate.
You can check if the certificates match by running
cat /fog_ws/install/sgc_launch/share/sgc_launch/configs/crypto/test_cert/test_cert-private.pem
The default certificate is named test_cert.
Build container with the following command:
docker build -f dev.Dockerfile --build-arg USER_ID=$(id -u) --build-arg GROUP_ID=$(id -g) --build-arg USER_NAME="$USER" -t dev-fr .then start two terminals, for both, run
docker run --user $(id -u):$(id -g) -ti --net=host -v ~/rt-fogros2:/fog_ws/rt-fogros2 dev-fr bash The parameters such as user ID and group are used to make sure the built targets are readable and modifiable by the host.
RUST_LOG=info cargo run
On each terminal, run
export RUST_LOG=info && export ROS_DOMAIN_ID=3 && source install/setup.bash && ros2 launch bench talker.launch.py
and
export RUST_LOG=info && export ROS_DOMAIN_ID=2 && source install/setup.bash && ros2 launch bench listener.launch.py
export RUST_LOG=info && export ROS_DOMAIN_ID=4 && source install/setup.bash && ros2 launch bench add.three.ints.service.launch.py
export RUST_LOG=info && export ROS_DOMAIN_ID=3 && source install/setup.bash && ros2 launch bench add.three.ints.service.launch.py
export RUST_LOG=info && export ROS_DOMAIN_ID=2 && source install/setup.bash && ros2 launch bench add.three.ints.client.launch.py
- If the lisenter side is not responsive (no logs show up) when the talker launches, double check the certificate. Sometimes it gets tricky with
/r/nand/nor whether the file ends with a newline character (the editor may do something stupid). Double check with
od -c /fog_ws/install/sgc_launch/share/sgc_launch/configs/crypto/test_cert/test_cert-private.pem
-
If the listner is responsive but they don't connect, check the internet firewall rules. We require one side of the machines are connectable via UDP (LMK if you want specific ports)
-
too many redis logs. Currently we haven't implemented automatic cleanup, but you can cleanup manually via
redis-cli -h 3.18.194.127 -p 8002
> FLUSHDB
The following are instructions of setting up FogROS2 SGC.
sudo apt update
sudo apt install build-essential curl pkg-config libssl-dev protobuf-compiler clang
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
ROS2 Dashing Eloquent Foxy Galactic Humble Rolling should work fine with FogROS2 SGC.
Here we show the instruction of installing ROS2 rolling with Debian packages.
First, install ROS2 from binaries with these instructions.
Setup your environment with these instructions.
Every terminal should be configured with
source /opt/ros/rolling/setup.bash
If you have custom types in a specific ROS colcon directory, source the setup.bash in that directory.
Either you run it as a ROS node, or as a standalone process, you need to generate a certificate unique to the task as the globally unique identifier. This set of certificate should be shared by all the robots that you wish to be connected.
The certificates can be generated by
cd sgc_launch/configs
./generate_crypto.sh
Every directory in ./sgc_launch/configs/crypto contains the cryptographic secrets needed for communication.
Distribute the crypto directory by from machine A and machine B. This can be done with USB copying, typing or SSH. Here is an example with scp:
scp -r crypto USER@MACHINE_B_IP_ADDR:/SGC_PATH/sgc_launch/configs/
replace USER, MACHINE_B_IP_ADDR, SGC_PATH with the actual paths.
After the crypto secrets are delivered, go back to project main directory.
cd ~
mkdir -p fog_ws/src
cd ~/fog_ws/src
git clone https://github.com/KeplerC/fogros2-sgc.git
cd ~/fog_ws
colcon build
Please make sure that the repo is cloned directly under the src directory.
Machine A:
ros2 launch sgc_launch talker.launch.py
Machine B:
ros2 launch sgc_launch listener.launch.py
Note that machine A and machine B do not need to configure any IP address, and they automatically connect.
Please refer to the README for writing the launch file for your application. FogROS2-SGC supports automatic topic discovery, but it is recommended to expose the public interface only if intended.
The repo can be built with
cargo run router
If you want to deploy with production system, use cargo build --release option for optimization level and removal of debug logs.
To disable the logs and run with high optimization, run with release option by
cargo run --release router
instead.
Adding and removing topics requires REST API. Example of such is
curl -X POST http://localhost:3000/topic \
-H 'Content-Type: application/json' \
-d '{"api_op":"add","ros_op":"pub", "crypto":"test_cert", "topic_name":"/chatter", "topic_type":"std_msgs/msg/String"}'
The port for the REST interface can be changed via SGC_API_PORT environment variable.
The configuration is currently coded with a Berkeley's signaling server that facilitates the routing inforamtion exchange. See Making your own signaling server section for creating your own signaling server. The system should also work if you don't specify the configuration file, then it uses automatic mode that constantly checking for new topics. We note that it is only a convenient interface and not FogROS2-SGC is designed for. As long as the talker and listener use the same crypto, the system should work.
In the updated version, we removed all the setup about protocols, ports, ips and gateways. Previous FogROS2-SGC carries a bag of protocols to support heterogenous demands and requirements. In this version, we streamline the routing setup by webrtc instead of building all protocols with raw DTLS sockets. webrtc is generally not compatible with the previous protocol.
Berkeley's public servers are for experimental purposes and do not intend to be used for production system. We do not provide any guarantee on avaialbility. Please use your own signaling server for benchmarking and deployment. The security guarnatees of FogROS2-SGC prevents other users/Berkeley from learning sensitive information, such as your ROS2 topic name and type, and on the actual data payload. What is visible is a random 256 bit name are published and subscribed by other random 256 bit names.
If you don't want to use Berkeley's infrastructure, having one on your own is very easy. This can be done by running
docker compose up -d signal rib
The only requirement is to expose port 8000 and 8002 to other robots/machines.
Signaling server faciliates the communication by exchanging the address information of webrtc. The details about how signaling server works can be found HERE. Then update the config files by replacing the IP address to your server's IP address.
- expiration time for stale keys (this may happen if the subscriber suddenly drops off and does not connect to an existing publisher)
- in some rare cases, the IP address and port provided cannot connect, which blocks the publisher and subscriber. The common root cause is that the firewall bans the ports larger than 50000, which may happen in some enterprise or restrictive settings. If you have one side of the machine (like cloud) that opens the port greater than 50000, it should be able to connect.