8000 feat: monitor if sim and robotic stack processes are running by MagdalenaKotynia · Pull Request #445 · RobotecAI/rai · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: monitor if sim and robotic stack processes are running #445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 10 commits into from
Closed

feat: monitor if sim and robotic stack processes are running #445

wants to merge 10 commits into from

Conversation

MagdalenaKotynia
Copy link
Member
@MagdalenaKotynia MagdalenaKotynia commented Feb 28, 2025 •
edited
Loading

Purpose

To monitor whether the sim and robotic stack processes are working in case the binary or robotic stack processes are unexpectedly terminated.

Proposed Changes

Implemented the _monitor_processes method that is run as a thread monitoring the required processes in the background. On the O3DExROS2Bridge example, when a problem with a process is detected, SIGINT is sent to the main script process and then the simulation bridge, connector, and ros2 are shut down.

Added minor fix of setup_scene method in 39e7541, related to bbc6bc2.

Issues

To handle the unexpected termination of the binary and robotic stack processes (like this #435 (comment)).

Testing

  1. Setup the repository
poetry install --with openset
colcon build --symlink-install
source setup_shell.sh

  1. Click to download GameLauncher binary from s3 bucket: humble or jazzy.

  2. Populate src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml with the following content and adjust the path to binary:

binary_path: /path/to/your/GameLauncher
level: RoboticManipulationBenchmark
robotic_stack_command: ros2 launch examples/manipulation-demo-no-binary.launch.py
required_simulation_ros2_interfaces:
  services:
    - /spawn_entity
    - /delete_entity
  topics:
    - /color_image5
    - /depth_image5
    - /color_camera_info5
  actions: []
required_robotic_ros2_interfaces:
  services:
    - /grounding_dino_classify
    - /grounded_sam_segment
    - /manipulator_move_to
  topics: []
  actions: []

Populate the following script in the rai root dir, and run it.

import rclpy
import logging
import time
from pathlib import Path

from rai_sim.o3de.o3de_bridge import O3DExROS2Bridge, O3DExROS2SimulationConfig
from rai.communication.ros2.connectors import ROS2ARIConnector


if __name__ == "__main__":
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
    )
    logger = logging.getLogger("O3DExROS2-Sim")
    
    o3de = None
    connector = None
    monitor = None
    
    try:
        rclpy.init()
        
        connector = ROS2ARIConnector()
        o3de = O3DExROS2Bridge(connector, logger=logger)
            
        scene_config = O3DExROS2SimulationConfig.load_config(
            base_config_path=Path("src/rai_bench/rai_bench/o3de_test_bench/configs/1a_1t.yaml"), 
            connector_config_path=Path("src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml")
        )
        
        o3de.setup_scene(scene_config)

        time.sleep(100)
        
        scene_config = O3DExROS2SimulationConfig.load_config(
            base_config_path=Path("src/rai_bench/rai_bench/o3de_test_bench/configs/1carrot_1a_1t_1bc_1corn.yaml"), 
            connector_config_path=Path("src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml")
        )
        o3de.setup_scene(scene_config)
        time.sleep(100)

    except KeyboardInterrupt:
        logger.info("Received KeyboardInterrupt.")
    except Exception as e:
        logger.exception(f"Error occurred: {e}")
        raise e
        
    finally:
        logger.info("Shutting down script...")

        if o3de:
            o3de.stop_monitoring()
            logger.info("Shutting down O3DE bridge")
            o3de.shutdown()
        
        if connector:
            logger.info("Shutting down connector")
            connector.shutdown()
        
        logger.info("Shutting down ROS2")
        rclpy.try_shutdown()
        
        logger.info("Cleanup complete")

Tested cases

I tested it by:

  • shutting down the binary window with close button
  • resizing the binary window and clicking Force Quit when binary window shows message: "RAIManipulationDemo" is not responding.
  • killing processes with binary and with robotic stack through terminal.
  • killing single nodes (grounded_sam, grounding_dino, robot_state_publisher, static_transform_publisher)

Note

I am going to add example script in README in separate PR #468 when this PR and #452 are merged. There are changes related to binary structure (possibility to use levels) in #452 and taking into account changes from both PRs to update README will be needed.
In #468 connector_config_path is also going to be changed to bridge_config_path to fit the changed naming convention.

Note

If the changes from this PR are accepted and #452 is merged, I suggest using monitoring in the o3de_test_benchmark_script.py as well.

Base automatically changed from feat/simbench to development March 7, 2025 11:59
@MagdalenaKotynia MagdalenaKotynia force-pushed the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch 2 times, most recently from f4a3e20 to 5c39152 Compare March 24, 2025 11:37
@MagdalenaKotynia MagdalenaKotynia marked this pull request as ready for review March 24, 2025 11:46
@MagdalenaKotynia MagdalenaKotynia mentioned this pull request Mar 24, 2025
Closed
@MagdalenaKotynia MagdalenaKotynia marked this pull request as draft March 24, 2025 15:56
@MagdalenaKotynia MagdalenaKotynia force-pushed the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch from f468252 to 0d81771 Compare March 25, 2025 20:52
@MagdalenaKotynia MagdalenaKotynia marked this pull request as ready for review March 25, 2025 20:56
@MagdalenaKotynia MagdalenaKotynia force-pushed the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch from 6901993 to a4476c5 Compare March 25, 2025 21:46
@MagdalenaKotynia MagdalenaKotynia force-pushed the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch from a4476c5 to 12b085c Compare March 26, 2025 12:34
@MagdalenaKotynia MagdalenaKotynia requested review from boczekbartek and removed request for maciejmajek March 26, 2025 16:57
@MagdalenaKotynia MagdalenaKotynia marked this pull request as draft March 26, 2025 17:01
@MagdalenaKotynia
Copy link
Member Author

Converted back to draft because minor change may be needed due to change from bbc6bc2 that was merged today to development

@MagdalenaKotynia
feat: add monitoring if sim and robotic stack processes are working
94da5d9
@MagdalenaKotynia
refactor: extracted functionality related to monitoring processes to …
647df53 
…separate class, modified monitor_process method by sending sigint to main process
@MagdalenaKotynia
chore: removed logging info left after debugging
5f54282
@MagdalenaKotynia
chore: remove unused ProcessMonitorError Exception
f9ed252
@MagdalenaKotynia
refactor: moved processes monitoring functionalities to SimulationBri…
e93c6b6 
…dge class
@MagdalenaKotynia
feat: monitor child processes of robotic stack to monitor ros2 nodes
12cc17f
@MagdalenaKotynia
fix: removed converting child process to psutil.Process because it al…
f091ca1 
…ready is psutil.Process
@MagdalenaKotynia
refactor: psutil.Process import refactor
3e1ebd4
@MagdalenaKotynia
test: adjust launch_binary and launch_robotic_stack tests
09481fb
@MagdalenaKotynia
fix: adjusted scene setup after adding level to the binary
39e7541 
Related to the change in bbc6bc2.
@MagdalenaKotynia MagdalenaKotynia force-pushed the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch from 12b085c to 39e7541 Compare March 27, 2025 09:33
@MagdalenaKotynia MagdalenaKotynia marked this pull request as ready for review March 27, 2025 09:42
@jmatejcz jmatejcz mentioned this pull request May 15, 2025
Open
@jmatejcz
Copy link
Contributor

closing this PR, the issue that it tried to resolve -> #571

@jmatejcz jmatejcz closed this May 15, 2025
@maciejmajek maciejmajek deleted the feat/monitor-if-sim-and-robotic-stack-processes-are-running branch June 6, 2025 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boczekbartek boczekbartek Awaiting requested review from boczekbartek

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MagdalenaKotynia @jmatejcz
0