[RLlib] Upgrade RLlink protocol for external env/simulator training. #53550
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR upgrades the RLlink protocol for external environment/simulator training by deprecating legacy example scripts and refactoring internal APIs to use the new rllink protocol and associated message‐packing via msgpack.
- Removal of outdated Unity3D and CartPole example scripts.
- Refactoring of annotations (e.g., replacing @publicapi with @OldAPIStack) and protocol messaging (using new RLlink API in rllink.py).
- Introduction of new modules (rllink.py and rllib_gateway.py) and updates in algorithm_config.py.
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| rllib/examples/envs/external_envs/*.py | Removed outdated example scripts for external env training. |
| rllib/env/wrappers/unity3d_env.py | Updated API annotation and trimmed outdated docstrings. |
| rllib/env/utils/external_env_protocol.py | Redirected RLlink import to new rllink implementation. |
| rllib/env/tcp_client_inference_env_runner.py | Updated message functions to use new RLlink API. |
| rllib/env/policy_server_input.py, rllib/env/policy_client.py | Removed legacy docstrings and updated annotations. |
| rllib/env/external/rllink.py, rllib/env/external/rllib_gateway.py | New modules for upgraded protocol and external gateway. |
| rllib/algorithms/algorithm_config.py | Minor comment fix and additional handling for spaces. |
Comments suppressed due to low confidence (1)
rllib/algorithms/algorithm_config.py:327
- Typo in comment: 'reaplce' should be corrected to 'replace'.
# TODO (sven): Once new ormsgpack system in place, reaplce the string
simonsays1980
approved these changes
Jun 5, 2025
| if isinstance(env, gym.vector.VectorEnv): | ||
| rl_module_spec.observation_space = env.single_observation_space | ||
| rl_module_spec.action_space = env.single_action_space | ||
| if rl_module_spec.observation_space is None: |
There was a problem hiding this comment.
Dumb question: Can't we reuse the self.obs_space, self.action_space here?
rllib/env/external/rllib_gateway.py
Outdated
| self._prev_action = None | ||
| self._prev_extra_model_outputs = None | ||
|
|
||
| def _connecto_to_server_thread_func(): |
There was a problem hiding this comment.
_connecto_... -> _connect_to_...? :)
| next_observation: The current observation, from which the action should be | ||
| computed. Note that first, `observation`, the previously returned | ||
| action, `prev_reward`, and `terminated/truncated` are logged with the running | ||
| episdode through `Episode.add_env_step()`, then the env-to-module |
There was a problem hiding this comment.
"epsidode" -> "episode"
Signed-off-by: sven1977 <svenmika1977@gmail.com>
…ade_rllink_protocol
…ade_rllink_protocol
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upgrade RLlink protocol for external env/simulator training.
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.