Add UFACTORY 850 as one of the robots supported, introduce various HRI utilities for teleop and next episode hinting while reseting #493
Conversation
|
Wow really cool! We are in the middle of a refactor. We will come back to this PR soon. In the meantime, I am curious if other people working on UFACTORY 850 are able to use this arm with LeRobot :) |
|
Let me know @Cadene if you'd like other flavours of xArm robots tested. I'm rather certain it will, as the UFACTORY API is rather well structured and known reproducible. I've got an |
|
@vmayoral Sounds good ;) Any chance you could run |
| @@ -497,15 +515,21 @@ def teleop_step( | |||
| # Cap goal position when too far away from present position. | |||
| # Slower fps expected due to reading from the follower. | |||
| if self.config.max_relative_target is not None: | |||
| present_pos = self.follower_arms[name].read("Present_Position") | |||
| if self.robot_type == "u850": | |||
There was a problem hiding this comment.
This approach of adding if/elif throughout the codebase doesn't seem scalable.
What do folks think about creating an abstract base class that all robots can implement their wrapper API's within, so we don't have to edit so much code in a brittle way to add new robot support?
I can take this on- I'm currently working on adding support for the elephant robotics MyArmM&C series.
The following abstract class would describe the current robot supported by manipulator.py, but personally I don't like the API and I think it's wayyy too specific to one robot:
from abc import ABC, abstractmethod
class BaseRobotWrapper(ABC):
@abstractmethod
def read(self, cmd: Literal["Present_Position]) -> npt.NDArray[np.float64]:
pass
@abstractmethod
def write(self, cmd: Literal["Operating_Mode", "Torque_Enable", "Goal_Position", "Lock", "Mode", "P_Coefficient", "I_Coefficient", "D_Coefficient", "Acceleration", "Maximum_Acceleration"], val: int, motors: str | list[str] | None=None) -> None:
pass
Something more generic could be made, leaving the API high level:
class BaseRobotWrapper(ABC):
@abstractmethod
def read_angles(self) -> npt.NDArray[np.float64]:
"""Read arbitrary number of joints to the robot. Mirrors write_angles"""
@abstractmethod
def write_angles(self, angles: npt.NDArray[np.float64]) -> None:
"""Write arbitrary number of joints to the robot. Mirrors read_angles"""
@abstractmethod
def set_follower_presets() -> none:
"""This would replace:
- set_koch_robot_preset
- set_aloha_robot_preset
- set_so100_robot_preset
- ...
"""
@abstractmethod
def set_leader_presets() -> None:
pass
@abstractmethod
def reset() -> None:
"""Used when 'reset_environment' is called. Optional to implement."""There was a problem hiding this comment.
@apockill, I've started reviewing my code following @Cadene's requests above, but I also like your suggestion and would be happy to work together on this. Frankly, an abstraction layer such as the one you propose would facilitate future ports, but I'm unsure if this is something desired at all, so I just took the most straightforward approach for my weekend of hacking.
Nevertheless, I think your suggestion is needed and a code rebase is probably the best way to do it. Happy to participate on that as well. For now, here, I'll focus on my initial intent and contribution.
There was a problem hiding this comment.
Okay, sounds good! Maybe let's keep things as is, once xarm / myarm support is merged in I can jump in and write wrappers for all the currently supported robots.
The only hard thing will just be to have everyone test their bots out for regressions.
- Added extensions to manipulator to support 850 - Create Motors' UFactory abstraction - Proposed a new example demonstrating local setup Signed-off-by: Víctor Mayoral Vilches <v.mayoralv@gmail.com>
Signed-off-by: Víctor Mayoral Vilches <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
This is useful while recording many episodes, since you just loose track of them and the verbose outputs from the teleop don't allow for clarity on this regard Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Also, it's a derivative of the previous ones Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
Signed-off-by: vmayoral <v.mayoralv@gmail.com>
@Cadene all ready concerning these requests. Should be good to go as of right now. Just rebased on top of |
|
Ping @Cadene and team. Let me know if you folks require anything else. |
|
Friendly ping @Cadene. Let me know if you're missing anything. |
Include an example called cooper.yaml which exemplifies the use of Gello with a bimanual setup Signed-off-by: vmayoral <v.mayoralv@gmail.com>
@vmayoral Thank you for this! We got the entire thing working on two xARM 850's. As you mentioned the teleop is quite jerky via control_robot.py or the 1_teleop_lerobot. Any idea(s) on where we should look to fix this? |
|
A few:
curious about your use case. Feel free to reach out and if you can elaborate more on your use, happy to find some time and help. |
Currently just trying to figure out if we can use an xARM 850 in an imitation learning framework (btw cool office:-)). If we get it working, we want to use both xARM's to move things around in a home like environment (dual arm config). Commenting out the max_relative_target seems to make the teleop OK but then replay still has issues.... Short video of replaying an episode here (https://youtube.com/shorts/Fzs2_MnPVe0) |
This PR comes as a result of the Oct 2024 Hackathon wherein I participated remotely. Results were successful within the weekend of hacking, so I thought I'd give back and contribute. Results first:
lerobotteleop operational in UFACTORY 850topwristThe contributions of this PR are summarized below:
xArmWrapperof the UFACTORY Python SDK, usable with other arms from the same vendormanipulator.pywith minimal changes (attempted respecting the already existing structure) to support the new arm testedSome notes and considerations: