8000 Enable control of the follower arm with PS4 joystick by IliaLarchenko · Pull Request #516 · huggingface/lerobot · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Enable control of the follower arm with PS4 joystick #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Enable control of the follower arm with PS4 joystick #516

wants to merge 14 commits into from

Conversation

IliaLarchenko
Copy link
Contributor

What this does

You can control the follower arm using a PlayStation 4 controller, which can be useful for people who don't have a leader arm or for anybody who wants to try an alternative way of teleoperation.

  • Control all action of the follower arm with the PS4 joystick
  • Can be used for dataset collection
  • Assign fixed positions to any buttons for quick movements
  • Has vibration and red color indication when motor position limits are reached
  • Can even control gripper position using controller IMU (don't think this one is useful, just fun)

Limitations:
The controller should be connected via USB (Bluetooth can require some extra fixes)

I added a new type of device - controllers. I think it makes sense in the long run to have them as a separate device type.

How it was tested

I tested it on Mac and Ubuntu and only with SO-ARM100
You can see the results here: https://www.youtube.com/shorts/Ini2DATLawU

How to checkout & try? (for the reviewer)

It requires hid to run, I didn't add it to the dependencies because it is not needed for most of the users who don't have PS4. So, if you want to try you will need to install it yourself.

I created a new config so100_ps4, adjust the port of the follower arm, connect PS4 joystick via USB and run:

python lerobot/scripts/control_robot.py teleoperate \
    --robot-path lerobot/configs/robot/so100_ps4.yaml \
    --robot-overrides '~cameras' \
    --display-cameras 0

@Cadene
Copy link
Collaborator
Cadene commented Nov 22, 2024

Interesting design! Thanks for your PR. We would like to include it after our improvements of the robot devices API.

@mydhui
Copy link
mydhui commented Dec 11, 2024

@IliaLarchenko Impressive！May I ask some questions,

  1. If I only have ps5 controllers available, can it seamlessly be applied?
  2. Does it support bluetooth connection?
  3. Is it efficient in this way for teleoperation (in terms of data quality)

@IliaLarchenko
Copy link
Contributor Author

@mydhui
Unfortunately, it is not so easy.

  1. The PS5 controller has a different structure of packets it sends/receives. It shouldn't work out of the box. But I don't have a PS5 controller and haven't tested it. My guess is it is straightforward to adapt it to PS5, but I didn't try it.
  2. For some reason the data structure when the controller is connected via Bluetooth is not the same as for USB. But it is very similar and if you comment out the accelerometer and feedback part it will work (though when I tested it I saw much higher latency and I would recommend using a USB connection).
  3. You can teleoperate the arm using a controller and do pretty much anything, but it feels very different from teleoperation using the leader's arm. In some cases, it is even a bit more convenient, but generally, it looks more artificial when you use the controller. But anyway, the data is stored in the same format, you can train some policy on top of it and so on.

@danielholl13
Copy link

Good job!
To get it to work with my PS4 controller I had to set the PRODUCT_ID to 0x5C4.

@L3SAIL3S
Copy link

hid is necessary on Mac OS ?

@IliaLarchenko
Copy link
Contributor Author

Yes. You need to install hid to run it on any OS

@L3SAIL3S
Copy link

Yes. You need to install hid to run it on any OS

thank you, all it's install, now I have
control_robot.py: error: unrecognized arguments: teleoperate --robot-path lerobot/configs/robot/so100_ps4.yaml --robot-overrides ~cameras --display-cameras 0

@IliaLarchenko
Copy link
Contributor Author

Are you sure you are using the correct branch? The current main branch uses different parameters and you probably using it instead.

You should use this one: https://github.com/IliaLarchenko/lerobot/tree/joystick

@L3SAIL3S
Copy link
L3SAIL3S commented Apr 16, 2025 •
edited
Loading

It's, now I use the correct branch, but I have : hydra.errors.InstantiationException: Error in call to target 'lerobot.common.robot_devices.controllers.joysticks.PS4JoystickController':
AttributeError("module 'hid' has no attribute 'Device'")
full_key: controller

@IliaLarchenko
Copy link
Contributor Author

I guess you installed pyhidapi that is another hid wrapper. Remove it and instal hidapi:
pip install hid
https://pypi.org/project/hid/

@imstevenpmwork imstevenpmwork added enhancement Suggestions for new features or improvements robots Issues concerning robots HW interfaces labels Apr 17, 2025
@L3SAIL3S
Copy link

I guess you installed pyhidapi that is another hid wrapper. Remove it and instal hidapi: pip install hid https://pypi.org/project/hid/

I checked, I have this:
Warning: hidapi 0.14.0 is already installed and up-to-date.
To reinstall 0.14.0, run:
brew reinstall hidapi

And always this problem:
hydra.errors.InstantiationException: Error locating target 'lerobot.common.robot_devices.controllers.joysticks.PS4JoystickController', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: controller

For information I'm on Mac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement Suggestions for new features or improvements robots Issues concerning robots HW interfaces
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@IliaLarchenko @Cadene @mydhui @danielholl13 @L3SAIL3S @imstevenpmwork
0