Open
Description
What happened + What you expected to happen
I'm getting this error, but this error is not easy to understand and it is easy to miss.
[2022-12-20 03:13:08,669 I 25449 25449] global_state_accessor.cc:357: This node has an IP address of 11.22.33.44, while we can not find the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
Can we improve it in the following way:
- What are the detected IP addresses on the node?
- What are the detected Raylet addresses, if there exists any?
- What can the user do to reconfigure Ray to start properly?
- [optional] Should we fail to start Ray if this happens?
An example idea to improve this error message.
[2022-12-20 03:13:08,669 I 25449 25449] global_state_accessor.cc:357: WARNING: Ray is configured to use the ip address 11.22.33.44 for this node but cannot find a corresponding Raylet address (detected raylet addresses of {11.22.33.55}), which may cause undefined behavior. This node has the following ip addresses: {11.22.33.44, 11.22.33.45, 11.22.33.55}. Try setting the Ray IP address by using `ray start --node-ip-address=CHOSEN_IP_ADDRESS ...`.
Versions / Dependencies
master
Reproduction script
run ray start --address=... on a node with multiple ip addresses
Issue Severity
High: It blocks me from completing my task.