Open
Description
Describe your feature request
cc @ericl This is to fix those too many open file issues.
We should track the number of connection and try to reduce them. It is very typical that gcs_server has more than 500~600 open files, and when the cluster gets bigger, it can be worse easily. This always caused errors like too many open files. We should
- Use shared pool whenever possible to reduce the number of duplicating grpc client
- Aggressively remove clients that are not needed.
- Track the number of connections to detect the leak (@ffbin will create a PR to report internal state of gcs server soon).
- Improve doc to encourage people to use ulimit.
- Raise a better message (write an action item instead of failing with too many open files)
More aggressive thing we can do is to use some sort of proxy to limit the number of ports for inter-node communication (ex, kubeproxy), but it is a little unrealistic to do this in a short time.