Description
What is the bug?
Hi Team,
Whenever we scale down or terminate an ingester, it transitions to the LEAVING state and then becomes UNHEALTHY without being removed from the ring until we manually forget the ingester from the ring. Even though pod has been completely terminated, resulting in write and read failures.
Same behavior observed with unregister_on_shutdown to true as well.
Mimir Version - 2.14.x and 2.15.x
Attaching screenshots for reference
Ingester Terminating
ACTIVE to LEAVING state in the ring
Ingester pod is completely terminated
As we can see though zone-b-2 pod is completely terminated, ingester is still stuck in UNHEALHTY state
Kindly help the fix this issue
How to reproduce it?
- Deploy Mimir version 2.14.x or 2.15.x.
- Scale down or terminate the ingesters.
- Observe the ingester ring—upon termination, the ingester transitions from ACTIVE to LEAVING, and after 10 minutes, it moves to the UNHEALTHY state, even though the pod has been fully terminated.
What did you think would happen?
Setting unregister_on_shutdown to true doesn't seem to have any effect—we're observing the same behavior regardless of its value.
What was your environment?
Kubernetes - Helm - Mimir 2.14.x and 2.15.x
Any additional context to share?
No response