Mimir Distributor OOM #10805
Replies: 3 comments 1 reply
-
|
Any update on this please? |
Beta Was this translation helpful? Give feedback.
-
|
Converting this to a discussion, as there's no Mimir bug AFAICT. |
Beta Was this translation helpful? Give feedback.
-
|
I also noticed that e.g. during deployments and ingester restart distributors consume more memory. I think that's unavoidable and you just need add extra memory to distributors. |
Beta Was this translation helpful? Give feedback.
-
|
We have updated the memory as recommended, we will monitor and update if there are any changes. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi Team,
We are landing in distributors OOMKilled continuously, issue is causing due to one ingester silent restart
Distributor logs
ts=2025-02-28T15:13:24.644631208Z caller=pool.go:250 level=warn msg="removing ingester failing healthcheck" addr=10.128.54.5:9095 reason="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 10.128.148.12:47456->10.128.54.5:9095: read: connection reset by peer""
ts=2025-02-28T15:13:33.722860622Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'esobb-mimir-distributor-enterprise-fc74dcccb-2qwsk-b94240a0' from=10.128.143.7:7946"
Complete write path is going down leading to data loss, can someone please provide a solution for this.
Beta Was this translation helpful? Give feedback.
All reactions