8000 M3DB nodes stuck in bootstraping and restarted with OOM signal (many large commitlogs) · Issue #4099 · m3db/m3 · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
M3DB nodes stuck in bootstraping and restarted with OOM signal (many large commitlogs) #4099
Open
@pahla1

Description

@pahla1

Hi all,

General Issues

  1. What service is experiencing the issue? M3DB
  2. What is the configuration of the service? https://pastebin.pl/view/7e4b79e2
  3. How are you using the service? via Prometheus
  4. Is there a reliable way to reproduce the behavior? high cardinal data with large block size and small size RAM.

We have 3 nodes M3DB cluster, all 3 of which are restarting due to OOM. During this time a large number of commitlogs are being generated in large volumes which has made the bootstrapping operation very slow and looped.

Is there a way to bootstrap the cluster and then apply commitlogs manually?
Or move commitlogs to another folder and return it to the commitlogs folder after bootstrap to be applied by m3db?

Any help appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0