vm.max_map_count growing steadily when vm.overcommit_memory is 2 #1328

jsteemann · 2018-09-20T12:26:54Z

In an application that uses jemalloc statically linked, I am seeing an ever-increasing value of the process' vm.max_map_count value. The overcommit_memory setting value is 2, so no overcommitting.
It seems that jemalloc reads the overcommit setting at startup, and later takes this setting's value into account when "returning" memory.
When overcommit_memory is set to 2, it seems to call mmap on the returned range, with a protection of PROT_NONE. It seems that this punches holes into existing mappings, so that the kernel will split them and create more of them. This would not be a problem if it happened only seldomly, but we have several use cases in which it happens so often that even increasing the value of vm.max_map_count to tens of millions does not help much.

I have created some (contrived) standalone test program which shows the behavior. I hope it is somewhat deterministic so others can reproduce it:

#include <iostream>
#include <fstream>
#include <deque>
#include <vector>
#include <thread>
#include <algorithm>
#include <cstdlib>
#include <cstring>

void fun() {
  std::string line;
  size_t allocated = 0;
  size_t allocations = 0;
  std::deque<std::pair<void*, size_t>> all;

  size_t n = 1024 * 1024 * 1024;
  ::srand(2848583);
  for (size_t i = 0; i < n; ++i) {
    if (allocations % 500000 == 0) {
      std::ifstream ifs("/proc/self/maps", std::ios_base::in);
      if (!ifs.is_open()) {
        std::cerr << "unable to open mappings file" << std::endl;
        std::abort();
      }
      size_t mappings = 0;
      while (std::getline(ifs, line)) {
        ++mappings;
      }
      std::cout << "- i: " << i << ", allocations: " << allocations << ", mappings: " << mappings << ", allocated: " << allocated << "\n";
    }
    size_t s = ::rand() % (4096 * 128 / 8);
    if (s > 0) {
      auto* p = ::malloc(s);
      if (p == nullptr) {
        std::cerr << "OOM. s: " << s << std::endl;
        std::abort();
      }
      ++allocations;
      allocated += s;
      all.push_back(std::make_pair(p, s));
    }
    while (!all.empty() && allocated > 1024 * 1024 * 128) {
      auto& f = all.front();
      ::free(f.first);
      allocated -= f.second;
      all.pop_front();
    }
  }
}

int main() {
  std::vector<std::thread> threads;

  size_t n = 8; 
  for (size_t i = 0; i < n; ++i) {
    threads.emplace_back(fun);
  }
  
  for (auto& it : threads) {
    it.join();
  }
}

The test program can be compiled and run as follows:

g++ -std=c++11 -O2 -Wall -Wextra test.cc -o test -lpthread
GLIBCXX_FORCE_NEW=1 LD_PRELOAD=~/jemalloc-5.1.0/lib/libjemalloc.so ./test

The program allocates memory of pseudo-random sizes and returns some of the memory. It does so with a few parallel threads. Each thread will not exceed a certain size of allocated memory, so it should not leak.
Each thread is writing out some values to std::cout. The only interesting figure to look at is the "mappings" value reported, e.g.

- i: 7500131, allocations: 7500000, mappings: 18347, allocated: 134196266

That "mappings" value is calculated as the number of lines in /proc/self/maps, which is not 100% accurate but should be a good-enough approximation.

The problem is that when overcommit_memory is set to 2, the number of mappings will grow crazily, both with jemalloc 5.0.1 and jemalloc 5.1.0.

A "fix" for the problem is to apply the following patch:

diff --git a/3rdParty/jemalloc/v5.1.0/src/pages.c b/3rdParty/jemalloc/v5.1.0/src/pages.c
index 26002692d6..3fbad076ad 100644
--- a/3rdParty/jemalloc/v5.1.0/src/pages.c
+++ b/3rdParty/jemalloc/v5.1.0/src/pages.c
@@ -23,7 +23,7 @@ static size_t os_page;
 
 #ifndef _WIN32
 #  define PAGES_PROT_COMMIT (PROT_READ | PROT_WRITE)
-#  define PAGES_PROT_DECOMMIT (PROT_NONE)
+#  define PAGES_PROT_DECOMMIT (PROT_READ | PROT_WRITE)
 static int     mmap_flags;
 #endif
 static bool    os_overcommits;

This makes the test program run with a very low number of memory mappings. It is obviously not a good fix, because it will leave the memory around with read & write access allowed. So please consider it just a demo.

I think it would be good to make jemalloc more usable with an overcommit_memory setting value of 2. Right now, it is kind of risky to use it, because applications may too quickly hit the default vm.max_map_count value of 65K. And even increasing that setting does not help much, because the number of mappings can increase much over time, which means long-running server processes can hit the threshold easily, even if increased.

I guess the current implementation is as it is for a reason, so I guess you will be pretty reluctant to change it. However, it would be good to suggest how to avoid that behavior on systems that don't use overcommit and where vm settings cannot be adjusted. Can an option be added to jemalloc to adjust the behavior on commit in this case, when explicitly configured as such? I think this would help plenty of users, as I have seen several issues in this repository that may have the same root cause. The last one I checked was #1324.
Thanks!

(btw. although I think it does make any difference: the above was tried on Linux kernels 4.15 both on bare metal and an Azure cloud instance, compilers in use were g++-7.3.0 and g++-5.4.0)

The text was updated successfully, but these errors were encountered:

davidtgoldblatt · 2018-09-20T18:24:38Z

Thank you for this great bug report and diagnosis. I don't think the current behavior was really chosen per se; it's just an emergent property of a combination of things we don't test very well. I think this needs some amount of philosophizing.

(I think it might have to wait for @interwq before anyone can take thorough look; we're stretched a little thin for now).

Ironically, one thing that might help is disabling purging settings (i.e. MALLOC_CONF="dirty_decay_ms:-1,muzzy_decay_ms:-1"); this would increase RSS, but not having fiddling around with mappings as much might make it net-better.

jsteemann · 2018-09-20T20:40:23Z

@davidtgoldblatt : thanks for looking into it and the suggestion for disabling purging.
I tried running the test program with purging disabled as you suggested (MALLOC_CONF="dirty_decay_ms:-1,muzzy_decay_ms:-1").

With jemalloc 5.1.0 this indeed makes the number of mappings grows much much more slowly, but the overall number still increases over time. It is the hundreds rather than in the tens of thousands now.
Worse however is that this setting makes the overall virtual memory size used by the process (as reported by top) quickly grow to more than 20 GB. On my local laptop it reaches 20 GB vsz after less than 2 minutes and then dies with OOM. With the default malloc settings the process is at around 2,5 GB vsz usage after the same time and can still allocate then.
In terms of rss the adjusted MALLOC_CONF value is also a bit worse, at least for this particular test.

egaudry · 2018-09-26T07:30:11Z

Just a side note regarding kernel if needed, this behavior could also be observed with 3.10.x and 4.4.x.

jsteemann · 2018-09-26T07:38:44Z

For reference, here is a "fix" for 5.1.0:

diff --git a/3rdParty/jemalloc/v5.1.0/src/pages.c b/3rdParty/jemalloc/v5.1.0/src/pages.c
index 26002692d6..3fbad076ad 100644
--- a/3rdParty/jemalloc/v5.1.0/src/pages.c
+++ b/3rdParty/jemalloc/v5.1.0/src/pages.c
@@ -23,7 +23,7 @@ static size_t os_page;
 
 #ifndef _WIN32
 #  define PAGES_PROT_COMMIT (PROT_READ | PROT_WRITE)
-#  define PAGES_PROT_DECOMMIT (PROT_NONE)
+#  define PAGES_PROT_DECOMMIT (PROT_READ | PROT_WRITE)
 static int     mmap_flags;
 #endif
 static bool    os_overcommits;

And for 5.0.1:

diff --git a/3rdParty/jemalloc/v5.0.1/src/pages.c b/3rdParty/jemalloc/v5.0.1/src/pages.c
index fec64dd01d..733652adf3 100644
--- a/3rdParty/jemalloc/v5.0.1/src/pages.c
+++ b/3rdParty/jemalloc/v5.0.1/src/pages.c
@@ -20,7 +20,7 @@ static size_t os_page;
 
 #ifndef _WIN32
 #  define PAGES_PROT_COMMIT (PROT_READ | PROT_WRITE)
-#  define PAGES_PROT_DECOMMIT (PROT_NONE)
+#  define PAGES_PROT_DECOMMIT (PROT_READ | PROT_WRITE)
 static int     mmap_flags;
 #endif
 static bool    os_overcommits;

These "fixes" prevent the endless growth of mappings.
However, I don't recommend using these fixes, because they will allow the application to still access memory it has returned to the allocator via free()/delete. The standard jemalloc behavior will lead to the kernel raising a SIGSEGV when accessing a memory page that the application or the allocator has given back after decommitting the memory. The patches above will however allow the application to still access memory it has given back, so they may hide errors in an application's memory handling and may also cause security issues, because memory given back by the application is still readable (only by the application itself - however, this is normally undesired).

egaudry · 2018-10-05T09:02:25Z

@jasone : do you think that a simple fix can be made to avoid this issue (this is likely related to c2f970c) ?

egaudry · 2019-01-11T09:55:25Z

@interwq Hello, is there any chance to get this fixed for milestone 5.2.0 ?

interwq · 2019-01-11T22:53:17Z

@egaudry : there doesn't seem to be a straightforward fix I can think of right now. Like David mentioned the current behavior under no overcommit isn't particularly optimized, as the environment we work with usually has overcommit enabled.

I wasn't able to change the overcommit setting on my dev box somehow. Can you guys help to try one more thing, running with malloc_conf retain:false? The retain option was specifically designed to reduce the number of mappings. I wonder if it doesn't interact properly with no overcommit.

egaudry · 2019-01-14T08:25:01Z

Do you mean that
$ /sbin/sysctl vm.overcommit_memory=2
would not work on your box ?
You can try to lower vm.max_map_count too, it would allow to reproduce a similar issue too.

I'm not sure about the retain option : as @jsteemann observed, any option that would reduce the number of mapping would not allow to avoid the issue in the long term (i.e. with numerous allocation and/or long living process).

interwq · 2019-01-18T20:07:38Z

I meant turning off retain is worth trying, since it could affect number of mappings even in the long term. Plus I believe the option was more designed with overcommit in mind; it may affect # of mappings negatively with no overcommit.

egaudry · 2019-01-19T12:57:30Z

Thanks, I will have a look (I though this was a compile time parameter).

egaudry · 2019-01-21T15:20:42Z

Do I need to rebuild jemalloc ?
When I set MALLOC_CONF=retain:false before launching my test, jemalloc complains:

<jemalloc>: Invalid conf pair: retain:false

I should have stated that I was using (and need to stick to this) version 4.5.0 here.

jsteemann · 2019-01-21T15:30:11Z

I tried MALLOC_CONF="retain:true" original_program original_args and it works for me.
Jemalloc prints error messages whenever I used some non-existing conf option or provide non-allowed values for config options.
As far as I can see the options are always processed and there is no way to disable options parsing at compile-time. "retain" should be present at least in v5.0.1 and v5.1.0.

egaudry · 2019-01-21T16:07:18Z

Jan, you are right : I relaunched my test with the current master branch and I was able to use retain:false at runtime.

Unfortunately, it didn't solve the large map_count issue (or not enough) I observed with one of our test-cases.

jsteemann · 2019-01-21T16:12:54Z

@egaudry : that is also what I had observed before.
The only "working solution" we found is to move away completely from vm.overcommit_memory=2, but that may not be an option for everyone and also makes the allocator less general-purpose than desired.

egaudry · 2019-01-21T16:32:33Z

My main concern is that our users are reluctant to such a change or are not in a position to request such a change (ex: cluster/centralized computing resources with different software running) and I cannot reasonably expect them to switch to a more permissive mode.

jsteemann · 2019-01-21T16:40:59Z

@egaudry : yes, I understand this.
IMHO the only ones that can properly fix the issue are the jemalloc maintainers, but I guess this would be a substantial change.
As I am just a user of jemalloc myself I am not in the position comment on their development plans w.r.t. this.

egaudry · 2019-01-21T16:46:05Z

@interwq Qi, I hope our feedbacks can help. Please let us know if you have another test we can perform.
If you could comment on the work needed to properly support the no-overcommit scenario in future releases, this would be great.

interwq · 2019-01-24T22:38:03Z

@jsteemann @egaudry thanks for all your feedback and help testing the cases! We did discuss this in our last team meeting, however no straightforward fix came to mind.

One thing that can for sure alleviate the issue is, using a larger page size, e.g. build jemalloc with --with-lg-page=16 will make jemalloc uses 64k as the base page size instead of the default 4k, which naturally reduces number of mappings (usually at the cost of higher fragmentation).

My best suggestion for right now, is to combine the following options:

build with 64k pages as mentioned above
disable muzzy decay, e.g. with MALLOC_CONF=muzzy_decay_ms:0; plus a longer dirty decay time, e.g. dirty_muzzy_ms:30000
use the current dev branch which contains a new feature to reduce VM fragmentation

For long term, it's unclear if we will be focusing on reducing the # of mappings w/o overcommit. On one hand, it's probably fair to consider this an limitation of the Linux kernel (require a max mapping limit / suffer big perf degradation as mappings grow). IIRC FreeBSD doesn't have such issues. On the other hand we already spent effort to workaround this (i.e. the retain feature but obviously only for w/ overcommit).

Let us know if the config above solves it for you, or how far it goes.

egaudry · 2019-01-25T13:31:25Z

@interwq Qi, this configuration indeed offers a solution (at least on a specific test-case I'm using).

I do understand that having vm.overcommit_memory=2 nowadays might not be really relevant and as such I won't discuss anymore the need for a fix. I will instead rely on this configuration when needed.

The downside of this solution is that for an external (i.e. not aware of the jemalloc behavior) user looking at VmRSS and VmSize, it will be difficult to understand when memory gets back to the system (because it is relying on muzzy/dirty decay time jemalloc-5 behavior, but that is out of the scope here).

Thanks you all for your feedback and the solutions offered.

interwq · 2019-01-25T21:32:52Z

@egaudry : glad it worked for you.

re: the time based decay, we did observe efficiency wins on the vast majority of our binaries -- given that memory reuse is usually very frequent, in theory we should only start purging memory after the workload is finished / reduced; time-based decay does that a lot better than the previous ratio-based decay which assumes a fixed ratio. However I also understand that memory not returned to OS immediately may cause some confusion, especially in micro benchmarks (we got quite a few questions on that front).

We had some discussion regarding combining time based decay with ratio based; however the exact approach is still a bit unclear. Please feel free to share your use cases / thoughts, or ask for features there.

adamnovak · 2019-09-03T17:25:02Z

Is there an open issue tracking the problem that jemalloc in its default configuration, on a system with overcommit disabled (with vm.overcommit_memory=2), will exhaust the default mapping limit under normal usage patterns? I'm having trouble finding the decay algorithm discussions that @interwq mentioned as being the right place to pursue this further.

I also don't understand jemalloc well enough to grasp how even the best decay algorithm would prevent eventually hitting the mapping limit. Eventually unused pages will still decay, and mappings will split, right? If they split more than they recombine, eventually the limit will be hit.

We're shipping jemalloc in our release binaries in our scientific computing project, with the result that on some high-performance computing systems where memory overcommit has been disabled by cluster administrators, our software crashes because it stops being able to allocate memory.

Is the recommended solution to add --with-lg-page=16 to the jemalloc build, and use mallctl calls to set muzzy_decay_ms to 0 and dirty_muzzy_ms to 30000 when our application detects that memory overcommit is disabled, on startup? Does this completely solve the issue of the number of mappings growing unboundedly, or only slow it down? Is there a plan or open issue to integrate these defaults into jemalloc itself? Has the "current dev branch" of January with its relevant improvements been integrated into current master?

davidtgoldblatt · 2019-09-03T22:32:02Z

(Sorry, accidentally posted and then deleted a half-done comment):

Reopening, since I don't think we yet have a good general solution to this class of issues, even though the original question seems solved; there's more left to do here.

I think that it may be the case that recent changes have helped some (opt.oversize_threshold). Even better would be to turn off retain for oversized allocations, even if it's on for smaller ones (which can't be done as a tuning change I think; it needs a little bit of extra jemalloc code written).

I don't know that there's something that would make us consider this problem "solved". Fundamentally, saying so with confidence would need prod performance testing across a range of applications, and I'm not sure that the core dev team has the ability to do that sort of "testing in anger", given the sorts of prod systems we touch day-to-day. (E.g. none of us work on HPC scientific computing applications, and so can't form and test guesses on what sorts of configurations work well there). We'd definitely be receptive to PRs updating configuration settings / tweaking allocation strategies in those cases.

egaudry · 2019-09-04T07:52:48Z

@adamnovak I'm still puzzled by the fact that people tends to believe that disabling overcommit and/or limiting the max_map_count is the way to go on computing nodes. I know for a fact that it is pretty difficult to get sysadmins to change those settings (and mainly because they would use it since more than a decade), but limiting virtual memory does not make much sense in the HPC world...

If you consider that, for instance, cuda will allocate a VM equals to the physical memory detected on the host when starting, the problem becomes broader too.

adamnovak · 2019-09-04T17:23:25Z

@egaudry I can't necessarily explain why someone would want to disable overcommit either. The best I can come up with is that they want jobs to fail fast at allocation time, rather than after wasting a bunch of cluster time filling in the pages that did happen to fit in memory.

My project's immediate users are the scientists who sometimes get handed clusters with overcommit off, not the people who decided to adopt that setting, so I need to provide at least passable, if not particularly performant, behavior in that environment.

As for limiting max_map_count, the default limit on my workstation is 65530, without me having done anything to reduce it. So I don't think that people are choosing to limit it so much as that they haven't thought to increase it.

interwq · 2019-09-04T18:16:27Z

Like mentioned currently there is no plan to focus on the no overcommit case. opt.retain properly solves the number of mappings issue. However it does so through keeping VM around and allocating in larger batch, which won't work well with no overcommit. Like David mentioned, we are certainly open to ideas though.

@adamnovak: the page size + decay tuning should alleviate the issue. For the decay setting, you can also run the binary with env var MALLOC_CONF, instead of having to do runtime mallctls. The longer decay helps because it gives the application and jemalloc more time to deallocate and merge separate extents, before trying to unmap them and create holes (extra mappings) in the VM space.

…Codec#catch_and_raise': ActiveSupport::MessageEncryptor::InvalidMessage error) 2025-02-28T07:42:28.618485307Z 8:M 28 Feb 2025 07:42:28.618 * Ready to accept connections 2025-02-28T07:42:30.095610705Z => Booting Puma 2025-02-28T07:42:30.095707345Z => Rails 8.0.1 application starting in production 2025-02-28T07:42:30.095744945Z => Run `bin/rails server --help` for more startup options 2025-02-28T07:42:30.356966530Z WARNING! : We didn't find an active_storage.service configured so we're falling back to the local store, but it's A VERY BAD IDEA to rely on it in production unless you know what you're doing. 2025-02-28T07:42:30.357004811Z Exiting 2025-02-28T07:42:30.357263852Z /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/messages/codec.rb:57:in 'ActiveSupport::Messages::Codec#catch_and_raise': ActiveSupport::MessageEncryptor::InvalidMessage 2025-02-28T07:42:30.357283812Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/message_encryptor.rb:242:in 'ActiveSupport::MessageEncryptor#decrypt_and_verify' 2025-02-28T07:42:30.357293612Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/encrypted_file.rb:109:in 'ActiveSupport::EncryptedFile#decrypt' 2025-02-28T07:42:30.357335212Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/encrypted_file.rb:72:in 'ActiveSupport::EncryptedFile#read' 2025-02-28T07:42:30.357346292Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/encrypted_configuration.rb:63:in 'ActiveSupport::EncryptedConfiguration#read' 2025-02-28T07:42:30.357354652Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/encrypted_configuration.rb:86:in 'ActiveSupport::EncryptedConfiguration#config' 2025-02-28T07:42:30.357364212Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/encrypted_configuration.rb:113:in 'ActiveSupport::EncryptedConfiguration#options' 2025-02-28T07:42:30.357372092Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/delegation.rb:186:in 'ActiveSupport::EncryptedConfiguration#method_missing' 2025-02-28T07:42:30.357420692Z from /usr/local/bundle/ruby/3.4.0/gems/bullet_train-1.18.0/lib/bullet_train.rb:149:in 'Object#two_factor_authentication_enabled?' 2025-02-28T07:42:30.357429412Z from /rails/config/initializers/devise.rb:10:in 'block in <main>' 2025-02-28T07:42:30.357435292Z from /usr/local/bundle/ruby/3.4.0/gems/devise-4.9.4/lib/devise.rb:314:in 'Devise.setup' 2025-02-28T07:42:30.357440852Z from /rails/config/initializers/devise.rb:3:in '<main>' 2025-02-28T07:42:30.357446172Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:693:in 'Kernel#load' 2025-02-28T07:42:30.357451852Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:693:in 'block in Rails::Engine#load_config_initializer' 2025-02-28T07:42:30.357457572Z from /usr/local/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/notifications.rb:212:in 'ActiveSupport::Notifications.instrument' 2025-02-28T07:42:30.357465292Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:692:in 'Rails::Engine#load_config_initializer' 2025-02-28T07:42:30.357471372Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:646:in 'block (2 levels) in <class:Engine>' 2025-02-28T07:42:30.357477292Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:645:in 'Array#each' 2025-02-28T07:42:30.357482732Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:645:in 'block in <class:Engine>' 2025-02-28T07:42:30.357509053Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:32:in 'BasicObject#instance_exec' 2025-02-28T07:42:30.357515533Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:32:in 'Rails::Initializable::Initializer#run' 2025-02-28T07:42:30.357521573Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:61:in 'block in Rails::Initializable#run_initializers' 2025-02-28T07:42:30.357527413Z from /usr/local/lib/ruby/3.4.0/tsort.rb:231:in 'block in TSort.tsort_each' 2025-02-28T07:42:30.357532653Z from /usr/local/lib/ruby/3.4.0/tsort.rb:353:in 'block (2 levels) in TSort.each_strongly_connected_component' 2025-02-28T07:42:30.357538173Z from /usr/local/lib/ruby/3.4.0/tsort.rb:425:in 'block (2 levels) in TSort.each_strongly_connected_component_from' 2025-02-28T07:42:30.357543813Z from /usr/local/lib/ruby/3.4.0/tsort.rb:434:in 'TSort.each_strongly_connected_component_from' 2025-02-28T07:42:30.357549173Z from /usr/local/lib/ruby/3.4.0/tsort.rb:424:in 'block in TSort.each_strongly_connected_component_from' 2025-02-28T07:42:30.357554613Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:50:in 'Array#each' 2025-02-28T07:42:30.357560053Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:50:in 'Rails::Initializable::Collection#tsort_each_child' 2025-02-28T07:42:30.357565853Z from /usr/local/lib/ruby/3.4.0/tsort.rb:418:in 'Method#call' 2025-02-28T07:42:30.357597933Z from /usr/local/lib/ruby/3.4.0/tsort.rb:418:in 'TSort.each_strongly_connected_component_from' 2025-02-28T07:42:30.357605093Z from /usr/local/lib/ruby/3.4.0/tsort.rb:352:in 'block in TSort.each_strongly_connected_component' 2025-02-28T07:42:30.357610653Z from /usr/local/lib/ruby/3.4.0/tsort.rb:350:in 'Rails::Initializable::Collection#each' 2025-02-28T07:42:30.357616053Z from /usr/local/lib/ruby/3.4.0/tsort.rb:350:in 'Method#call' 2025-02-28T07:42:30.357621133Z from /usr/local/lib/ruby/3.4.0/tsort.rb:350:in 'TSort.each_strongly_connected_component' 2025-02-28T07:42:30.357627333Z from /usr/local/lib/ruby/3.4.0/tsort.rb:229:in 'TSort.tsort_each' 2025-02-28T07:42:30.357632813Z from /usr/local/lib/ruby/3.4.0/tsort.rb:208:in 'TSort#tsort_each' 2025-02-28T07:42:30.357638093Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/initializable.rb:60:in 'Rails::Initializable#run_initializers' 2025-02-28T07:42:30.357643813Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/application.rb:440:in 'Rails::Application#initialize!' 2025-02-28T07:42:30.357649413Z from /rails/config/environment.rb:5:in '<main>' 2025-02-28T07:42:30.357673813Z from config.ru:3:in 'Kernel#require_relative' 2025-02-28T07:42:30.357679173Z from config.ru:3:in 'block (2 levels) in <main>' 2025-02-28T07:42:30.357711333Z from /usr/local/bundle/ruby/3.4.0/gems/rack-3.1.10/lib/rack/builder.rb:108:in 'Kernel#eval' 2025-02-28T07:42:30.357723973Z from /usr/local/bundle/ruby/3.4.0/gems/rack-3.1.10/lib/rack/builder.rb:108:in 'Rack::Builder.new_from_string' 2025-02-28T07:42:30.357730133Z from /usr/local/bundle/ruby/3.4.0/gems/rack-3.1.10/lib/rack/builder.rb:97:in 'Rack::Builder.load_file' 2025-02-28T07:42:30.357735813Z from /usr/local/bundle/ruby/3.4.0/gems/rack-3.1.10/lib/rack/builder.rb:67:in 'Rack::Builder.parse_file' 2025-02-28T07:42:30.357741373Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:354:in 'Rackup::Server#build_app_and_options_from_config' 2025-02-28T07:42:30.357747333Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:263:in 'Rackup::Server#app' 2025-02-28T07:42:30.357752773Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:424:in 'Rackup::Server#wrapped_app' 2025-02-28T07:42:30.357759374Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:326:in 'block in Rackup::Server#start' 2025-02-28T07:42:30.357765094Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:382:in 'Rackup::Server#handle_profiling' 2025-02-28T07:42:30.357770654Z from /usr/local/bundle/ruby/3.4.0/gems/rackup-2.2.1/lib/rackup/server.rb:325:in 'Rackup::Server#start' 2025-02-28T07:42:30.357776094Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/commands/server/server_command.rb:38:in 'Rails::Server#start' 2025-02-28T07:42:30.357813214Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/commands/server/server_command.rb:145:in 'block in Rails::Command::ServerCommand#perform' 2025-02-28T07:42:30.357828334Z from <internal:kernel>:91:in 'Kernel#tap' 2025-02-28T07:42:30.357834534Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/commands/server/server_command.rb:136:in 'Rails::Command::ServerCommand#perform' 2025-02-28T07:42:30.357841654Z from /usr/local/bundle/ruby/3.4.0/gems/thor-1.3.2/lib/thor/command.rb:28:in 'Thor::Command#run' 2025-02-28T07:42:30.357847414Z from /usr/local/bundle/ruby/3.4.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in 'Thor::Invocation#invoke_command' 2025-02-28T07:42:30.357853534Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/command/base.rb:178:in 'Rails::Command::Base#invoke_command' 2025-02-28T07:42:30.357859214Z from /usr/local/bundle/ruby/3.4.0/gems/thor-1.3.2/lib/thor.rb:538:in 'Thor.dispatch' 2025-02-28T07:42:30.357864454Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/command/base.rb:73:in 'Rails::Command::Base.perform' 2025-02-28T07:42:30.357887094Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/command.rb:65:in 'block in Rails::Command.invoke' 2025-02-28T07:42:30.357894774Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/command.rb:143:in 'Rails::Command.with_argv' 2025-02-28T07:42:30.357900454Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/command.rb:63:in 'Rails::Command.invoke' 2025-02-28T07:42:30.357905934Z from /usr/local/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/commands.rb:18:in '<main>' 2025-02-28T07:42:30.357911494Z from <internal:/usr/local/lib/ruby/3.4.0/rubygems/core_ext/kernel_require.rb>:37:in 'Kernel#require' 2025-02-28T07:42:30.357917134Z from <internal:/usr/local/lib/ruby/3.4.0/rubygems/core_ext/kernel_require.rb>:37:in 'Kernel#require' 2025-02-28T07:42:30.357922734Z from /usr/local/bundle/ruby/3.4.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in 'Kernel#require' 2025-02-28T07:42:30.357928654Z from ./bin/rails:4:in '<main>' 2025-02-28T07:42:36.987868425Z 8:C 28 Feb 2025 07:42:36.986 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2025-02-28T07:42:36.987899425Z 8:C 28 Feb 2025 07:42:36.986 # Redis version=7.0.15, bits=64, commit=00000000, modified=0, pid=8, just started 2025-02-28T07:42:36.987903425Z 8:C 28 Feb 2025 07:42:36.986 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 2025-02-28T07:42:36.987906945Z 8:M 28 Feb 2025 07:42:36.987 * monotonic clock: POSIX clock_gettime 2025-02-28T07:42:36.988104746Z 8:M 28 Feb 2025 07:42:36.987 * Running mode=standalone, port=6379. 2025-02-28T07:42:36.988227546Z 8:M 28 Feb 2025 07:42:36.988 # Server initialized 2025-02-28T07:42:36.988332907Z 8:M 28 Feb 2025 07:42:36.988 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 2025-02-28T07:42:36.991131598Z 8:M 28 Feb 2025 07:42:36.991 * Ready to accept connections INFO [fab3b1fa] Running docker container ls --all --filter name=^socialgames-cc-web-177b833df3bb72515df19bf25623e7715d19a15a$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 138.199.206.130 INFO [fab3b1fa] Finished in 0.201 seconds with exit status 0 (successful). ERROR null INFO [8e68af83] Running docker container ls --all --filter name=^socialgames-cc-web-177b833df3bb72515df19bf25623e7715d19a15a$ --quiet | xargs docker stop on 138.199.206.130 INFO [8e68af83] Finished in 0.990 seconds with exit status 0 (successful). Finished all in 131.2 seconds Releasing the deploy lock... Finished all in 133.2 seconds ERROR (SSHKit::Command::Failed): Exception while executing on host 138.199.206.130: docker exit status: 1 docker stdout: Nothing written docker stderr: Error: target failed to become healthy within configured timeout (30s)

currently, the CI outputs the foollowing error when running redis tests: > 752:M 06 May 2025 05:31:27.212 # Server initialized > 3752:M 06 May 2025 05:31:27.212 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. not a big fan of warnings on CI

egaudry mentioned this issue Sep 20, 2018

bad_alloc due to overcommit setup in sysctl #1324

Closed

interwq closed this as completed Jan 25, 2019

adamnovak mentioned this issue Aug 30, 2019

Signal 11 occurred for vg prune vgteam/vg#2417

Open

davidtgoldblatt reopened this Sep 3, 2019

interwq mentioned this issue Jan 17, 2020

Application using jemalloc causes mmapped regions to grow beyond 65530 and result in an application to crash with std::bad_alloc #1256

Closed

luanderson23 mentioned this issue Jan 26, 2025

We're sorry, but something went wrong. chatwoot/chatwoot#10765

Closed

pulpoff mentioned this issue Feb 1, 2025

not possible to install twentyhq/twenty#9966

Open

pesmith8a mentioned this issue Feb 5, 2025

[BUG] valkey search in VM ubuntu valkey-io/valkey-search#34

Closed

kirantnath mentioned this issue Feb 9, 2025

Immich Server Wont Start (Redis/DB Timeout) immich-app/immich#15987

Closed

3 tasks

micjac99 mentioned this issue Feb 15, 2025

[Question]: Connection exception of redis infiniflow/ragflow#4994

Open

QwQPoP1234567 mentioned this issue Feb 20, 2025

qq动态扫码登陆后仍然报ck失效 idoknow/Campux#44

Open

Flv-cmd mentioned this issue Feb 20, 2025

bug: Always 0 vulns, Nuclei not work yogeshojha/rengine#1480

Open

1 task

jessefmoore mentioned this issue Mar 2, 2025

Error details: Failed to get CSRF token and won't login. seahop/Clio#1

Closed

simowilso mentioned this issue Mar 3, 2025

password authentication failed, brand new installation docmost/docmost#816

Closed

tensei888 mentioned this issue Mar 3, 2025

Failed authentication for any account sekai-soft/guide-nitter-self-hosting#32

Open

jarozamena mentioned this issue Mar 3, 2025

Problem reimporting the feeders vulnerability-lookup/vulnerability-lookup#104

Closed

Fubero8 mentioned this issue Mar 4, 2025

Error when loading immich-app/immich#16587

Closed

4 tasks

yefl2064 mentioned this issue Mar 5, 2025

update 1.0.0>>> Error response from daemon: driver failed programming external connectivity on endpoint docker-db-1 langgenius/dify#15023

Closed

5 tasks

sienar1 mentioned this issue Mar 9, 2025

Update to v0.5.0 discussion thread tubearchivist/tubearchivist#883

Closed

2 tasks

cw957 mentioned this issue Mar 11, 2025

SQL server error Resgrid/Core#169

Closed

nima-abdpoor mentioned this issue Mar 18, 2025

Memory overcommit is disabled in redis nima-abdpoor/brain-blitz#20

Open

SeptSkyTime mentioned this issue Mar 19, 2025

[Question]: container ragflow-mysql is unhealthy infiniflow/ragflow#4424

Open

TT-web-byte mentioned this issue Mar 24, 2025

http://localhost/install reports an error 504 Gateway Time-out langgenius/dify#16639

Closed

5 tasks

nodecentral mentioned this issue Apr 7, 2025

[BUG] django.db.utils.OperationalError: [Errno -2] Name or service not known paperless-ngx/paperless-ngx#9569

Closed

4 tasks

mmagithub mentioned this issue Apr 16, 2025

Assistance with bespokefit Installation openforcefield/openff-bespokefit#389

Closed

ljhardy mentioned this issue Apr 19, 2025

What am I doing wrong? bluewave-labs/Checkmate#2097

Closed

This was referenced May 1, 2025

InvenTree server starts successfully but UI is not accessible on localhost:8080 SHIWANI-DHURAD12/InvenTree-issue#1

Open

InvenTree server starts successfully but UI is not accessible on localhost:8080 inventree/InvenTree#9611

Open

RealFascinated mentioned this issue May 5, 2025

[Bug]: High CPU from Coolify Redis Container coollabsio/coolify#5768

Closed

niedomnie mentioned this issue May 12, 2025

[BUG] 7.2.4 crashed enexpedly on AlmaLinux redis/redis#14030

Closed

michaelthegreat mentioned this issue May 18, 2025

Redis warning michaelthegreat/south-tampa-plants-chatbot#2

Open

rhpeZ mentioned this issue May 20, 2025

Failed to connect to authentik backend: authentik starting goauthentik/authentik#14602

Closed

irishpadres mentioned this issue May 23, 2025

immich_server exited with code 128 after upgrade to v1.133.0 immich-app/immich#18544

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

vm.max_map_count growing steadily when vm.overcommit_memory is 2 #1328

vm.max_map_count growing steadily when vm.overcommit_memory is 2 #1328

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vm.max_map_count growing steadily when vm.overcommit_memory is 2 #1328

vm.max_map_count growing steadily when vm.overcommit_memory is 2 #1328

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!