8000 Wekan Crash if more than 2GB of RAM is used · Issue #3585 · wekan/wekan · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Wekan Crash if more than 2GB of RAM is used #3585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
r4nc0r opened this issue Feb 18, 2021 · 15 comments
Closed

Wekan Crash if more than 2GB of RAM is used #3585

r4nc0r opened this issue Feb 18, 2021 · 15 comments
8000
Labels
Bug

Comments

@r4nc0r
Copy link
r4nc0r commented Feb 18, 2021

Issue

Other Wekan issues can be added here.

Server Setup Information:

  • Did you test in newest Wekan?: yes
  • For new Wekan install, did you configure root-url correctly so Wekan cards open correctly https://github.com/wekan/wekan/wiki/Settings ?
  • Wekan version: 4.96
  • Deployment Method: docker swarm
  • Http frontend if any (Caddy, Nginx, Apache, see config examples from Wekan GitHub wiki first):
  • Node Version:
  • MongoDB Version: 4.0.11

Problem description:
if NodeJs uses more than 2GB of Memory, Wekan crashes with a Exception:

I researched a little and as it turns out NodeJs v12 has a standard limit of 2GB Memory Consumption, but it can be changed with the following Nodejs CLI Command: --max-old-space-size xxxx where xxxx is the memory in MB (source) Can this be implemented to raise the Wekan max Memory Limit?

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory,
,
<--- Last few GCs --->,
,
[1:0x38a2e80] 324979 ms: Mark-sweep 2029.2 (2080.6) -> 2018.8 (2083.1) MB, 1608.6 / 0.1 ms (average mu = 0.120, current mu = 0.022) allocation failure scavenge might not succeed,
[1:0x38a2e80] 327768 ms: Mark-sweep 2031.4 (2083.1) -> 2022.9 (2063.4) MB, 2738.9 / 0.1 ms (+ 0.1 ms in 30 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2751 ms) (average mu = 0.061, current mu = 0.018) allocation f,
,
<--- JS stacktrace --->,
,
==== JS stack trace =========================================,
,
0: ExitFrame [pc: 0x140a859],
Security context: 0x22ae44d408d1 ,
1: changeField [0x369877f3a6b9] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x273d069851f8](this=0x36d784d60269 ,0x06e82165b7a9 <String[18]: Ngb63ADH8RE3BT3C8Y>,0x24027c246289 <String[#6]: status>,0x1a40249114f9 <String[6]: online>,0x36d784d603e1 ,0x24027c240639 ),
2: ...,
,
1: 0xa18150 node::Abort() [node],
2: 0xa1855c node::OnFatalError(char const*, char const*) [node],
3: 0xb970be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node],
4: 0xb97439 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node],
5: 0xd546b5 [node],
6: 0xd54d46 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node],
7: 0xd61605 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node],
8: 0xd624b5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node],
9: 0xd64f6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node],
10: 0xd2b98b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node],
11: 0x106e0bc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node],
12: 0x140a859 [node],
@xet7 xet7 added the Bug label Feb 19, 2021
@xet7
Copy link
Member
xet7 commented Feb 19, 2021

Related https://github.com/wekan/wekan/discussions/3580

@r4nc0r
Copy link
Author
r4nc0r commented Feb 19, 2021

Do you mean the env var NODE_OPTIONS: --max_old_space_size=4096? If this is enough to fix the problem it would be great :)

@xet7
Copy link
Member
xet7 commented Feb 19, 2021

@r4nc0r

Please try does it work.

8000

@r4nc0r
Copy link
Author
r4nc0r commented Feb 22, 2021

Works :)

@r4nc0r
Copy link
Author
r4nc0r commented Feb 22, 2021

Could you add this to the offical wekan documentation?

@e-gaulue e-gaulue mentioned this issue May 3, 2021
Closed
@r4nc0r r4nc0r closed this as completed Jun 7, 2021
@xet7 xet7 reopened this Jul 3, 2021
@jankapunkt
Copy link
Contributor

The real question is why it's using >2GB of RAM.

Are there any unbusted chaches?
Memory leaks?
Too many publictations/observers?
Assets Like Images loaded into RAM?

@xet7
Copy link
Member
xet7 commented Jul 3, 2021

@jankapunkt

Yes sure. When Wekan is running, CPU usage and RAM usage is increasing all the time, as mentioned in some of these issues.

@xet7
Copy link
Member
xet7 commented Jul 3, 2021

Also too much data is published to browserside, making browsers crash #3893

@xet7 xet7 mentioned this issue Sep 22, 2021
Closed
@xet7 xet7 mentioned this issue Oct 20, 2021
Closed
@Ben0it-T
Copy link
Contributor
Ben0it-T commented Nov 5, 2021 •
edited
Loading

Definitely a stupid question ...
Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 with snap (for prevent Wekan crashes if NodeJs uses more than 2GB of Memory) ?

If yes, where/how ?
On Debian 10, the unit file snap.wekan.wekan.service, we can see
EnvironmentFile=-/etc/environment

Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 in /etc/environment ?

Thanks

@xet7
Copy link
Member
xet7 commented Nov 5, 2021 •
edited
Loading

@Ben0it-T

There are no stupid questions. There are only my stupid answers.

WeKan Snap does not have that setting option yet.

To try that option, you could use start-wekan.sh from https://github.com/wekan/wekan or some service file, and bundle version from https://github.com/wekan/wekan/wiki/Raspberry-Pi using bundle version for your server CPU architecture.

If adding that option helps, that setting could be added to some Snap settings at https://github.com/wekan/wekan/tree/master/snap-src/bin .

If that option does not help, then you should wait when I have finished partial stupid rewrite of WeKan.

@Ben0it-T
Copy link
Contributor
Ben0it-T commented Nov 6, 2021

@xet7
Thank you for your answer

For the moment I have not yet deployed Wekan within our team. I use it for my personal use.
I plan to deploy it in a few weeks for our team. So I have not yet encountered this type of problem...

At first, it could probably be interesting to know the details about the heap and size, with getHeapStatistics().
Maybe in Admin Panel > Version ?

  • total_heap_size
  • total_heap_size_executable
  • total_physical_size
  • total_available_size
  • used_heap_size
  • heap_size_limit
  • ...

What do you think about that ?

@xet7
Copy link
Member
xet7 commented Nov 6, 2021

@Ben0it-T

You can send PR to add them to:

Thanks!

@Ben0it-T Ben0it-T mentioned this issue Nov 6, 2021
Merged
@Ben0it-T
Copy link
Contributor

@xet7
A little feedback on my stupid question about NODE_OPTIONS with snap
On the latest Wekan version (5.75) with snap, if I put NODE_OPTIONS in /etc/environment, it works !

Ex1 :

root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=2048"
root@xxxxxxxx:~# snap restart wekan

max_old_space_size_2048

Ex2 :

root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=4096"
root@xxxxxxxx:~# snap restart wekan

max_old_space_size_4096

@xet7
Copy link
Member
xet7 commented Jan 19, 2022

Somebody got this:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[1:0x2d9ee80]  1270397 ms: Mark-sweep 2036.4 (2065.4) -> 2030.9 (2066.9) MB, 1982.8 / 0.2 ms  (average mu = 0.185, current mu = 0.131) allocation failure scavenge might not succeed
[1:0x2d9ee80]  1272372 ms: Mark-sweep 2036.5 (2066.9) -> 2032.0 (2067.9) MB, 1951.8 / 0.2 ms  (average mu = 0.098, current mu = 0.012) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x140dff9]
Security context: 0x1f42ab6808d1 <JSObject>
    1: changeField [0x3899d570e079] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x158bfe4d16b8](this=0x09e2b6008079 <SessionDocumentView map = 0xff51671a929>,0x00191a67ef61 <String[18]: NdHSHs6pNEj8woP53M>,0x2babc7c8c171 <String[#4]: sort>,0x3fc3dcd2d069 <HeapNumber -2>,0x09e2b60081f1 <Object map = 0x292df8f7e0a9>,0x1a2b41b80639 <true>)
    2: added(...

 1: 0xa1a640 node::Abort() [node]
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9a9fe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ad79 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd57ff5  [node]
 6: 0xd58686 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd64f45 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd65df5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd688ac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd2f2cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10719fc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x140dff9  [node]

So does any of these help?

https://stackoverflow.com/questions/53230823/fatal-error-ineffective-mark-compacts-near-heap-limit-allocation-failed-javas

@mfilser

Is there already some setting in WeKan for this, or should it be added?

@xet7 xet7 pinned this issue Jan 19, 2022
@xet7
Copy link
Member
xet7 commented Jan 25, 2022 •
edited
Loading

Some progress:

I did this testing on Debian VM that is running on bare metal server donated to WeKan development. That Debian VM has 60 GB RAM, just in case, it could work with less.

  1. So that right click Inspect console would not show Templates related errors, I clicked right top username / Templates, to open Templates board and have those created.
  2. One customer gave me example board export, size 4 MB of json.
  3. I checked node path:
which node
  1. In start-wekan.sh I have added these, with node path, and additional stack size mentioned here https://stackoverflow.com/questions/20936486/node-js-maximum-call-stack-size-exceeded

UPDATE: This works best with size 4096. Bigger values do increase CPU usage too much.

cd .build/bundle
export NODE_OPTIONS="--max_old_space_size=4096"
/bin/bash -c "ulimit -s 65500; exec /home/wekan/.volta/bin/node --stack-size=65500 main.js"
  1. At Debian VM, I installed xclip, that copies textfile to clipboard:
sudo apt -y install xclip
  1. I copied board JSON to clipboard:
cat board.json | xclip -se c
  1. In Chrome, I did: All Boards page / Add Board / Import / From previous export / right click to input field / paste
  2. (I did not map any users in my case, because my test WeKan install has only one user. This is optional.)
  3. There could be another button about mapping users, without list of users to map. In this step, I checked is there any errors at browser right click Inspect Console. There were no errors. Anyway, I did not click again (sometime it could cause duplicate imported boards), I just clicked All Boards, and opened that imported board.
  4. Memory usage in that Debian VM:
$ free -h
               total        used        free      shared  buff/cache   available
Mem:            61Gi       3,4Gi        47Gi       135Mi        10Gi        57Gi
Swap:          975Mi          0B       975Mi
  1. I don't see any errors in browser right click Inspect console, other than one notice about autofocus that does not matter.

xet7 added a commit that referenced this issue Feb 3, 2022
@xet7
Add more stack and heap to Node.js at all WeKan platforms.
ff13571 
Thanks to xet7 !

Related #3585
@xet7 xet7 closed this as completed Feb 4, 2022
@xet7 xet7 unpinned this issue Feb 4, 2022
imajus pushed a commit to majus/wekan that referenced this issue Feb 6, 2022
@xet7 @imajus
Add more stack and heap to Node.js at all WeKan platforms.
8a94b23 
Thanks to xet7 !

Related wekan#3585
imajus pushed a commit to majus/wekan that referenced this issue Feb 6, 2022
@xet7 @imajus
Add more stack and heap to Node.js at all WeKan platforms.
43ac30a 
Thanks to xet7 !

Related wekan#3585
@xet7 xet7 mentioned this issue Nov 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@xet7 @jankapunkt @r4nc0r @Ben0it-T
0