Replies: 6 comments 6 replies
-
|
You can refer to here: #2726 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I've followed the post but it seems it is not working, I setup 128MB of cache and ram went up to 480MB. I made a second test limiting docker ram to 256MB and it stopped there so it seems it uses RAM up to the limit. |
Beta Was this translation helpful? Give feedback.
-
|
Could you explain a bit about "128MB cache" here? Is it a sum of block cache and write buffer? |
Beta Was this translation helpful? Give feedback.
-
|
I tried to set --rocksdb.block_cache_size 128 but without limiting docker image doesn't not limi ram usage. The only way I found to limit memory usage is to limit container memory and nothing else. |
Beta Was this translation helpful? Give feedback.
-
|
@webfrank You could reduce the following configurations as well:
You could reduce the write_buffer_size to 16MiB and max_write_buffer_numbner to 1 if you write traffic is low. |
Beta Was this translation helpful? Give feedback.
-
|
@webfrank Does this work well for you after reducing those two parameters? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, the ram reduced but not so much as expected, it continues to increase up to container limits. Now I'm trying with 256MB block cache and 4x16MB write cache and a container limit of 512MB and after several batches seems stable around 480MB |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have a similar question. How can I limit the RAM usage without impacting the write buffer? I have the following configurations: TLDR: I'm using kvrocks version 2.11.1 I have cascading replicas with a similar configuration. The topology is [master ---> cascading_replica ---> replica] During a high volume of writes to the master, the memory usage increases in all three Kvrocks instances. I loaded the master with a sizable database (~150MB) (~2mil keys) before doing the heavy writes. Writes to master: Memory usage on cascading replica (not reported in rss_bytes metrics): On master: On replica: |
Beta Was this translation helpful? Give feedback.
-
|
This is a very big problem that prevents kvrocks from being used in production tasks. It consumes all available memory. If you limit it in a container, the container simply dies on resource-intensive tasks. This is really sad. |
Beta Was this translation helpful? Give feedback.
-
|
@rizkymma @hosting2000me We have multiple column families in Kvrocks, so you could reduce the write buffer number if you would like to control the memory usage:
You could reduce |
Beta Was this translation helpful? Give feedback.
-
These are my current settings. This doesn't prevent the container from consuming 20GB(for my case) of RAM if it's not limited in the container. So, kvrocks caches all database accesses, if possible. |
Beta Was this translation helpful? Give feedback.
-
|
@hosting2000me @rizkymma @webfrank Could you please run with Kvrocks with the environment variable:
Apache James also encounters similar issues when integrating with Kvrocks, and they report it might be related to Jemalloc's retain policy. For more information please refer to https://lists.apache.org/thread/xzn62z7xz3bvxwgsd9toh6wq2g2j8zng. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I've come to KVRocks from Redis as I need to limit RAM usage on a Job Queue process. The main process will create thousands of keys with some JSON inside, once the job is done it will update the key, not much job and no latency issue, only memory allocation. On redis I reach 16GB of RAM which cannot be handle on the current environment. I would like to lower this figure to 1/2GB mantaining the data and this is why I reached kvrocks.
I exchanged redis for kvrocks and it worked as expected (apart from a corruption error for which I've opened an issue), now I would like to limit ram usage, I set --rocksdb.block_cache_size 128 on command line but the ram is increasing. Which flags I need to set to limit it? (128 is just a test to limit with very low usage). Now I'm using 2.10.1 as per the corruption issue with 2.11.0
Beta Was this translation helpful? Give feedback.
All reactions