8000 Added more flags to resolve high memory consumption by jeongkyun-oh · Pull Request #1918 · klaytn/klaytn · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Aug 19, 2024. It is now read-only.

Added more flags to resolve high memory consumption #1918

Merged
merged 4 commits into from
Aug 7, 2023

Conversation

jeongkyun-oh
Copy link
Contributor
@jeongkyun-oh jeongkyun-oh commented Aug 3, 2023

Proposed changes

Introduced two flags

  • MaxOpenFiles and CacheIndexAndFilter flags are added
  • RocksDB used extra memory for index and filter blocks other than block cache.
    • MaxOpenFiles controls the number of index/filter blocks by limiting this number.
    • CacheIndexAndFilter uses block cache instead of using extra.

ref: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks

Minor fixes

  • Free key/value of iterator
  • Rebalance cache size of sharded database

Types of changes

Please put an x in the boxes related to your change.

  • Bugfix
  • New feature or enhancement
  • Others

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING GUIDELINES doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes ($ make test)
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

@jeongkyun-oh jeongkyun-oh self-assigned this Aug 3, 2023
@jeongkyun-oh jeongkyun-oh changed the title Added more flags Added more flags for high memory consumption Aug 3, 2023
@jeongkyun-oh jeongkyun-oh mentioned this pull request Aug 3, 2023
20 tasks
@jeongkyun-oh jeongkyun-oh force-pushed the 230801-fix-memory-leak branch from 04cae62 to 50373f2 Compare August 4, 2023 00:17
@jeongkyun-oh jeongkyun-oh added the need to merge Need to merge for the next time label Aug 4, 2023
@jeongkyun-oh jeongkyun-oh changed the title Added more flags for high memory consumption Added more flags to resolve high memory consumption Aug 4, 2023
@jeongkyun-oh jeongkyun-oh requested a review from ian0371 August 4, 2023 05:01
@jeongkyun-oh
Copy link
Contributor Author

@ian0371 @kjeom @ethan-kr @aidan-kwon Please take a look.

ian0371
ian0371 previously approved these changes Aug 4, 2023
Copy link
Contributor
@ian0371 ian0371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, aliases only exist for "DstRocksDB*", so I'll add aliases to "RocksDB*" in another PR.

@jeongkyun-oh
Copy link
Contributor Author

btw, aliases only exist for "DstRocksDB*", so I'll add aliases to "RocksDB*" in another PR.

My bad. Thanks!

@jeongkyun-oh jeongkyun-oh force-pushed the 230801-fix-memory-leak branch from 813e684 to f29addc Compare August 7, 2023 02:31
@jeongkyun-oh
Copy link
Contributor Author

@ian0371 @kjeom @ethan-kr PTAL.

@jeongkyun-oh jeongkyun-oh merged commit 8c4ebcd into klaytn:dev Aug 7, 2023
@jeongkyun-oh jeongkyun-oh deleted the 230801-fix-memory-leak branch August 7, 2023 07:51
@ian0371 ian0371 mentioned this pull request Sep 11, 2023
9 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
need to merge Need to merge for the next time
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0