8000 Reconsider representation of keys for state and block store · Issue #1041 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Reconsider representation of keys for state and block store #1041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
4 of 10 tasks
Tracked by #44
jmalicevic opened this issue Jun 27, 2023 · 2 comments · Fixed by #2327
Closed
4 of 10 tasks
Tracked by #44

Reconsider representation of keys for state and block store #1041

jmalicevic opened this issue Jun 27, 2023 · 2 comments · Fixed by #2327
Assignees
@jmalicevic
Labels
P:storage-optimization Priority: Give operators greater control over storage and storage optimization
Milestone
2024-Q1

Comments

@jmalicevic
Copy link
Contributor
jmalicevic commented Jun 27, 2023 •
edited
Loading

To save state and block data, CometBFT generates keys with string + height prefixes. They are at the moment internally sorted lexicographically.

This potentiall impacts both data access time (lookup) as well as the performance of compaction.

Some work in Q2 2023 indicates that changing the way keys are represented and forcing them to be ordered by height improved compaction and the disk footprint of the blockstore (28MB shrank to 18MB). This was however a POC and the generated data size was probably not big enough to be certain of gains.

Also, it is not 100% clear whether sorting all types of data by height is desired. Talking to users and analyzing the access patterns should help with that.

We need to:

This would potentially resolve or improve informalsystems/interchain#1

DoD
@jmalicevic jmalicevic mentioned this issue Jun 27, 2023
Open
21 tasks
@jmalicevic jmalicevic added the P:storage-optimization Priority: Give operators greater control over storage and storage optimization label Jun 27, 2023
@jmalicevic jmalicevic added this to the 2023-Q3 milestone Jun 27, 2023
@jmalicevic jmalicevic changed the title Reconsider representation of keys for state and block store (some work done for Tendermint v0.35 and v0.36) Reconsider representation of keys for state and block store Jun 27, 2023
@jmalicevic jmalicevic modified the milestones: 2023-Q3, 2023-Q4 Oct 16, 2023
@jmalicevic
Copy link
Contributor Author

During Q3 2023, we had an intern experiment with different backends and it was significantly faster to insert 5GB of data into the database in order rather then in a random order (16s vs. 8min). Currently the keys in CometBFT are randomly inserted.

Based on those results pruning and compaction seem to also be more efficient and better reflected when the keys are properly ordered. This is why in Q4 2023, we will backport the changes from 0.36 and experiment to confirm their efficiency.

@adizere adizere mentioned this issue Oct 19, 2023
Closed
@melekes melekes mentioned this issue Dec 7, 2023
Closed
3 tasks
@melekes melekes self-assigned this Dec 11, 2023
@melekes
Copy link
Contributor
melekes commented Dec 12, 2023 •
edited
Loading
vanilla 4 node setup: 
#key: blockStore
#key: genesisDocHash
#key: stateKey
#key: stateKey
#key: stateKey
#key: AppRetainHeightKey
#key: offlineStateSyncHeightKey
#key: AppRetainHeightKey
#key: validatorsKey:1
#key: validatorsKey:1
#key: H:1
#key: validatorsKey:2
#key: validatorsKey:1
#key: validatorsKey:2
#key: validatorsKey:1
#key: validatorsKey:2
#key: validatorsKey:1
#key: H:2
#key: validatorsKey:3
#key: validatorsKey:1
#key: validatorsKey:3
#key: validatorsKey:1
#key: H:3
#key: validatorsKey:4
#key: validatorsKey:1
#key: validatorsKey:4
#key: validatorsKey:1
#key: H:4
#key: validatorsKey:5
#key: validatorsKey:1
#key: validatorsKey:5
#key: validatorsKey:1
#key: H:5
#key: SC:6
#key: SC:6
#key: SC:6
#key: validatorsKey:6
#key: validatorsKey:1
#key: validatorsKey:6
#key: validatorsKey:1
#key: validatorsKey:6
#key: validatorsKey:1
#key: H:6
#key: AppRetainHeightKey
#key: validatorsKey:7
#key: validatorsKey:1
#key: validatorsKey:7
#key: validatorsKey:1
#key: H:7
#key: validatorsKey:8
#key: validatorsKey:1
#key: validatorsKey:8
#key: validatorsKey:1
#key: H:8
#key: validatorsKey:9
#key: validatorsKey:1
#key: validatorsKey:9
#key: validatorsKey:1
#key: H:9
#key: validatorsKey:10
#key: validatorsKey:1
#key: validatorsKey:10
#key: validatorsKey:1
#key: validatorsKey:10
#key: validatorsKey:1
#key: H:10
#key: validatorsKey:11
#key: validatorsKey:1
#key: validatorsKey:11
#key: validatorsKey:1
#key: H:11
#key: validatorsKey:12
#key: validatorsKey:1
#key: validatorsKey:12
#key: validatorsKey:1
#key: H:12
#key: validatorsKey:13
#key: validatorsKey:1
#key: validatorsKey:13
#key: validatorsKey:1
#key: H:13
#key: validatorsKey:14
#key: validatorsKey:1
#key: validatorsKey:14
#key: validatorsKey:1
#key: validatorsKey:14
#key: validatorsKey:1
#key: H:14
#key: AppRetainHeightKey
#key: validatorsKey:15
#key: validatorsKey:1
#key: validatorsKey:15
#key: validatorsKey:1
#key: H:15
#key: validatorsKey:16
#key: validatorsKey:1
#key: validatorsKey:16
#key: validatorsKey:1
#key: H:16
#key: validatorsKey:17
#key: validatorsKey:1
#key: validatorsKey:17
#key: validatorsKey:1
#key: H:17
#key: validatorsKey:18
#key: validatorsKey:1
#key: validatorsKey:18
#key: validatorsKey:1
#key: validatorsKey:18
#key: validatorsKey:1
#key: H:18
#key: validatorsKey:19
#key: validatorsKey:1
#key: validatorsKey:19
#key: validatorsKey:1
#key: H:19
#key: validatorsKey:20
#key: validatorsKey:1
#key: validatorsKey:20
#key: validatorsKey:1
#key: H:20
#key: validatorsKey:21
#key: validatorsKey:1
#key: validatorsKey:21
#key: validatorsKey:1
#key: H:21
#key: validatorsKey:22
#key: validatorsKey:1
#key: AppRetainHeightKey
#key: validatorsKey:22
#key: validatorsKey:1
#key: validatorsKey:22
#key: validatorsKey:1
#key: H:22
#key: validatorsKey:23
#key: validatorsKey:1
#key: validatorsKey:23
#key: validatorsKey:1
#key: H:23
#key: validatorsKey:24
#key: validatorsKey:1
#key: validatorsKey:24
#key: validatorsKey:1
#key: H:24
#key: validatorsKey:25
#key: validatorsKey:1
#key: validatorsKey:25
#key: validatorsKey:1
#key: H:25
#key: validatorsKey:26
#key: validatorsKey:1
#key: validatorsKey:26
#key: validatorsKey:1
#key: validatorsKey:26
#key: validatorsKey:1
#key: H:26
#key: validatorsKey:27
#key: validatorsKey:1
#key: validatorsKey:27
#key: validatorsKey:1
#key: H:27
#key: validatorsKey:28
#key: validatorsKey:1
#key: validatorsKey:28
#key: validatorsKey:1
#key: H:28
#key: validatorsKey:29
#key: validatorsKey:1
#key: validatorsKey:29
#key: validatorsKey:1
#key: H:29
#key: AppRetainHeightKey
#key: validatorsKey:30
#key: validatorsKey:1
#key: validatorsKey:30
#key: validatorsKey:1
#key: validatorsKey:30
#key: validatorsKey:1
#key: H:30
#key: validatorsKey:31
#key: validatorsKey:1
#key: validatorsKey:31
#key: validatorsKey:1
#key: H:31
#key: validatorsKey:32
#key: validatorsKey:1
#key: validatorsKey:32
#key: validatorsKey:1
#key: H:32
#key: validatorsKey:33
#key: validatorsKey:1
#key: validatorsKey:33
#key: validatorsKey:1
#key: H:33
#key: validatorsKey:34
#key: validatorsKey:1
#key: validatorsKey:34
#key: validatorsKey:1
#key: validatorsKey:34
#key: validatorsKey:1
#key: H:34
#key: validatorsKey:35
#key: validatorsKey:1
#key: validatorsKey:35
#key: validatorsKey:1
#key: H:35
#key: AppRetainHeightKey
#key: validatorsKey:36
#key: validatorsKey:1
#key: validatorsKey:36
#key: validatorsKey:1
#key: H:36
#key: validatorsKey:37
#key: validatorsKey:1
#key: validatorsKey:37
#key: validatorsKey:1
#key: AppRetainHeightKey
#key: H:37
#key: validatorsKey:38
#key: validatorsKey:1
#key: validatorsKey:38
#key: validatorsKey:1
#key: validatorsKey:38
#key: validatorsKey:1
#key: H:38
#key: SC:39
#key: SC:39
#key: SC:39
#key: validatorsKey:39
#key: validatorsKey:1
#key: validatorsKey:39
#key: validatorsKey:1
#key: H:39
#key: AppRetainHeightKey
#key: validatorsKey:40
#key: validatorsKey:1
#key: validatorsKey:40
#key: validatorsKey:1
#key: H:40
#key: SC:41
#key: SC:41
#key: validatorsKey:41
#key: validatorsKey:1
#key: validatorsKey:41
#key: validatorsKey:1
#key: AppRetainHeightKey
#key: AppRetainHeightKey
#key: H:41
#key: SC:42
#key: SC:42
#key: validatorsKey:42
#key: validatorsKey:1
#key: validatorsKey:42
#key: validatorsKey:1
#key: SC:42
#key: validatorsKey:42
#key: validatorsKey:1
#key: H:42
#key: validatorsKey:43
#key: validatorsKey:1
#key: validatorsKey:43
#key: validatorsKey:1
#key: H:43
#key: validatorsKey:44
#key: validatorsKey:1
#key: validatorsKey:44
#key: validatorsKey:1
#key: H:44
#key: validatorsKey:45
#key: validatorsKey:1
#key: validatorsKey:45
#key: validatorsKey:1
#key: H:45
#key: validatorsKey:46
#key: validatorsKey:1
#key: validatorsKey:46
#key: validatorsKey:1
#key: validatorsKey:46
#key: validatorsKey:1
#key: H:46
#key: AppRetainHeightKey
#key: validatorsKey:47
#key: validatorsKey:1
#key: validatorsKey:47
#key: validatorsKey:1
#key: H:47
#key: validatorsKey:48
#key: validatorsKey:1
#key: validatorsKey:48
#key: validatorsKey:1
#key: H:48
#key: validatorsKey:49
#key: validatorsKey:1
#key: validatorsKey:49
#key: validatorsKey:1
#key: H:49
#key: validatorsKey:50
#key: validatorsKey:1
#key: validatorsKey:50
#key: validatorsKey:1
#key: validatorsKey:50
#key: validatorsKey:1
#key: H:50
#key: validatorsKey:51
#key: validatorsKey:1
#key: validatorsKey:51
#key: validatorsKey:1
#key: H:51
#key: validatorsKey:52
#key: validatorsKey:1
#key: validatorsKey:52
#key: validatorsKey:1
#key: H:52
#key: validatorsKey:53
#key: validatorsKey:1
#key: validatorsKey:53
#key: validatorsKey:1
#key: H:53
#key: validatorsKey:54
#key: validatorsKey:1
#key: validatorsKey:54
#key: validatorsKey:1
#key: validatorsKey:54
#key: validatorsKey:1
#key: H:54
vanilla 4 node setup w/ pruning 
#key: blockStore
#key: genesisDocHash
#key: stateKey
#key: H:1
#key: P:1:0
#key: H:2
#key: P:2:0
#key: validatorsKey:1
#key: H:3
#key: P:3:0
#key: validatorsKey:2
#key: validatorsKey:1
#key: H:4
#key: P:4:0
#key: validatorsKey:3
#key: validatorsKey:1
#key: H:5
#key: P:5:0
#key: validatorsKey:4
#key: validatorsKey:1
#key: H:6
#key: P:6:0
#key: validatorsKey:5
#key: validatorsKey:1
#key: H:7
#key: P:7:0
#key: validatorsKey:6
#key: validatorsKey:1
#key: H:8
#key: P:8:0
#key: validatorsKey:7
#key: validatorsKey:1
#key: H:9
#key: P:9:0
#key: validatorsKey:8
#key: validatorsKey:1
#key: H:10
#key: P:10:0
#key: validatorsKey:9
#key: validatorsKey:1
#key: H:11
#key: P:11:0
#key: validatorsKey:10
#key: validatorsKey:1
#key: H:12
#key: P:12:0
#key: validatorsKey:11
#key: validatorsKey:1
#key: H:13
#key: P:13:0
#key: validatorsKey:12
#key: validatorsKey:1
#key: H:14
#key: P:14:0
#key: validatorsKey:13
#key: validatorsKey:1
#key: H:15
#key: P:15:0
#key: validatorsKey:14
#key: validatorsKey:1
#key: H:16
#key: P:16:0
#key: validatorsKey:15
#key: validatorsKey:1
#key: H:17
#key: P:17:0
#key: validatorsKey:16
#key: validatorsKey:1
#key: H:18
#key: P:18:0
#key: validatorsKey:17
#key: validatorsKey:1
#key: H:19
#key: P:19:0
#key: validatorsKey:18
#key: validatorsKey:1
#key: H:20
#key: P:20:0
#key: validatorsKey:19
#key: validatorsKey:1
#key: H:21
#key: P:21:0
#key: validatorsKey:20
#key: validatorsKey:1
#key: H:22
#key: P:22:0
#key: validatorsKey:21
#key: validatorsKey:1
#key: H:23
#key: P:23:0
#key: validatorsKey:22
#key: validatorsKey:1
#key: H:24
#key: P:24:0
#key: validatorsKey:23
#key: validatorsKey:1
#key: H:25
#key: P:25:0
#key: validatorsKey:24
#key: validatorsKey:1
#key: H:26
#key: P:26:0
#key: validatorsKey:25
#key: validatorsKey:1
#key: H:27
#key: P:27:0
#key: validatorsKey:26
#key: validatorsKey:1
#key: H:28
#key: P:28:0
#key: validatorsKey:27
#key: validatorsKey:1
#key: H:29
#key: P:29:0
#key: validatorsKey:28
#key: validatorsKey:1
#key: H:30
#key: P:30:0
#key: validatorsKey:29
#key: validatorsKey:1
#key: H:31
#key: P:31:0
#key: validatorsKey:30
#key: validatorsKey:1
#key: H:32
#key: P:32:0
#key: validatorsKey:31
#key: validatorsKey:1
#key: H:33
#key: P:33:0
#key: validatorsKey:32
#key: validatorsKey:1
#key: H:34
#key: P:34:0
#key: validatorsKey:33
#key: validatorsKey:1
#key: H:35
#key: P:35:0
#key: validatorsKey:34
#key: validatorsKey:1
#key: H:36
#key: P:36:0
#key: P:36:1
#key: P:36:2
#key: P:36:3
#key: P:36:4
#key: P:36:5
#key: P:36:6
#key: P:36:7
#key: P:36:8
#key: P:36:9
#key: P:36:10
#key: P:36:11
#key: P:36:12
#key: P:36:13
#key: P:36:14
#key: P:36:15
#key: P:36:16
#key: P:36:17
#key: P:36:18
#key: P:36:19
#key: P:36:20
#key: P:36:21
#key: P:36:22
#key: P:36:23
#key: P:36:24
#key: P:36:25
#key: P:36:26
#key: P:36:27
#key: P:36:28
#key: P:36:29
#key: P:36:30
#key: P:36:31
#key: P:36:32
#key: P:36:33
#key: P:36:34
#key: P:36:35
#key: P:36:36
#key: P:36:37
#key: P:36:38
#key: P:36:39
#key: P:36:40
#key: P:36:41
#key: P:36:42
#key: P:36:43
#key: P:36:44
#key: P:36:45
#key: P:36:46
#key: P:36:47
#key: P:36:48
#key: P:36:49
#key: P:36:50
#key: P:36:51
#key: P:36:52
#key: P:36:53
#key: P:36:54
#key: P:36:55
#key: P:36:56
#key: P:36:57
#key: P:36:58
#key: P:36:59
#key: P:36:60
#key: P:36:61
#key: P:36:62
#key: P:36:63
#key: validatorsKey:35
#key: validatorsKey:1
#key: H:37
#key: P:37:0
#key: P:37:1
#key: P:37:2
#key: P:37:3
#key: P:37:4
#key: P:37:5
#key: P:37:6
#key: P:37:7
#key: P:37:8
#key: P:37:9
#key: P:37:10
#key: P:37:11
#key: P:37:12
#key: P:37:13
#key: P:37:14
#key: P:37:15
#key: P:37:16
#key: P:37:17
#key: P:37:18
#key: P:37:19
#key: P:37:20
#key: P:37:21
#key: P:37:22
#key: P:37:23
#key: P:37:24
#key: P:37:25
#key: P:37:26
#key: P:37:27
#key: P:37:28
#key: P:37:29
#key: P:37:30
#key: P:37:31
#key: P:37:32
#key: P:37:33
#key: P:37:34
#key: P:37:35
#key: P:37:36
#key: P:37:37
#key: P:37:38
#key: P:37:39
#key: P:37:40
#key: P:37:41
#key: P:37:42
#key: P:37:43
#key: P:37:44
#key: P:37:45
#key: P:37:46
#key: P:37:47
#key: P:37:48
#key: P:37:49
#key: P:37:50
#key: P:37:51
#key: P:37:52
#key: P:37:53
#key: P:37:54
#key: P:37:55
#key: P:37:56
#key: P:37:57
#key: P:37:58
#key: P:37:59
#key: P:37:60
#key: P:37:61
#key: P:37:62
#key: P:37:63
#key: validatorsKey:36
#key: validatorsKey:1
#key: H:38
#key: P:38:0
#key: P:38:1
#key: P:38:2
#key: P:38:3
#key: P:38:4
#key: P:38:5
#key: P:38:6
#key: P:38:7
#key: P:38:8
#key: P:38:9
#key: P:38:10
#key: P:38:11
#key: P:38:12
#key: P:38:13
#key: P:38:14
#key: P:38:15
#key: P:38:16
#key: P:38:17
#key: P:38:18
#key: P:38:19
#key: P:38:20
#key: P:38:21
#key: P:38:22
#key: P:38:23
#key: P:38:24
#key: P:38:25
#key: P:38:26
#key: P:38:27
#key: P:38:28
#key: P:38:29
#key: P:38:30
#key: P:38:31
#key: P:38:32
#key: P:38:33
#key: P:38:34
#key: P:38:35
#key: P:38:36
#key: P:38:37
#key: P:38:38
#key: P:38:39
#key: P:38:40
#key: P:38:41
#key: P:38:42
#key: P:38:43
#key: P:38:44
#key: P:38:45
#key: P:38:46
#key: P:38:47
#key: P:38:48
#key: P:38:49
#key: P:38:50
#key: P:38:51
#key: P:38:52
#key: P:38:53
#key: P:38:54
#key: P:38:55
#key: P:38:56
#key: P:38:57
#key: P:38:58
#key: P:38:59
#key: P:38:60
#key: P:38:61
#key: P:38:62
#key: P:38:63
#key: validatorsKey:37
#key: validatorsKey:1
#key: H:39
#key: P:39:0
#key: P:39:1
#key: P:39:2
#key: P:39:3
#key: P:39:4
#key: P:39:5
#key: P:39:6
#key: P:39:7
#key: P:39:8
#key: P:39:9
#key: P:39:10
#key: P:39:11
#key: P:39:12
#key: P:39:13
#key: P:39:14
#key: P:39:15
#key: P:39:16
#key: P:39:17
#key: P:39:18
#key: P:39:19
#key: P:39:20
#key: P:39:21
#key: P:39:22
#key: P:39:23
#key: P:39:24
#key: P:39:25
#key: P:39:26
#key: P:39:27
#key: P:39:28
#key: P:39:29
#key: P:39:30
#key: P:39:31
#key: P:39:32
#key: P:39:33
#key: P:39:34
#key: P:39:35
#key: P:39:36
#key: P:39:37
#key: P:39:38
#key: P:39:39
#key: P:39:40
#key: P:39:41
#key: P:39:42
#key: P:39:43
#key: P:39:44
#key: P:39:45
#key: P:39:46
#key: P:39:47
#key: P:39:48
#key: P:39:49
#key: P:39:50
#key: P:39:51
#key: P:39:52
#key: P:39:53
#key: P:39:54
#key: P:39:55
#key: P:39:56
#key: P:39:57
#key: P:39:58
#key: P:39:59
#key: P:39:60
#key: P:39:61
#key: P:39:62
#key: P:39:63
#key: validatorsKey:38
#key: validatorsKey:1
#key: H:40
#key: P:40:0
#key: P:40:1
#key: P:40:2
#key: P:40:3
#key: P:40:4
#key: P:40:5
#key: P:40:6
#key: P:40:7
#key: P:40:8
#key: P:40:9
#key: P:40:10
#key: P:40:11
#key: P:40:12
#key: P:40:13
#key: P:40:14
#key: P:40:15
#key: P:40:16
#key: P:40:17
#key: P:40:18
#key: P:40:19
#key: P:40:20
#key: P:40:21
#key: P:40:22
#key: P:40:23
#key: P:40:24
#key: P:40:25
#key: P:40:26
#key: P:40:27
#key: P:40:28
#key: P:40:29
#key: P:40:30
#key: P:40:31
#key: P:40:32
#key: P:40:33
#key: P:40:34
#key: P:40:35
#key: P:40:36
#key: P:40:37
#key: P:40:38
#key: P:40:39
#key: P:40:40
#key: P:40:41
#key: P:40:42
#key: P:40:43
#key: P:40:44
#key: P:40:45
#key: P:40:46
#key: P:40:47
#key: P:40:48
#key: P:40:49
#key: P:40:50
#key: P:40:51
#key: P:40:52
#key: P:40:53
#key: P:40:54
#key: P:40:55
#key: P:40:56
#key: P:40:57
#key: P:40:58
#key: P:40:59
#key: P:40:60
#key: P:40:61
#key: P:40:62
#key: P:40:63
#key: validatorsKey:39
#key: validatorsKey:1
#key: H:41
#key: P:41:0
#key: P:41:1
#key: P:41:2
#key: P:41:3
#key: P:41:4
#key: P:41:5
#key: P:41:6
#key: P:41:7
#key: P:41:8
#key: P:41:9
#key: P:41:10
#key: P:41:11
#key: P:41:12
#key: P:41:13
#key: P:41:14
#key: P:41:15
#key: P:41:16
#key: P:41:17
#key: P:41:18
#key: P:41:19
#key: P:41:20
#key: P:41:21
#key: P:41:22
#key: P:41:23
#key: P:41:24
#key: P:41:25
#key: P:41:26
#key: P:41:27
#key: P:41:28
#key: P:41:29
#key: P:41:30
#key: P:41:31
#key: P:41:32
#key: P:41:33
#key: P:41:34
#key: P:41:35
#key: P:41:36
#key: P:41:37
#key: P:41:38
#key: P:41:39
#key: P:41:40
#key: P:41:41
#key: P:41:42
#key: P:41:43
#key: P:41:44
#key: P:41:45
#key: P:41:46
#key: P:41:47
#key: P:41:48
#key: P:41:49
#key: P:41:50
#key: P:41:51
#key: P:41:52
#key: P:41:53
#key: P:41:54
#key: P:41:55
#key: P:41:56
#key: P:41:57
#key: P:41:58
#key: P:41:59
#key: P:41:60
#key: P:41:61
#key: P:41:62
#key: P:41:63
#key: validatorsKey:40
#key: validatorsKey:1
#key: H:42
#key: P:42:0
#key: P:42:1
#key: P:42:2
#key: P:42:3
#key: validatorsKey:41
#key: validatorsKey:1
#key: H:43
#key: P:43:0
#key: validatorsKey:42
#key: validatorsKey:1
#key: H:44
#key: P:44:0
#key: validatorsKey:43
#key: validatorsKey:1
#key: H:45
#key: P:45:0
#key: validatorsKey:44
#key: validatorsKey:1
#key: H:46
#key: P:46:0
#key: validatorsKey:45
#key: validatorsKey:1
#key: H:47
#key: P:47:0
#key: validatorsKey:46
#key: validatorsKey:1
#key: H:48
#key: P:48:0
#key: validatorsKey:47
#key: validatorsKey:1
#key: H:49
#key: P:49:0
#key: validatorsKey:48
#key: validatorsKey:1
#key: H:50
#key: P:50:0
#key: validatorsKey:49
#key: validatorsKey:1
#key: H:51
#key: P:51:0
#key: validatorsKey:50
#key: validatorsKey:1
#key: H:52
#key: P:52:0
#key: validatorsKey:51
#key: validatorsKey:1
#key: H:53
#key: P:53:0
#key: validatorsKey:52
#key: validatorsKey:1
#key: H:54
#key: P:54:0
#key: validatorsKey:53
#key: validatorsKey:1
#key: H:55
#key: P:55:0
#key: validatorsKey:54
#key: validatorsKey:1
#key: stateKey
#key: stateKey
#key: AppRetainHeightKey
#key: DCBlockRetainHeightKey
#key: ABCIResRetainHeightKey
#key: SC:55
#key: EC:55
#key: AppRetainHeightKey
#key: DCBlockRetainHeightKey
#key: ABCIResRetainHeightKey
#key: lastABCIResponsesRetainHeight
#key: TxIndexerRetainHeightKey
#key: block_eventsBlockIndexerRetainHeightKey
#key: SC:55
#key: validatorsKey:55
#key: validatorsKey:1
#key: validatorsKey:55
#key: validatorsKey:1
#key: H:55
#key: validatorsKey:56
#key: validatorsKey:1
#key: validatorsKey:56
#key: validatorsKey:1
#key: H:56
#key: validatorsKey:57
#key: validatorsKey:1
#key: validatorsKey:57
#key: validatorsKey:1
#key: H:57
#key: validatorsKey:58
#key: validatorsKey:1
#key: validatorsKey:58
#key: validatorsKey:1
#key: validatorsKey:58
#key: validatorsKey:1
#key: H:58
#key: validatorsKey:59
#key: validatorsKey:1
#key: validatorsKey:59
#key: validatorsKey:1
#key: H:59
#key: validatorsKey:60
#key: validatorsKey:1
#key: validatorsKey:60
#key: validatorsKey:1
#key: H:60
#key: validatorsKey:61
#key: validatorsKey:1
#key: validatorsKey:61
#key: validatorsKey:1
#key: H:61
#key: AppRetainHeightKey
#key: DCBlockRetainHeightKey
#key: ABCIResRetainHeightKey
#key: TxIndexerRetainHeightKey
#key: block_eventsBlockIndexerRetainHeightKey
#key: validatorsKey:62
#key: validatorsKey:1
#key: validatorsKey:62
#key: validatorsKey:1
#key: validatorsKey:62
#key: validatorsKey:1
#key: H:62
#key: validatorsKey:63
#key: validatorsKey:1
#key: validatorsKey:63
#key: validatorsKey:1
#key: H:63
#key: validatorsKey:64
#key: validatorsKey:1
#key: validatorsKey:64
#key: validatorsKey:1
#key: H:64
#key: validatorsKey:65
#key: validatorsKey:1
#key: validatorsKey:65
#key: validatorsKey:1
#key: H:65

The above are the results I got from running main (148eebd) CometBFT w/ a slightly modified cometbft-db (added the fmt.Println statement to Get func) using docker localnet. As you can see, the access pattern is very much sequential. We access block header X followed by any block parts plus validator set X-1 and the last validator set where changes were made (in this example, it's 1 because the validator set never changed). If pruning is enabled, we access static keys like AppRetainHeightKey.

The ideal layout, therefore, is one where the header, block parts, seen commit, ext commit, commit, and validator set from the previous height are grouped under the single prefix:

header: 11/0
block party: 11/1/0
block part 2: 11/1/1
...
seen commit: 11/2
ext commit: 11/3
commit: 11/4

validator set (H-1): 11/5

Note that #1764 is still an improvement over the old design because keys are lex sorted.

Note 2: While these are very simple experiments and in the actual network, the access pattern is more random (light clients asking for headers, nodes synching after downtime, RPC fetching blocks), sticking to grouping under a single height is best because well there's close to little pattern in random requests from the entities above.

melekes added a commit that referenced this issue Dec 13, 2023
@melekes
implement ideal key layout
4fc8862 
#1041 (comment)

Most keys are now grouped by height with the exception of block hash's key (-1)
and pending evidence (-2):

```
header: 11/0
block part: 11/1/0
block part 2: 11/1/1
...
commit: 11/2
seen commit: 11/3
ext commit: 11/4

validator set: 11/5
consensus params: 11/6
abci responses: 11/7

committed evidence: 11/8
```

```
block hash: -1/{hash}
pending evidence: -2/11/{ev.Hash}
```

```
light block: {chainID}/9/11
size: {chainID}/10
```

Refs #1041
@melekes melekes mentioned this issue Dec 13, 2023
Closed
3 tasks
@melekes melekes moved this from Todo to In Progress in CometBFT 2023 Dec 19, 2023
@adizere adizere modified the milestones: 2023-Q4, 2024-Q1 Jan 10, 2024
@adizere adizere added this to CometBFT Jan 11, 2024
@github-project-automation github-project-automation bot moved this to Todo in CometBFT Jan 11, 2024
@adizere adizere moved this from Todo to Blocked in CometBFT Jan 11, 2024
This was referenced Jan 17, 2024
Closed
Closed
@jmalicevic jmalicevic mentioned this issue Feb 19, 2024
Merged
4 tasks
@jmalicevic jmalicevic moved this from Blocked to In Progress in CometBFT Feb 19, 2024
@melekes melekes assigned jmalicevic and unassigned melekes Mar 4, 2024
@adizere adizere linked a pull request Mar 12, 2024 that will close this issue
feat(store)!: Support for different key layout #2327
Merged
4 tasks
github-merge-queue bot pushed a commit that referenced this issue Mar 15, 2024
@jmalicevic @melekes
feat(store)!: Support for different key layout (#2327)
55638e8 
Closes #2057 , #1041

If we keep the current design, we are also closing
#1822

Supersedes #1764 

This PR implements support for an additional DB key representation. The
different key layout sorts the entries by height instead of
lexicographically as in the current version of Comet.

When starting this work, we hoped that the new layout would
significantly outperform the current layout. As we do not have
sufficient real world evidence for this, this PR introduces a DB key
layout interface that would allow Comet to easily integrate a more
preferential key representation without major breaking changes.

The layout using ordercode is introduced as experimental, allowing users
to easily experiment with this.

This layout was thoroughly tested as part of #1044 and all results will
be in a report closing the mentioned PR.
 
Locally tested:
- Empty stores get initialized with v2
- Existing stores without a version key get initialized to v1 and the
key is set
- When a nodes' stores are deleted and we boot it up again that node
uses v2 while the rest of the nodes use v1
<!--

Please add a reference to the issue that this PR addresses and indicate
which
files are most critical to review. If it fully addresses a particular
issue,
please include "Closes #XXX" (where "XXX" is the issue number).

If this PR is non-trivial/large/complex, please ensure that you have
either
created an issue that the team's had a chance to respond to, or had some
discussion with the team prior to submitting substantial pull requests.
The team
can be reached via GitHub Discussions or the Cosmos Network Discord
server in
the #cometbft channel. GitHub Discussions is preferred over Discord as
it
allows us to keep track of conversations topically.
https://github.com/cometbft/cometbft/discussions

If the work in this PR is not aligned with the team's current
priorities, please
be advised that it may take some time before it is merged - especially
if it has
not yet been discussed with the team.

See the project board for the team's current priorities:
https://github.com/orgs/cometbft/projects/1

-->

---

#### PR checklist

- [x] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
@github-project-automation github-project-automation bot moved this from In Progress to Done in CometBFT Mar 15, 2024
mergify bot pushed a commit that referenced this issue Mar 18, 2024
@jmalicevic @melekes @mergify
feat(store)!: Support for different key layout (#2327)
e5b6492 
Closes #2057 , #1041

If we keep the current design, we are also closing
#1822

Supersedes #1764

This PR implements support for an additional DB key representation. The
different key layout sorts the entries by height instead of
lexicographically as in the current version of Comet.

When starting this work, we hoped that the new layout would
significantly outperform the current layout. As we do not have
sufficient real world evidence for this, this PR introduces a DB key
layout interface that would allow Comet to easily integrate a more
preferential key representation without major breaking changes.

The layout using ordercode is introduced as experimental, allowing users
to easily experiment with this.

This layout was thoroughly tested as part of #1044 and all results will
be in a report closing the mentioned PR.

Locally tested:
- Empty stores get initialized with v2
- Existing stores without a version key get initialized to v1 and the
key is set
- When a nodes' stores are deleted and we boot it up again that node
uses v2 while the rest of the nodes use v1
<!--

Please add a reference to the issue that this PR addresses and indicate
which
files are most critical to review. If it fully addresses a particular
issue,
please include "Closes #XXX" (where "XXX" is the issue number).

If this PR is non-trivial/large/complex, please ensure that you have
either
created an issue that the team's had a chance to respond to, or had some
discussion with the team prior to submitting substantial pull requests.
The team
can be reached via GitHub Discussions or the Cosmos Network Discord
server in
the #cometbft channel. GitHub Discussions is preferred over Discord as
it
allows us to keep track of conversations topically.
https://github.com/cometbft/cometbft/discussions

If the work in this PR is not aligned with the team's current
priorities, please
be advised that it may take some time before it is merged - especially
if it has
not yet been discussed with the team.

See the project board for the team's current priorities:
https://github.com/orgs/cometbft/projects/1

-->

---

#### PR checklist

- [x] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
(cherry picked from commit 55638e8)
@mergify mergify bot mentioned this issue Mar 18, 2024
Merged
4 tasks
melekes pushed a commit that referenced this issue Mar 18, 2024
@mergify @jmalicevic
feat(store)!: Support for different key layout (backport #2327) (#2649)
63471c8 
Closes #2057 , #1041

If we keep the current design, we are also closing
#1822

Supersedes #1764 

This PR implements support for an additional DB key representation. The
different key layout sorts the entries by height instead of
lexicographically as in the current version of Comet.

When starting this work, we hoped that the new layout would
significantly outperform the current layout. As we do not have
sufficient real world evidence for this, this PR introduces a DB key
layout interface that would allow Comet to easily integrate a more
preferential key representation without major breaking changes.

The layout using ordercode is introduced as experimental, allowing users
to easily experiment with this.

This layout was thoroughly tested as part of #1044 and all results will
be in a report closing the mentioned PR.
 
Locally tested:
- Empty stores get initialized with v2
- Existing stores without a version key get initialized to v1 and the
key is set
- When a nodes' stores are deleted and we boot it up again that node
uses v2 while the rest of the nodes use v1


---

#### PR checklist

- [x] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec
<hr>This is an automatic backport of pull request #2327 done by
[Mergify](https://mergify.com).

Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jmalicevic jmalicevic

Labels
P:storage-optimization Priority: Give operators greater control over storage and storage optimization
Projects
No open projects
CometBFT
Status: Done
CometBFT 2023
Status: In Progress
Milestone
2024-Q1
3 participants
@melekes @adizere @jmalicevic
0