ADR 101: Recommended updates to pruner service #1201

thanethomson · 2023-08-02T20:19:56Z

Recommendations to update #1150. Also pulls in the latest changes from the feature branch, resolving the conflict in the docs.

Since this already contains a bunch of commits that have been reviewed (i.e. the ones from the feature branch), please only review from 04a7039 onwards.

PR checklist

Tests written/updated
Changelog entry added in .changelog (we use unclog to manage our changelog)
Updated relevant documentation (docs/ or spec/) and code comments

…dations

Signed-off-by: Thane Thomson <connect@thanethomson.com>

thanethomson · 2023-08-02T20:23:51Z

state/pruner.go

@@ -23,11 +24,14 @@ var (
 type Pruner struct {
 	service.BaseService
 	logger log.Logger
+
+	mtx sync.Mutex


This was added after speaking to @mzabaluev today where we clearly want to serialize all requests that set any retain heights. For example, in the case where users may ignore our recommendations to only attach a single data companion and two calls are made simultaneously to set a particular retain height, allowing both calls to execute simultaneously may have undefined consequences.

We discussed this initially and were hoping to rely on the databases' guarantees that these conflicts should be handled. But I agree this is a much cleaner approach, especially given the fact that we support multiple DB backends atm.

thanethomson · 2023-08-02T20:25:01Z

state/pruner.go

@@ -178,6 +178,24 @@ func (p *Pruner) SetABCIResRetainHeight(height int64) error {
 	return err
 }

+// GetApplicationRetainHeight is a convenience method for accessing the
+// GetApplicationRetainHeight method of the underlying state store.
+func (p *Pruner) GetApplicationRetainHeight() (int64, error) {


These getters are added such that #1154 can access these methods without needing to access the underlying state store.

thanethomson · 2023-08-02T20:25:33Z

state/errors.go

@@ -51,6 +51,20 @@ type (
 	ErrNoABCIResponsesForHeight struct {
 		Height int64
 	}
+
+	ErrPrunerFailedToLoadState struct {


As per #1140.

Signed-off-by: Thane Thomson <connect@thanethomson.com>

…-recommendations

jmalicevic

LGTM, thanks Thane!

@thanethomson

* done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

* done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

…#1476) * indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * Fixed changelog order --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

) * indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * TxIndexer and BlockIndexer pruning metrics (#1334) * added tx and block indexer pruning metrics * add changelog * added base height metric for tx and block indexer * Fixed changelog order --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

* indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * Fixed changelog order * gRPC for Tx and Block indexer pruning (#1327) * done gRPC endpoint for TxIndexer and BlockIndexer pruning * re-ordering function signatures (#1260) * added .changelog --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> * fixed lint --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

…#1476) * indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * Fixed changelog order --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

) * indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * TxIndexer and BlockIndexer pruning metrics (#1334) * added tx and block indexer pruning metrics * add changelog * added base height metric for tx and block indexer * Fixed changelog order --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

* indexer: Implement pruning mechanism (#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> * Fixed bad merge * Fixed changelog structure * Fixed changelog order * gRPC for Tx and Block indexer pruning (#1327) * done gRPC endpoint for TxIndexer and BlockIndexer pruning * re-ordering function signatures (#1260) * added .changelog --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> * fixed lint --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: werty144 <anton-paramonov2000@yandex.ru> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

@thanethomson

* done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unused method Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <connect@thanethomson.com> * store: Format comment for readability Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <connect@thanethomson.com> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <connect@thanethomson.com> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <connect@thanethomson.com> * light: Remove sleep from test Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <connect@thanethomson.com> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <connect@thanethomson.com> * Add changelog entries Signed-off-by: Thane Thomson <connect@thanethomson.com> * docs: Update configuration-related content Signed-off-by: Thane Thomson <connect@thanethomson.com> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <connect@thanethomson.com> * Remove redundunt variable Co-authored-by: Thane Thomson <connect@thanethomson.com> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename helper for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <connect@thanethomson.com> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (cometbft#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Refactor - early returns Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Export Pruner errors Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix error message references Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR 101: Add observer for pruner (cometbft#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Wire in pruner observer Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Make test more deterministic Signed-off-by: Thane Thomson <connect@thanethomson.com> * Fix renames Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <connect@thanethomson.com> * state: Clarify field meanings Signed-off-by: Thane Thomson <connect@thanethomson.com> --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (cometbft#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <connect@thanethomson.com> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Thane Thomson <connect@thanethomson.com> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * ADR-101: Metrics to monitor the pruning (cometbft#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <connect@thanethomson.com> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Adi Seredinschi <a@seredinschi.net> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Mikhail Zabaluev <mikhail@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev>

thanethomson added 7 commits August 2, 2023 15:00

Merge branch 'feature/adr101-pull-companion' into thane/1096-recommen…

03fda01

…dations

state: Reference retain heights instead of pruned heights for clarity

04a7039

Signed-off-by: Thane Thomson <connect@thanethomson.com>

state: Explicitly assign variable values for clarity

ce8a601

Signed-off-by: Thane Thomson <connect@thanethomson.com>

state: Refactor - early returns

801c5db

Signed-off-by: Thane Thomson <connect@thanethomson.com>

state: Add mutex to Pruner to serialize retain height setting requests

4e1b839

Signed-off-by: Thane Thomson <connect@thanethomson.com>

state: Add convenience getters to Pruner

ce82e29

Signed-off-by: Thane Thomson <connect@thanethomson.com>

state: Export Pruner errors

08cd7c4

Signed-off-by: Thane Thomson <connect@thanethomson.com>

thanethomson added this to the 2023-Q3 milestone Aug 2, 2023

thanethomson requested review from a team as code owners August 2, 2023 20:19

thanethomson commented Aug 2, 2023

View reviewed changes

thanethomson requested review from mzabaluev and jmalicevic August 2, 2023 20:26

state: Rename PrunerInterval option to WithPrunerInterval for clarity

5ca8ef4

Signed-off-by: Thane Thomson <connect@thanethomson.com>

This was referenced Aug 2, 2023

ADR 101: Add observer for pruner #1183

Merged

ADR 101: Implement pruning mechanism #1150

Merged

state: Fix error message references

86c1798

Signed-off-by: Thane Thomson <connect@thanethomson.com>

thanethomson force-pushed the thane/1096-recommendations branch from b4ad16b to 86c1798 Compare August 2, 2023 20:57

Merge branch 'jasmina/1096-adr-101-pruning-mechanism' into thane/1096…

5f44e42

…-recommendations

jmalicevic approved these changes Aug 3, 2023

View reviewed changes

thanethomson merged commit 5085f5f into jasmina/1096-adr-101-pruning-mechanism Aug 3, 2023

thanethomson deleted the thane/1096-recommendations branch August 3, 2023 10:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ADR 101: Recommended updates to pruner service #1201

ADR 101: Recommended updates to pruner service #1201

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ADR 101: Recommended updates to pruner service #1201

ADR 101: Recommended updates to pruner service #1201

Uh oh!

Conversation

Uh oh!

PR checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!