10000 feat(e2e): Added e2e config to disable mempool lanes by jmalicevic · Pull Request #3993 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(e2e): Added e2e config to disable mempool lanes #3993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 139 commits into from
Sep 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
c198134
Extend `InfoResponse` and `CheckTxResponse` with lanes info
hvanz Jul 11, 2024
79709e2
Add lanes to kvstore
hvanz Jul 11, 2024
54509af
Add lanes to e2e's app
hvanz Jul 11, 2024
4f64cac
Merge branch 'feature/mempool-qos' into jasmina/3503-mempool-lanes-to…
jmalicevic Aug 5, 2024
946a974
Added changelog
jmalicevic Aug 5, 2024
b9c88da
fetch lanes info from app
hvanz Jul 10, 2024
8912a24
Moved Info call outside Handshake
jmalicevic Aug 5, 2024
261b83a
Fixed mempool unit tests
jmalicevic Aug 6, 2024
0c06f56
Reverted go API breaking change
jmalicevic Aug 6, 2024
f7f2e15
Add lanes to kvstore
hvanz Jul 11, 2024
a4740bf
Add lanes to e2e's app
hvanz Jul 11, 2024
206fe8c
Added changelog
jmalicevic Aug 5, 2024
2f7e084
fetch lanes info from app
hvanz Jul 10, 2024
a105e24
Moved Info call outside Handshake
jmalicevic Aug 5, 2024
b7c3c3e
Fixed mempool unit tests
jmalicevic Aug 6, 2024
18e4685
Reverted go API breaking change
jmalicevic Aug 6, 2024
649bf2a
Merge branch 'jasmina/3457-mempool-lanes-app-query' of github.com:com…
jmalicevic Aug 6, 2024
357ab87
Merge branch 'feature/mempool-qos' into jasmina/3457-mempool-lanes-ap…
jmalicevic Aug 6, 2024
6167d38
Added changelog
jmalicevic Aug 6, 2024
630b483
Added validation unit test, fixed check for empty list
jmalicevic Aug 6, 2024
f74230f
Unified naming accross errors
jmalicevic Aug 6, 2024
bcad9a1
Moved errors to errors file
jmalicevic Aug 6, 2024
40acb97
Merge branch 'feature/mempool-qos' into jasmina/3457-mempool-lanes-ap…
jmalicevic Aug 6, 2024
8924fd7
Update .changelog/unreleased/features/3634-query-app-for-lanes-info.md
jmalicevic Aug 6, 2024
e89cb6c
Fixed comments in lane info validation
jmalicevic Aug 6, 2024
ac16404
Apply suggestions from code review
jmalicevic Aug 6, 2024
a969168
t cMerge branch 'jasmina/3457-mempool-lanes-app-query' of github.com:…
jmalicevic Aug 6, 2024
8249a12
Updated test name
jmalicevic Aug 6, 2024
9d0b30a
Prepare recheck for lanes: we will call recheck multiple, consecutive…
hvanz Jul 10, 2024
aec9adf
Add lanes to CListMempool, including WRR scheduling algorithm.
hvanz Jul 10, 2024
8d19465
Apply suggestions from code review
jmalicevic Aug 12, 2024
827f40c
Mock Info calls with empty response
jmalicevic Aug 12, 2024
2c4782a
Updated TestMempoolIterator to just count
jmalicevic Aug 12, 2024
ad13c72
Initial version of test reproducing lane order, reproduces the e2e be…
jmalicevic Aug 12, 2024
da06145
Merge branch 'feature/mempool-qos' into jasmina/3457-mempool-lanes-ap…
hvanz Aug 13, 2024
fbfc4f1
Updated kvstore to increase chances of more than one lane being used
jmalicevic Aug 13, 2024
7069564
Fixed check for empty lanePriorities - they will never be nil if not …
jmalicevic Aug 13, 2024
985a07b
Reverse sort of the priorities - needs to be done more elegantly by p…
jmalicevic Aug 13, 2024
3987a13
Updated test
jmalicevic Aug 13, 2024
b37ce8a
Merge branch 'jasmina/3457-mempool-lanes-app-query' into jasmina/3484…
jmalicevic Aug 13, 2024
064853b
Removed deprecated methods
jmalicevic Aug 13, 2024
282ab7e
Fixed failing tests due to expected order of txs
jmalicevic Aug 13, 2024
83327f7
Updated tests to either disable lanes or pass lane info to proxyApp
jmalicevic Aug 14, 2024
c9112e7
Add test to make sure mempool is blocked until a tx comes in
jmalicevic Aug 14, 2024
8cb6660
blocking when all lanes empty. 1st try
sergio-mena Aug 14, 2024
90cb9c3
Added test to check for race conditions
jmalicevic Aug 16, 2024
5c04ea8
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
jmalicevic Aug 16, 2024
f158921
Added laneSize to debug message
jmalicevic Aug 16, 2024
cb63168
Apply suggestions from code review
jmalicevic Aug 16, 2024
38f8200
Added lane size in debug message of removal
jmalicevic Aug 16, 2024
887178b
Merge branch 'jasmina/3484-add-lanes-to-mempool' of github.com:cometb…
jmalicevic Aug 16, 2024
153f64e
Merge branch 'feature/mempool-qos' into hvanz/merge-feature-mempool-qos
hvanz Aug 16, 2024
5650478
Make all txs-related variables guarded by txsMtx
hvanz Aug 16, 2024
ad86e74
Comment out failing check in `TestMempoolIteratorRace`
hvanz Aug 16, 2024
0cd1198
Change `updateMtx.RLock` to `updateMtx.Lock` in `Flush` so that tests…
hvanz Aug 16, 2024
efaff6f
chore(mempool/lanes): merge feature/mempool-qos (#3724)
hvanz Aug 16, 2024
2d8c323
initial version of simpler recheck
jmalicevic Aug 16, 2024
6e25a5b
Diversified lane assgnment in kvstore app
jmalicevic Aug 19, 2024
4606c76
Updated tests
jmalicevic Aug 19, 2024
7397b06
feat(mempool/lanes): Store lane in `mempoolTx` and remove `txsLanes` …
hvanz Aug 19, 2024
9f2f75a
BenchTest
jmalicevic Aug 20, 2024
39bd658
Merge branch 'jasmina/3484-add-lanes-to-mempool' of github.com:cometb…
jmalicevic Aug 20, 2024
a90c4bd
Merge branch 'feature/mempool-qos' into hvanz/lanes-merge-feature-branch
hvanz Aug 23, 2024
3f5904c
chore(lanes): merge feature branch (#3846)
hvanz Aug 23, 2024
ad084b7
fix benchmark
hvanz Aug 23, 2024
b0307fd
Revert "initial version of simpler recheck"
hvanz Aug 23, 2024
d5acb7e
Modify selection algorithm in apps to make them testable
hvanz Aug 20, 2024
87f83d8
Add TestMempoolAddTxLane
hvanz Aug 20, 2024
046364a
simplify test
hvanz Aug 20, 2024
64a9e54
simplify test
hvanz Aug 21, 2024
2e4bc13
revert changes to app.go
hvanz Aug 22, 2024
39222df
empty commit to trigger ci
hvanz Aug 22, 2024
e46e5d2
fix(lanes): revert changes to recheck mechanism (#3833)
hvanz Aug 23, 2024
ddfdb7f
Merge branch 'jasmina/3484-add-lanes-to-mempool' of github.com:cometb…
jmalicevic Aug 23, 2024
9de6cd7
Merge branch 'jasmina/3484-add-lanes-to-mempool' into hvanz/lanes-tes…
mergify[bot] Aug 23, 2024
b0e00f9
Do early returns when code != ok on CheckTx
hvanz Aug 23, 2024
0a6bdf7
Update abci/example/kvstore/kvstore.go
hvanz Aug 23, 2024
024b3a8
comment
hvanz Aug 23, 2024
784d9ca
feat(lanes): Change lane selection algorithm in kvstore app (#3762)
hvanz Aug 23, 2024
279ab9d
feat(lanes): Add non-blocking iterator (#3847)
hvanz Aug 26, 2024
3068f3b
fix(lanes): Use iterators for rechecking (#3862)
hvanz Aug 26, 2024
f6d1797
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
hvanz Aug 26, 2024
89884f8
Removed debugging logs
jmalicevic Aug 26, 2024
258254f
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
hvanz Aug 27, 2024
ba8dfae
fix indentation
hvanz Aug 28, 2024
22d0f15
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
hvanz Aug 28, 2024
ccd9f6e
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
jmalicevic Aug 28, 2024
c093c19
refactor lanes initialisation
hvanz Sep 2, 2024
b51f4bd
refactor: move iterators to iterators.go
hvanz Sep 2, 2024
c57f23c
comments
hvanz Sep 2, 2024
ee52113
remove debug logs
hvanz Sep 2, 2024
e305a4f
restore test/e2e/networks/simple.toml
hvanz Sep 2, 2024
1c3657f
remove unneeded BenchmarkUpdateWithConcurrentCheckTx
hvanz Sep 2, 2024
940ca46
remove commented code
hvanz Sep 2, 2024
9419e87
forgot newline in simple.toml
hvanz Sep 2, 2024
8572947
merge feature branch
jmalicevic Sep 3, 2024
fe89e45
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
hvanz Sep 3, 2024
8451751
Added test to check order without lanes
jmalicevic Sep 3, 2024
9b3f213
revert go.mod
hvanz Sep 3, 2024
a1bd31c
test(mempool/lanes): Use app with lanes in tests + do not test orderi…
hvanz Sep 3, 2024
ed635c3
Merge remote-tracking branch 'origin/jasmina/3484-add-lanes-to-mempoo…
jmalicevic Sep 3, 2024
c227f13
Added DoNotUseLanes to e2e config
jmalicevic Sep 3, 2024
1bde84a
fix(mempool/lanes): Do not reuse iterator when reaping (#3996)
hvanz Sep 4, 2024
643b251
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 4, 2024
91d3714
Added an option for the generator to generate apps w/o lanes for nigh…
jmalicevic Sep 4, 2024
f8c41b5
Merge branch 'jasmina/3977-lanes-tests' of github.com:cometbft/cometb…
jmalicevic Sep 4, 2024
fc8405b
Added optional list of lanes to e2e testnet
jmalicevic Sep 4, 2024
d2fe3ff
Removed printfs and newline
jmalicevic Sep 5, 2024
307c8a2
Removed printf
jmalicevic Sep 5, 2024
c3c80c4
Revert "Added test to check order without lanes"
jmalicevic Sep 5, 2024
44c6ff1
Update test/e2e/generator/generate.go
jmalicevic Sep 5, 2024
d1b4fd8
feat(mempool/test): Add test to check order when lanes are disabled (…
jmalicevic Sep 5, 2024
96b1341
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 5, 2024
0411274
Apply suggestions from code review
jmalicevic Sep 5, 2024
ac2376d
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 5, 2024
2539b8e
Reverted length check condition
jmalicevic Sep 5, 2024
c97f825
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 5, 2024
931ef34
fixed var check
jmalicevic Sep 5, 2024
ac8c13d
Renamed no_lanes variables
jmalicevic Sep 5, 2024
e67f4da
Merge branch 'feature/mempool-qos' into jasmina/3484-add-lanes-to-mem…
hvanz Sep 5, 2024
0e91760
fix logging tx hash
hvanz Sep 5, 2024
2126892
slight refactoring to addTx
hvanz Sep 5, 2024
b623c0e
Merge remote-tracking branch 'origin/jasmina/3484-add-lanes-to-mempoo…
jmalicevic Sep 5, 2024
c6d20d1
Removed redundant lane related field in testnet
jmalicevic Sep 5, 2024
26a86b9
Renamed constructors
jmalicevic Sep 6, 2024
0ff1967
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
17c6f6f
Reset is private now
jmalicevic Sep 6, 2024
e5f741f
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
0ea3071
Changed argument for NewRecheck
jmalicevic Sep 6, 2024
9614e6e
Fixed comment text
jmalicevic Sep 6, 2024
6a0c3f4
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
dcb43cc
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
3c9f921
Closing channels properly- by @sergio-mena
jmalicevic Sep 6, 2024
194e7be
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
ff01cbc
Added reference to mempool into recheck struct
jmalicevic Sep 6, 2024
2c00d46
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
8000
c9fc2fd
Update mempool/clist_mempool.go
jmalicevic Sep 6, 2024
a1371fe
Merge branch 'jasmina/3484-add-lanes-to-mempool' into jasmina/3977-la…
mergify[bot] Sep 6, 2024
58e804b
Merge branch 'feature/mempool-qos' into jasmina/3977-lanes-tests
mergify[bot] Sep 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 38 additions & 8 deletions test/e2e/app/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,17 @@ type Config struct {
// -1 denotes it is set at genesis.
// 0 denotes it is set at InitChain.
PbtsUpdateHeight int64 `toml:"pbts_update_height"`

// If true, disables the use of lanes by the application.
// Used to simulate networks that do not want to use lanes, running
// on top of CometBFT with lane support.
NoLanes bool `toml:"no_lanes"`

// Optional custom definition of lanes to be used by the application
// If not used the application has a default set of lanes:
// {"foo"= 9,"bar"=4,"default"= 1}
// Note that the default key has to be present in your list of custom lanes.
Lanes map[string]uint32 `toml:"lanes"`
}

func DefaultConfig(dir string) *Config {
Expand All @@ -150,13 +161,15 @@ func DefaultConfig(dir string) *Config {
}

// LaneDefinitions returns the (constant) list of lanes and their priorities.
func LaneDefinitions() (map[string]uint32, []uint32) {
func LaneDefinitions(lanes map[string]uint32) (map[string]uint32, []uint32) {
// Map from lane name to its priority. Priority 0 is reserved. The higher
// the value, the higher the priority.
lanes := map[string]uint32{
"foo": 9,
"bar": 4,
defaultLane: 1,
if len(lanes) == 0 {
lanes = map[string]uint32{
"foo": 9,
"bar": 4,
defaultLane: 1,
}
}

// List of lane priorities
8000 Expand All @@ -178,11 +191,18 @@ func NewApplication(cfg *Config) (*Application, error) {
if err != nil {
return nil, err
}
lanes, lanePriorities := LaneDefinitions()

logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout))
logger.Info("Application started!")
if cfg.NoLanes {
return &Application{
logger: logger,
state: state,
snapshots: snapshots,
cfg: cfg,
}, nil
}

lanes, lanePriorities := LaneDefinitions(cfg.Lanes)
return &Application{
logger: logger,
state: state,
Expand All @@ -201,6 +221,14 @@ func (app *Application) Info(context.Context, *abci.InfoRequest) (*abci.InfoResp
}

height, hash := app.state.Info()
if app.cfg.NoLanes {
return &abci.InfoResponse{
Version: version.ABCIVersion,
AppVersion: appVersion,
LastBlockHeight: int64(height),
LastBlockAppHash: hash,
}, nil
}
return &abci.InfoResponse{
Version: version.ABCIVersion,
AppVersion: appVersion,
Expand Down Expand Up @@ -324,8 +352,10 @@ func (app *Application) CheckTx(_ context.Context, req *abci.CheckTxRequest) (*a
time.Sleep(app.cfg.CheckTxDelay)
}

if app.cfg.NoLanes {
return &abci.CheckTxResponse{Code: kvstore.CodeTypeOK, GasWanted: 1}, nil
}
lane := extractLane(value)

return & 6D47 abci.CheckTxResponse{Code: kvstore.CodeTypeOK, GasWanted: 1, Lane: lane}, nil
}

Expand Down
3 changes: 3 additions & 0 deletions test/e2e/generator/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ var (
map[string]string{"initial01": "a", "initial02": "b", "initial03": "c"},
},
"validators": {"genesis", "initchain"},
"no_lanes": {true, false},
}
nodeVersions = weightedChoice{
"": 2,
Expand Down Expand Up @@ -320,6 +321,8 @@ func generateTestnet(r *rand.Rand, opt map[string]any, upgradeVersion string, pr
)
}

manifest.NoLanes = opt["no_lanes"].(bool)

return manifest, nil
}

Expand Down
6 changes: 6 additions & 0 deletions test/e2e/node/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,10 @@ type Config struct {

PbtsEnableHeight int64 `toml:"pbts_enable_height"`
PbtsUpdateHeight int64 `toml:"pbts_update_height"`

NoLanes bool `toml:"no_lanes"`

Lanes map[string]uint32 `toml:"lanes"`
}

// App extracts out the application specific configuration parameters.
Expand All @@ -72,6 +76,8 @@ func (cfg *Config) App() *app.Config {
ABCIRequestsLoggingEnabled: cfg.ABCIRequestsLoggingEnabled,
PbtsEnableHeight: cfg.PbtsEnableHeight,
PbtsUpdateHeight: cfg.PbtsUpdateHeight,
NoLanes: cfg.NoLanes,
Lanes: cfg.Lanes,
}
}

Expand Down
13 changes: 13 additions & 0 deletions test/e2e/pkg/manifest.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ type Manifest struct {

// Weight for each lane defined by the app. The transaction loader will
// assign lanes to generated transactions proportionally to their weights.
// Note that the lanes are sorted by priority in descending order, thus the
// first weight in the array will be the weight of the highest priority lane.
LoadLaneWeights []uint `toml:"load_lane_weights"`

// LogLevel specifies the log level to be set on all nodes.
Expand Down Expand Up @@ -146,6 +148,17 @@ type Manifest struct {
// 0 denotes it is set at InitChain.
PbtsUpdateHeight int64 `toml:"pbts_update_height"`

// Used to disable lanes for testing behavior of
// networks that upgrade to a version of CometBFT
// that supports lanes but do not opt for using them.
NoLanes bool `toml:"no_lanes"`

// Optional custom definition of lanes to be used by the application
// If not used the application has a default set of lanes:
// {"foo"= 9,"bar"=4,"default"= 1}
// Note that the default key has to be present in your list of custom lanes.
Lanes map[string]uint32 `toml:"lanes"`

// Genesis is a set of key-value config entries to write to the
// produced genesis file. The format is "key = value".
// Example: "consensus_params.evidence.max_bytes = 1024".
Expand Down
17 changes: 8 additions & 9 deletions test/e2e/pkg/testnet.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,7 @@ func NewTestnetFromManifest(manifest Manifest, file string, ifd InfrastructureDa
return nil, fmt.Errorf("invalid IP network address %q: %w", ifd.Network, err)
}
// Pre-load hard-coded lane values from app.
_, lanePriorities := app.LaneDefinitions()

_, lanePriorities := app.LaneDefinitions(manifest.Lanes)
testnet := &Testnet{
Manifest: manifest,

Expand Down Expand Up @@ -182,15 +181,15 @@ func NewTestnetFromManifest(manifest Manifest, file string, ifd InfrastructureDa
if testnet.LoadTxSizeBytes == 0 {
testnet.LoadTxSizeBytes = defaultTxSizeBytes
}
if len(testnet.LoadLaneWeights) == 0 {
if len(testnet.Manifest.LoadLaneWeights) == 0 {
// Assign same weight to all lanes.
testnet.LoadLaneWeights = make([]uint, len(testnet.lanePriorities))
testnet.Manifest.LoadLaneWeights = make([]uint, len(testnet.lanePriorities))
for i := 0; i < len(testnet.lanePriorities); i++ {
testnet.LoadLaneWeights[i] = 1
testnet.Manifest.LoadLaneWeights[i] = 1
}
}
// Pre-calculate the sum of all lane weights.
for _, w := range testnet.LoadLaneWeights {
for _, w := range testnet.Manifest.LoadLaneWeights {
testnet.sumWeights += w
}

Expand Down Expand Up @@ -414,13 +413,13 @@ func (t Testnet) Validate() error {
)
}
}
if len(t.LoadLaneWeights) != len(t.lanePriorities) {
if len(t.Manifest.LoadLaneWeights) != len(t.lanePriorities) {
return fmt.Errorf("number of lane weights (%d) must be equal to "+
"the number of lanes defined by the app (%d)",
len(t.LoadLaneWeights), len(t.lanePriorities),
)
}
for _, w := range t.LoadLaneWeights {
for _, w := range t.Manifest.LoadLaneWeights {
if w <= 0 {
return fmt.Errorf("weight must be greater than 0: %v", w)
}
Expand Down Expand Up @@ -650,7 +649,7 @@ func weightedRandomIndex(weights []uint, sumWeights uint) int {
// NextLane returns the next element in the list of lanes, according to a
// predefined weight for each lane in the list.
func (t *Testnet) NextLane() uint32 {
return t.lanePriorities[weightedRandomIndex(t.LoadLaneWeights, t.sumWeights)]
return t.lanePriorities[weightedRandomIndex(t.Manifest.LoadLaneWeights, t.sumWeights)]
}

//go:embed templates/prometheus-yaml.tmpl
Expand Down
2 changes: 2 additions & 0 deletions test/e2e/runner/setup.go
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,8 @@ func MakeAppConfig(node *e2e.Node) ([]byte, error) {
"abci_requests_logging_enabled": node.Testnet.ABCITestsEnabled,
"pbts_enable_height": node.Testnet.PbtsEnableHeight,
"pbts_update_height": node.Testnet.PbtsUpdateHeight,
"no_lanes": node.Testnet.Manifest.NoLanes,
"lanes": node.Testnet.Manifest.Lanes,
}
switch node.ABCIProtocol {
case e2e.ProtocolUNIX:
Expand Down
Loading
0