Closed
Description
Feature Request
Summary
In order to enable better management of transaction traffic at P2P level, we need to allow the app to classify transactions. For that, we introduce the concept of Quality of Service (QoS), well known in IP networks, which we realize as mempool lanes. ADR-118 describes this feature more in detail.
This issue will track:
- the discussion with our users
- design work
- implementation work
- any follow up
Teams interested
- Osmosis
- Injective
- dYdX
- Astria
Execution Steps
As with previous sizeable features, the design and implementation work will be carried out in a feature branch -- feature/mempool-qos, bifurcating from main -- to avoid the risk of last-minute serious problems. A feature branch enables the (management) decision to defer shipping a feature in a new release if it is considered not ready.
If we need our users to validate our design/implementation while this feature is in progress, we will probably need to maintain some temporary branch off a released version (likely v0.38.x) that reflects our work in feature/mempool-qos. We will work out the details as we start creating PRs with code.
Initial steps
- Create branch
feature/mempool-qos - Initial ADR skeleton with structure and discussions
Other related pending improvements on mempool
- mempool: Recheck improvements and fixes #2317
- mempool: Handle concurrent requests in recheck callback #895
-
Remove(not actually needed)TxsAvailableandCreateEmptyBlocks(laundry list)
Design: core work
These tasks are mostly sequential, although some of them can be done in parallel
- ADR: Writeup of Problem definition #2934
- ADR: Writeup of Mempool QoS specification #3027
- ADR: Due-diligence on the three items in "Alternative Approaches" section #3110
- [2 days left]
- mempool QoS ADR: Interaction with users #3088
- ADR: Iterate on the design with output from users
- mempool QoS ADR: Writeup of detailed design #3138
- [2 days left]
- Decide where the design/spec will land [1 day]
- Shall we keep the design/spec in the ADR? Or do we need to create a section in the spec?
- We decided to keep it all in the ADR because it makes sense to understand the whole text. Later we will copy the Properties section to its own file in the spec folder.
- Detailed implementation plan: flesh out tasks in section below
- mempool/lanes: Move ADR from feature branch to
main#3499 - mempool/lanes: Update ADR after initial design #4120
-
Finalize spec/design document (ADR or move elsewhere) [7 days]- We decided to leave the current mini spec as part of the ADR.
Initial estimate (this section): 12 days
Implementation
- mempool: tx could be sent back to sender #3084
- mempool: fix
TxsFrontandTxsWaitChaninCListMempoolthat leak implementation details #3303 - add new method
TryAddTxto mempool reactor #3296Implemented in Dog's code: Take it from there
- Post the rest of the code of current prototype implementation in one or more PRs (in feature branch) [1 day]
- Done in branch hvanz/mempool-lanes-prototype
- mempool/lanes: Extend proto messages with lanes info #3502
- [1 day]
- mempool/lanes: Extend
kvstoreande2eapps with lanes #3503- [3 days]
- mempool/lanes: Query app for lane info #3475
- [6 days]
- Move
Infoout of Handshake (likely) toNewNode
- mempool/lanes: Implement lanes in
CListMempool#3484- [6 days]
- This will change the internal data structures of CListMempool. It will be difficult to split it in multiple tasks/PRs.
- Probably the prioritisation algorithm (WRR) will also be part of this PR.
- Implement lanes as multiple CLists
- Implement WRR scheduling algorithm [5 days]
- Classical WRR is implemented in the prototype, though only tested on the surface.
- Basic testing
- mempool/lanes: Add new metrics for size of each lane on number of txs and bytes #3506
- [1 day]
- mempool/lanes: Unit-test WRR iterator #3974
- [4 days]
- mempool/lanes: Write lane-specific unit tests #3977
- [8 days]
- mempool/lanes: add lanes to e2e tests #4005
- mempool/lanes: Replace Classic WRR scheduling with Interleaved WRR #4011
- mempool/lanes: Enforce limits on lane capacities (number of txs and bytes) #4012
- mempool/lanes: Allow lanes to have the same priority #4014
- mempool/lanes: Validate that app versions match on P2P handshake #4043
- mempool/lanes: Test at scale to validate scheduling algorithm and predictability of latencies #4049
- [7 days]
Initial estimate (this section): 50 days
Final steps
- mempool/lanes: Validate MVP implementation on 200-nodes testnet #4168
- Merge
feature/mempool-qosback intomainwhen the implementation is validated (200 node run, without report) - Figure out how to expose
feature/mempool-qoson top ofv0.38.x,v0.38.x-experimental, orv1.x-experimental[1 day]- chore(mempool/lanes): backport lanes to
v1.x-experimental#4179 -
Deal with the work done offv0.38.x(merge or squash it intov0.38.x-experimental) [7 days] - We have backported Lanes to
v1.x-experimental. Cosmos SDK will integrate Lanes from that branch ([Feature]: Mempool lanes cosmos/cosmos-sdk#21780). For the moment we won't backport tov0.38.xorv0.38.x-experimentalunless there are interested users especially requiring these versions.
- chore(mempool/lanes): backport lanes to
- mempool/lanes: Add documentation for developers on how to use lanes #4138
- Revisit the existing mempool spec (by @hvanz and @otrack): is it compatible with mempool QoS? Do we need to work on it? [2 days]
- There are three specs (not in this repo): one high-level in English, one high-level in Quint, and one low-level in Quint. The two high-level specs are compatible with Lanes. The low-level spec follows closely the v0.38 code because the original goal was to use it to apply MBT. This last spec would need to be updated to be compatible with lanes.
Initial estimate (this section): 12 days
Additional changes after merging to main:
- fix(test): Initialise all mempools with lane info from app #4229
- feat(e2e): Add
load_max_secondsto manifest + optional flags toloadcommand #4175 (required for sending tx load on DigitalOcean testnets) - test(mempool): Add benchmarks for blocking iterator #4181
- fix(kvstore): Remove unused lanes-related variable #4241
- feat(spec): Add lanes to abci spec #4483