Better management of `timeout` params #2266

adizere · 2024-02-08T08:55:10Z

adizere
Feb 8, 2024
Maintainer

Better management of `timeout` params

This is feedback from app-chain team:

It’s becoming increasily hard to maintain consistency on these values. Our validators change them frequently as they assume it will help them with uptime, any thoughts on making these consensus params?

timeout_propose = "3s"
timeout_propose_delta = "500ms"
timeout_prevote = "1s"
timeout_prevote_delta = "500ms"
timeout_precommit = "1s"
timeout_precommit_delta = "500ms"
timeout_commit = "1s"

skip_timeout_commit = false

Historical docs

ADR 74
- The previous team reached the decision to move these params on-chain and the implementation was supposed to land in 0.36
- The work depends on soft upgrades
- Implementation considerations in RFC 009
Related issues

Previous comments from the community and core devs

Pros and cons

Pro: Simplify somehow block time management:

wanting longer or shorter block times is probably something app devs would want
but usually app devs do not understand the nitty gritty details of the consensus timeouts

Pro: We should timeouts into consensus params, specifically TimeoutCommit

At the very least it feels really wrong 8000 that TimeoutCommit is a local value instead of a network wide value. It (together with SkipTimeoutCommit ) are currently the only way to establish (a network wide) block time interval. This is definetely sth that needs to be a network global param.
It is not the case that every chain wants to produce blocks as fast as possible. Sure, pos networks are heterogeneous but I think it makes sense to establish some minimum requirements in terms of bandwidth/cpu and such per network. Hence, I am not sure having all these other values local is a good idea in general. Especially, given that a single large validator could change the block time or at least mess with network stability and force everyone into restarting many rounds or other missing a bunch of blocks. With abci++ it seems at least giving validators more time to propose (and hence the need to increase the propose timeout per app/network) is also a requirement.
Zarko and others bring up a relation/dependencies between these timeouts in general. But if they exist they are really very implicit. This needs to be documented better and ideally the config (or network wide params) would even be validated.
Agree re TimeoutCommit. I always consider it application specific parameter, and not something internal to the consensus. Re dependencies between timeouts, it is a known issue (tendermint/tendermint#2920 (comment)), and in my view we definitely need to do something about it; at minimum to better document it, but probably better would be to either compute it from system parameters, or to enforce it by validation.

Against: E2E argument:

not sure how we can set timeouts as consensus params as we don’t have homogeneous network. Consensus timeouts are end-to-end property and proper value depends on various parameters: network topology (inter node latency), node bandwidth/cpu, node set up, load, block size, etc. Having this local gives validator flexibility to adjust it based on its set up; I don’t understand how they would adjust once this become consensus param.
to adjust it, you'd have to change it via governance
Which frankly does not make any sense in my view. I just don’t see how we can choose values for timeouts that works for all validators in heterogenous, dynamic wide area network.
Nodes have different end-to-end latencies in heterogeneous networks like the one tendermint is operated in, so when A is a proposer it can take 300 ms to reach all, while with proposer B it can take 500 ms. This is something dynamic as it depends on network topology. Therefore, how you pick one value for timeoutPropose? In my view, we should figure out how to better support node operators picking right timeouts, and note that if we really want to optimise network, timeouts might even be dynamic and change based on current proposer, block size, network topology, etc. I am talking here more from conceptual perspective of network topology and uncertainty of end to end latencies on top of gossip. Has anyone talked with node operators regarding this and if they find timeouts useful tool to adjust in case of potentially changing network conditions?

Against: it's slow to upgrade these params via governance:

Even if we want network to align on timeout values, I don’t see chain governance as a proper way for doing it, as it is super slow. It is very operational aspect of the network, and in my view, does not feel like something that is good fit for consensus parameter.
“super slow” chain governance is however a property of the application not of Tendermint. Theoretically applications could have protocols that adjust the timeouts of the network per proposer (i.e. each block)

A balanced approach:

we probably want to keep to a minimum what is in consensus params and only put things there if there are very good reasons from the consensus/protocol perspective
TimeoutCommit is not really a consensus timeout, although it does potentially affect consensus. So that could potentially be a candidate for a consensus param, although I don’t understand why would node operators synchronise (pick) right timeout values using chain governance? I would assume that node operators have a communication channel open as they need to sync because of upgrades and security issues, so why not using the same channel to align on timeout values?
going in a direction of deriving timeouts from small set of system parameters seems like more reasonable path, especially as end-to-end network latency is something more intuitive to node operators. Maybe this could be consensus param, but again, not sure I understand why we would need chain governance to agree on that.

A summary:

locally chosen (very differently) consensus timeouts could lead to liveness issues due to round synchronisation, 2) we currently have two many timeout config parameters, so it might be tricky for node operator to pick right values. Relations between different timeouts need to be respected, otherwise we could have liveness issue. 3) we need a way to change timeout values to react to changing network topology and runtime parameters.

Interactions between timeouts and ABCI++

Interactions with FinalizeBlock:

Let me explain the case of TimeoutPropose. We decided to move this parameter to ConsensusParams in one of the ABCI++ design meetings last December. The main reason being that, with the introduction of ABCI++, we now enable the Application to (optionally) fully execute the proposed block. This means that, now, the block execution stands in the "critical" path, so if the proposer's App takes a long time to execute the block, the other nodes have the risk to timeout and prevote nil.
After a few days looking for a solution that could be fully managed by Tendermint we decided that it had to be the App's responsibility as it knows best how long executing PrepareProposalis likely to take. This had two implications:
Moving TimeoutPropose to ConsensusParams, so that the App has control over it and can make it evolve depending on changing App workload conditions
In the spec, we added a new requirement (see the requirements, Requirement 3: "timeliness") to make it explicit that it is up to the Application to ensure (via controlling the value of TimeoutPropose) that non-proposers don't time out waiting for the proposal because PrepareProposal took too long.

I have slightly different view to this issue. In my view, PrepareProposal is indeed potentially extending the overall block propagation time (could be significant) and therefore needs to be take into consideration for TimeoutPropose. However, I would not equal it to TimeoutPropose as it does not cover whole story (block propagation time). In my view, an application can compute/estimate how much TimeoutPropose should be extended (note that it can still be negligible if an application decides not to use PrepareProposal), and this value should be added to the base TimeoutPropose that should be mostly dominated by the network delays (plus signature verification most probably). In my view, exposing TimeoutPropose directly to the end user (be it node operator or app developer) is not a good practice as it is implementation detail: we should get input from users regarding application properties and network properties and then derive proper values for timeouts behind the scene.
So you are basically saying that the app should be able to add a global constant to timeoutPropose?
Yes

Next steps

Clarify the different underlying concerns that users care about
- block time configuration?
- simplicity of configuration & reducing footguns?
Discuss at community calls & establish a path towards short-term and long-term fixes
- short-term: better docs & communicating sane defaults
- long-term: potential UX improvements by moving or consolidating (some) timeout parameters

cason · 2024-02-08T10:18:21Z

cason
Feb 8, 2024

We could get rid of skip_timeout_commit. We can implement it by setting timeout_commit to 0.

timeout_commit is in fact the minimum latency for a block. It should match at all validators. The use of this parameters for full nodes that are not validators, is not clear at all to me.

Regarding to the argument that changing consensus params is too slow, I think that the minimum latency for a block is something decided at genesis, and changing it has relevant impact on how users perceive performance, so changing should indeed require some discussion, which means time to discuss.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Better management of `timeout` params #2266

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Better management of timeout params #2266

Uh oh!

Uh oh!

adizere Feb 8, 2024 Maintainer