cometbft · cason · Jun 20, 2023 · Jun 14, 2023 · Jun 14, 2023 · Jun 14, 2023
@@ -78,7 +78,7 @@ from the node.
 
 The p2p layer implements banning peers by marking them
 as bad and removing them from the list of peers to connect to for *at least* a predefined amount of time. This is done by calling the 
-[`MarkBad`](https://github.com/cometbft/cometbft/blob/main/spec/p2p/v0.34/addressbook.md#bad-peers) routine implemented by the `Switch`.
+[`MarkBad`](https://github.com/cometbft/cometbft/blob/main/spec/p2p/implementation/addressbook.md#bad-peers) routine implemented by the `Switch`.
 If the node does not set the amount of time to be banned, a default value is used. 
 Note that the timing parameter sets the lower bound for when a peer will be unbanned. 
 But the p2p layer will only try to connect to banned peers if the node is not sufficiently connected. Thus the node has no
@@ -510,4 +510,4 @@ Otehr than avoiding relying solely on the response code values, there are no imm
 
 - Github discussions on this RFC:
   - [CometBFT repo - PR \#78](https://github.com/cometbft/cometbft/pull/78)
-  - [Tendermint repo - PR \#9675](https://github.com/tendermint/tendermint/pull/9675)
+  - [Tendermint repo - PR \#9675](https://github.com/tendermint/tendermint/pull/9675)
@@ -4,7 +4,7 @@ The p2p package provides an abstraction around peer-to-peer communication.
 
 Docs:
 
-- [Connection](https://github.com/cometbft/cometbft/blob/main/spec/p2p/connection.md) for details on how connections and multiplexing work
-- [Peer](https://github.com/cometbft/cometbft/blob/main/spec/p2p/node.md) for details on peer ID, handshakes, and peer exchange
-- [Node](https://github.com/cometbft/cometbft/blob/main/spec/p2p/node.md) for details about different types of nodes and how they should work
-- [Config](https://github.com/cometbft/cometbft/blob/main/spec/p2p/config.md) for details on some config option
+- [Connection](https://github.com/cometbft/cometbft/blob/main/spec/p2p/legacy-docs/connection.md) for details on how connections and multiplexing work
+- [Peer](https://github.com/cometbft/cometbft/blob/main/spec/p2p/legacy-docs/node.md) for details on peer ID, handshakes, and peer exchange
+- [Node](https://github.com/cometbft/cometbft/blob/main/spec/p2p/legacy-docs/node.md) for details about different types of nodes and how they should work
+- [Config](https://github.com/cometbft/cometbft/blob/main/spec/p2p/legacy-docs/config.md) for details on some config option
@@ -35,12 +35,12 @@ please submit them to our [bug bounty](https://cometbft.com/security)!
 
 ### P2P and Network Protocols
 
-- [The Base P2P Layer](./p2p/node.md): multiplex the protocols ("reactors") on authenticated and encrypted TCP connections
-- [Peer Exchange (PEX)](./p2p/messages/pex.md): gossip known peer addresses so peers can find each other
-- [Block Sync](./p2p/messages/block-sync.md): gossip blocks so peers can catch up quickly
-- [Consensus](./p2p/messages/consensus.md): gossip votes and block parts so new blocks can be committed
-- [Mempool](./p2p/messages/mempool.md): gossip transactions so they get included in blocks
-- [Evidence](./p2p/messages/evidence.md): sending invalid evidence will stop the peer
+- [The Base P2P Layer](./p2p/legacy-docs/node.md): multiplex the protocols ("reactors") on authenticated and encrypted TCP connections
+- [Peer Exchange (PEX)](./p2p/legacy-docs/messages/pex.md): gossip known peer addresses so peers can find each other
+- [Block Sync](./p2p/legacy-docs/messages/block-sync.md): gossip blocks so peers can catch up quickly
+- [Consensus](./p2p/legacy-docs/messages/consensus.md): gossip votes and block parts so new blocks can be committed
+- [Mempool](./p2p/legacy-docs/messages/mempool.md): gossip transactions so they get included in blocks
+- [Evidence](./p2p/legacy-docs/messages/evidence.md): sending invalid evidence will stop the peer
 
 ### RPC
 

@@ -152,7 +152,7 @@ call sequences of these methods.
 
 State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying
 state machine (application) snapshots instead of replaying historical blocks. For more details, see the
-[state sync documentation](../p2p/messages/state-sync.md).
+[state sync documentation](../p2p/legacy-docs/messages/state-sync.md).
 
 New nodes discover and request snapshots from other nodes in the P2P network.
 A CometBFT node that receives a request for snapshots from a peer will call

@@ -255,7 +255,7 @@ title: Methods
     can be spoofed by adversaries, so applications should employ additional verification schemes
     to avoid denial-of-service attacks. The verified `AppHash` is automatically checked against
     the restored application at the end of snapshot restoration.
-    * For more information, see the `Snapshot` data type or the [state sync section](../p2p/messages/state-sync.md).
+    * For more information, see the `Snapshot` data type or the [state sync section](../p2p/legacy-docs/messages/state-sync.md).
 
 ### ApplySnapshotChunk
 
@@ -775,7 +775,7 @@ Most of the data structures used in ABCI are shared [common data structures](../
     | metadata | bytes  | Arbitrary application metadata, for example chunk hashes or other verification data.                                                                                              | 5            |
 
 * **Usage**:
-    * Used for state sync snapshots, see the [state sync section](../p2p/messages/state-sync.md) for details.
+    * Used for state sync snapshots, see the [state sync section](../p2p/legacy-docs/messages/state-sync.md) for details.
     * A snapshot is considered identical across nodes only if _all_ fields are equal (including
     `Metadata`). Chunks may be retrieved from all nodes that have the same snapshot.
     * When sent across the network, a snapshot message can be at most 4 MB.

@@ -0,0 +1,46 @@
+---
+order: 1
+parent:
+  title: P2P
+  order: 6
+---
+
+# Peer-to-Peer Communication
+
+A CometBFT network is composed of multiple CometBFT instances, hereafter called
+`nodes`, that interact by exchanging messages.
+
+The CometBFT protocols are designed under the assumption of a partially-connected network model.
+This means that a node is not assumed to be directly connected to every other
+node in the network.
+Instead, each node is directly connected to only a subset of other nodes,
+hereafter called its `peers`.
+
+The peer-to-peer (p2p) communication layer is then the component of CometBFT that:
+
+1. establishes connections between nodes in a CometBFT network
+2. manages the communication between a node and the connected peers
-2. manages the communication between a node and the connected peers
+2. manages the communication between a node and its connected peers
-2. manages the communication between a node and the connected peers
+2. manages the communication between a node and its connected peers
+3. intermediates the exchange of messages between peers in CometBFT protocols
+
+The specification the p2p layer is a work in progress,
+tracked by [issue #19](https://github.com/cometbft/cometbft/issues/19).
+The current content is organized as follows:
+
+- [`implementation`](./implementation/README.md): documents the current state
+  of the implementation of the p2p layer, covering the main components of the
+  `p2p` package. The documentation covers, in a fairly comprehensive way,
+   the items 1. and 2. from the list above.
+- [`reactor-api`](./reactor-api/README.md): specifies the API offered by the
+  p2p layer to the protocol layer, through the `Reactor` abstraction.
+  This is a high-level specification (i.e., it should not be implementation-specific)
+  of the p2p layer API, covering item 3. from the list above.
+- [`legacy-docs`](./legacy-docs/): We keep older documentation in 
+  the `legacy-docs` directory, as overall, it contains useful information. 
+  However, part of this content is redundant,
+  being more comprehensively covered in more recent documents,
+  and some implementation details might be outdated
+  (see [issue #981](https://github.com/cometbft/cometbft/issues/981)).
+
+In addition to this content, some unfinished, work in progress, and auxiliary
+material can be found in the
+[knowledge-base](https://github.com/cometbft/knowledge-base/tree/main/p2p) repository.
@@ -0,0 +1,38 @@
+# Implementation of the p2p layer
+
+This section documents the implementation of the peer-to-peer (p2p)
+communication layer in CometBFT.
+
+The documentation was [produced](https://github.com/tendermint/tendermint/pull/9348)
+using the `v0.34.*` releases
+and the branch [`v0.34.x`](https://github.com/cometbft/cometbft/tree/v0.34.x)
+of this repository as reference.
+As there were no substancial changes in the p2p implementation, the
-As there were no substancial changes in the p2p implementation, the
+As there were no substantial changes in the p2p implementation, the
-As there were no substancial changes in the p2p implementation, the
+As there were no substantial changes in the p2p implementation, the
+documentation also applies to the releases `v0.37.*` and `v0.38.*` [^v35].
+
+[^v35]: The releases `v0.35.*` and `v0.36.*`, which included a major
+  refactoring of the p2p layer implementation, were [discontinued][v35postmorten].
-  refactoring of the p2p layer implementation, were [discontinued][v35postmorten].
+  refactoring of the p2p layer implementation, were retracted and [discontinued][v35postmorten].
-  refactoring of the p2p layer implementation, were [discontinued][v35postmorten].
+  refactoring of the p2p layer implementation, were retracted and [discontinued][v35postmorten].
+
+[v35postmorten]: https://interchain-io.medium.com/discontinuing-tendermint-v0-35-a-postmortem-on-the-new-networking-layer-3696c811dabc
+
+## Contents
+
+The documentation follows the organization of the
+[`p2p` package](https://github.com/cometbft/cometbft/tree/v0.34.x/p2p),
+which implements the following abstractions:
+
+- [Transport](./transport.md): establishes secure and authenticated
+   connections with peers;
+- [Switch](./switch.md): responsible for dialing peers and accepting
+   connections from peers, for managing established connections, and for
+   routing messages between the reactors and peers,
+   that is, between local and remote instances of the CometBFT protocols;
+- [PEX Reactor](./pex.md): due to the several roles of this component, the
+  documentation is split in several parts:
+    - [Peer Exchange protocol](./pex-protocol.md): enables nodes to exchange peer addresses, thus implementing a peer discovery service;
+    - [Address Book](./addressbook.md): stores discovered peer addresses and
+  quality metrics associated to peers with which the node has interacted;
+    - [Peer Manager](./peer_manager.md): defines when and to which peers a node
+  should dial, in order to establish outbound connections;
+- [Types](./types.md) and [Configuration](./configuration.md) provide a list of
+  existing types and configuration parameters used by the p2p package.
@@ -303,7 +303,7 @@ The `MarkBad` method marks a peer as bad and bans it for a period of time.
 This method is only invoked within the PEX reactor, with a banning time of 24
 hours, for the following reasons:
 
-- A peer misbehaves in the [PEX protocol](pex-protocol.md#misbehavior)
+- A peer misbehaves in the [PEX protocol](./pex-protocol.md#misbehavior)
 - When the `maxAttemptsToDial` limit (`16`) is reached for a peer
 - If an `ErrSwitchAuthenticationFailure` error is returned when dialing a peer
 

@@ -6,25 +6,25 @@ This document contains configurable parameters a node operator can use to tune t
 | --- | --- | ---|
 |   ListenAddress               |   "tcp://0.0.0.0:26656" |   Address to listen for incoming connections (0.0.0.0:0 means any interface, any port)  |
 |   ExternalAddress             |  ""                 |  Address to advertise to peers for them to dial |
-|   [Seeds](pex-protocol.md#seed-nodes) | empty               | Comma separated list of seed nodes to connect to (ID@host:port )|
-|   [Persistent peers](peer_manager.md#persistent-peers)            | empty               | Comma separated list of nodes to keep persistent connections to (ID@host:port )  |
+|   [Seeds](./pex-protocol.md#seed-nodes) | empty               | Comma separated list of seed nodes to connect to (ID@host:port )|
+|   [Persistent peers](./peer_manager.md#persistent-peers)            | empty               | Comma separated list of nodes to keep persistent connections to (ID@host:port )  |
 |	UPNP                        | false               | UPNP port forwarding enabled |
-|	[AddrBook](addressbook.md)                    | defaultAddrBookPath | Path do address book |
+|	[AddrBook](./addressbook.md)                    | defaultAddrBookPath | Path do address book |
 |	AddrBookStrict              | true                | Set true for strict address routability rules and false for private or local networks |
-|	[MaxNumInboundPeers](switch.md#accepting-peers)          |  40 | Maximum number of inbound peers |
-|	[MaxNumOutboundPeers](peer_manager.md#ensure-peers)         |  10 |  Maximum number of outbound peers to connect to, excluding persistent peers |
-|   [UnconditionalPeers](switch.md#accepting-peers)          | empty                | These are IDs of the peers which are allowed to be (re)connected as both inbound or outbound regardless of whether the node reached `max_num_inbound_peers` or `max_num_outbound_peers` or not. |
+|	[MaxNumInboundPeers](./switch.md#accepting-peers)          |  40 | Maximum number of inbound peers |
+|	[MaxNumOutboundPeers](./peer_manager.md#ensure-peers)         |  10 |  Maximum number of outbound peers to connect to, excluding persistent peers |
+|   [UnconditionalPeers](./switch.md#accepting-peers)          | empty                | These are IDs of the peers which are allowed to be (re)connected as both inbound or outbound regardless of whether the node reached `max_num_inbound_peers` or `max_num_outbound_peers` or not. |
 |   PersistentPeersMaxDialPeriod| 0 * time.Second      | Maximum pause when redialing a persistent peer (if zero, exponential backoff is used)    |
 |	FlushThrottleTimeout        |100 * time.Millisecond| Time to wait before flushing messages out on the connection |
 |	MaxPacketMsgPayloadSize     |  1024 | Maximum size of a message packet payload, in bytes |
 |	SendRate                    | 5120000 (5 mB/s) | Rate at which packets can be sent, in bytes/second  |
 |	RecvRate                    | 5120000 (5 mB/s) | Rate at which packets can be received, in bytes/second|
-|	[PexReactor](pex.md)                  |  true            | Set true to enable the peer-exchange reactor |
+|	[PexReactor](./pex.md)                  |  true            | Set true to enable the peer-exchange reactor |
 |	SeedMode                    |       false      | Seed mode, in which node constantly crawls the network and looks for. Does not work if the peer-exchange reactor is disabled.  |
 |   PrivatePeerIDs              | empty            | Comma separated list of peer IDsthat we do not add to the address book or gossip to other peers. They stay private to us. |
 |	AllowDuplicateIP            | false            | Toggle to disable guard against peers connecting from the same ip.|
-|	[HandshakeTimeout](transport.md#connection-upgrade)            | 20 * time.Second | Timeout for handshake completion between peers |
-|	[DialTimeout](switch.md#dialing-peers)                 |  3 * time.Second | Timeout for dialing a peer |
+|	[HandshakeTimeout](./transport.md#connection-upgrade)            | 20 * time.Second | Timeout for handshake completion between peers |
+|	[DialTimeout](./switch.md#dialing-peers)                 |  3 * time.Second | Timeout for dialing a peer |
 
 
 These parameters can be set using the `$CMTHOME/config/config.toml` file. A subset of them can also be changed via command line using the following command line flags:

@@ -124,19 +124,19 @@ This is not done in the p2p package, but it is part of the procedure to set up a
 
 The picture below is a first attempt of illustrating the life cycle of an outbound peer:
 
-<img src="img/p2p_state.png" width="50%" title="Outgoing peers lifecycle">
+<img src="../images/p2p_state.png" width="50%" title="Outgoing peers lifecycle">
 
 A peer can be in the following states:
 
 - Candidate peers: peer addresses stored in the address boook, that can be
   retrieved via the [`PickAddress`](./addressbook.md#pick-address) method
-- [Dialing](switch.md#dialing-peers): peer addresses that are currently being
+- [Dialing](./switch.md#dialing-peers): peer addresses that are currently being
   dialed. This state exists to ensure that a single dialing routine exist per peer.
-- [Reconnecting](switch.md#reconnect-to-peer): persistent peers to which a node
+- [Reconnecting](./switch.md#reconnect-to-peer): persistent peers to which a node
   is currently reconnecting, as a previous connection attempt has failed.
 - Connected peers: peers that a node has successfully dialed, added as outbound peers.
-- [Bad peers](addressbook.md#bad-peers): peers marked as bad in the address
-  book due to exhibited [misbehavior](pex-protocol.md#misbehavior).
+- [Bad peers](./addressbook.md#bad-peers): peers marked as bad in the address
+  book due to exhibited [misbehavior](./pex-protocol.md#misbehavior).
   Peers can be reinstated after being marked as bad.
 
 ## Pending of documentation

@@ -78,7 +78,7 @@ Sending a PEX response to a peer that has not requested peer addresses
 is also considered a misbehavior.
 So, if a PEX response is received from a peer that is not registered in
 the `requestsSent` set, a `ErrUnsolicitedList` error is produced.
-This leads the peer to be disconnected and [marked as a bad peer](addressbook.md#bad-peers).
+This leads the peer to be disconnected and [marked as a bad peer](./addressbook.md#bad-peers).
 
 ## Providing Addresses
 
@@ -102,7 +102,7 @@ The `receiveRequest` method is responsible for verifying this condition.
 The node keeps a `lastReceivedRequests` map with the time of the last PEX
 request received from every peer.
 If the interval between successive requests is less than the minimum accepted
-one, the peer is disconnected and [marked as a bad peer](addressbook.md#bad-peers).
+one, the peer is disconnected and [marked as a bad peer](./addressbook.md#bad-peers).
 An exception is made for the first two PEX requests received from a peer.
 
 > The probably reason is that, when a new peer is added, the two conditions for
@@ -150,7 +150,7 @@ peers, the seed node sends a PEX request.
 
 Dialing a selected peer address can fail for multiple reasons.
 The seed node might have attempted to dial the peer too many times.
-In this case, the peer address is marked as [bad in the address book](addressbook.md#bad-peers).
+In this case, the peer address is marked as [bad in the address book](./addressbook.md#bad-peers).
 The seed node might have attempted to dial the peer recently, without success,
 and the exponential `backoffDuration` has not yet passed.
 Or the current connection attempt might fail, which is registered in the address book.

@@ -50,11 +50,11 @@ The `DialPeersAsync` method receives a list of peer addresses (strings)
 and dials all of them in parallel.
 It is invoked in two situations:
 
-- In the [setup](https://github.com/cometbft/cometbft/blob/29c5a062d23aaef653f11195db55c45cd9e02715/node/node.go#L985) of a node, to establish connections with every configured
-  persistent peer
+- In the [setup](https://github.com/cometbft/cometbft/blob/v0.34.x/node/node.go#L987)
+of a node, to establish connections with every configured persistent peer
 - In the RPC package, to implement two unsafe RPC commands, not used in production:
-  [`DialSeeds`](https://github.com/cometbft/cometbft/blob/29c5a062d23aaef653f11195db55c45cd9e02715/rpc/core/net.go#L47) and
-  [`DialPeers`](https://github.com/cometbft/cometbft/blob/29c5a062d23aaef653f11195db55c45cd9e02715/rpc/core/net.go#L87)
+  [`DialSeeds`](https://github.com/cometbft/cometbft/blob/v0.34.x/rpc/core/net.go#L47) and
+  [`DialPeers`](https://github.com/cometbft/cometbft/blob/v0.34.x/rpc/core/net.go#L87)
 
 The received list of peer addresses to dial is parsed into `NetAddress` instances.
 In case of parsing errors, the method returns. An exception is made for