It has been five years since the launch of CKB mainnet. Our ecosystem has since grown much stronger. To further facilitate the dApp development on CKB platform, we have to make CKB foundation even more robust and more versatile. One critical component is CKB’s network stack, tentacle, whose development has been ossified for a while.
Below we review some blockchain p2p network fundamentals and the history of tentacle development, and then list some use cases that is not viable with current tentacle architecture. More generally, we ask a simple question of how to make CKB’s network stack more adaptive to future updates. Although this question can not be answered completely in this article, we will propose some necessary enhancements to CKB’s network stack to make it more adaptive to future updates. These proposed enhancements are by no means final. We’d love to hear more from the community about what p2p functionalities CKB currently lacks are needed, their use cases, and the priority of those functionalities. Please don’t hesitate to comment below or engage in discussions on Discord HaCKBee server.
A Fistful of Dollars Code: What Makes a P2P Network for Blockchain Projects?
To answer the question of what we need in order to make CKB’s network stack more adaptive to future updates, let’s first take a look at the general functionality of p2p network for any blockchain projects.
High level overview
Let’s first inspect the architecture of some famous blockchain projects.
A good overview of Bitcoin p2p network can be found in P2P Network — Bitcoin. And here is a list of supported protocols in CKB.
Ignoring all the technicalities, the main functionality of CKB and Bitcoin network is similar. Their main task is to support discovering and connecting mesh network nodes, and block propagation (both download and broadcast).These, in more technical terms, correspond to supporting node discovery, RPC-like request/response interface, and gossip broadcast. Let’s review them one by one.
Node discovery
Node discovery, also called peer discovery, is the process of discovering nodes to be connected to this p2p network. Below is some commonly used node discovery techniques.
Rendezvous point
Traditionally, nodes discovery can be easily done by making all the nodes connecting to the same rendezvous server which can be hard-coded or looked up through DNS or other protocols. As all nodes are connected to this server, we can easily ask the server for information of other nodes. This solution introduces a single point of fault, is thus not ideal in a decentralized application like Bitcoin. In Bitcoin uses bootstrapping nodes (bootnodes) to obtain an initial list of other nodes in the network. More nodes can be obtained by recursively sending addr message and connecting to nodes returned from addr message. CKB uses the same technique.
Distributed hash table (DHT)
DHT is also widely used in node discovery. In DHT, instead of a single server keeping all the network information, the information (mostly as key/value pairs) is kept at a server (actually a number of servers for resilience) close enough to the key (See Kademlia for the concrete meaning of closeness). Moreover, we can query the closest nodes we know so far and obtain a list of nodes that are even closer to the key. Repeated querying the currently closest nodes, and we can finally reach a node that has the value to the key.
MDNS local peers discovery
A good way to discover peers providing certain services is MDNS (Multicast DNS). Unfortunately, multicast protocols like MDNS only work in local network, and are thus unsuitable for wide-area Internet applications like CKB. Albeit, it is a good addition to CKB network.
Block propagation
Another important task for blockchain p2p network nodes is to disseminate newly mined blocks, while keeping itself up to date with the network by requesting blocks from other nodes.
Technically, the block downloading process is different from the broadcasting process. The block download is like traditional client/server request/response, where the client asks the server to transmit a list of blocks within some height interval, while the block broadcast is normally accomplished by implementing a certain Gossip protocol, which spreads blocks to a broadcasting group (normally the whole network).
Block download
The following messages play a crucial role in Bitcoin block download:
- getdata: Requests a single block or transaction by hash.
- getblocks: Requests an inv of all blocks in a given range.
- getheaders: Requests a headers message containing all block headers in a given range.
When these message are sent to Bitcoin peers, the peers will return a corresponding response.
Following Bitcoin, CKB defined a few protocols, which are essentially block/header request/response in Bitcoin.
Block/transaction broadcast
Block broadcast is a little bit more complicated as it involves the whole network. CKB leverages the underlying network library Tentacle to broadcast messages, while Bitcoin has its own implementation of the gossip protocol.
Supporting protocols
To fulfill those high level requirements, we may need to develop several supporting protocols.
Identify
Not all the instances running the same blockchain node program are identical in the behavior and functionality. We may need to first check a few things, for example, are we in the same network with the same genesis block? Is the peer a full node or just light one? Do we need to broadcast all new blocks to this peer?
We need to identify the connected peers’ capabilities and disconnect them if they are incompatible.
CKB does this by registering an identify protocol in Tentacle, while Bitcoin has its [ver message](Protocol documentation - Bitcoin Wiki) to check the compatibility of its nodes.
Encryption
Another thing that is becoming increasingly important is to encrypt the whole transport to avoid eavesdropping, censorship, and tampering by the man in the middle.
Bitcoin, initially sending all its traffic in plain text, is now considering adding encrypted transport in its p2p network. See bips/bip-0324.mediawiki at master · bitcoin/bips.
As every blockchain node is expected to generate a secret key on startup, the encryption process is more or less standard. Nodes exchange public key information with Diffie–Hellman key exchange, and generate encryption/MAC keys with key derivation function and agreed output key material from Diffie-Hellman key exchange. Noise Protocol Framework is a popular choice to do such thing.
An unfortunate (and inevitable given the timing) choice of Tentacle is that we used libp2p’s now deprecated for transport encryption.
Multiplexing
Sometimes, a big enum message type is enough for blockchain p2p network (at least for Bitcoin). But this tightly-coupled message definition is not flexible enough. In CKB and many other p2p networks, there is a much higher level of abstraction called stream. For different type of messages, we may register different protocols and protocol handlers to a single connection (think http path like http://host/protocol_1 and http://host/protocol_2, we may use different handler to process requests to these two URLs, but these handlers are on the same server). This makes the processing or different kinds of messages much more ergonomic and extensible. Under the hood, tentacle uses yamux protocol to multiplex streams over a single connection, and then to dispatch streams to registered protocol handlers.
Once Upon a Time in the West CKB: A Brief History of Tentacle
Having discussed all the requirements of a blockchain p2p network, we now give a short retrospection of Tentacle, CKB’s current network stack, and then explain why it is not enough for CKB’s future development.
- Tentacle started 5 years ago as the network framework for CKB by forking some libp2p code. We initially tried to use libp2p, but encountered numerous problems, for instance, the Rust version was not mature enough, the specification of libp2p was not general enough to meet CKB’s needs (it has some golang-specific items and sometime serves only for IPFS).
- Some critical components are not updated for a while (e.g. secio), because they are initially forked from rust-libp2p, and we made some incompatible modifications.
- Given the current architecture of Tentacle, it is not entirely easy to add non-trivial features like NAT traversal through TCP/UDP hole punching techniques and webRTC support to Tentacle, which are quite important in CKB nodes’ dynamic network environment.
For a Few Money Improvements More: Making CKB’s Network Stack More Adaptive to Future Updates
Having talked about the status quo of popular blockchain projects p2p network, and the history of Tentacle, we now come to the question of what CKB’s network stack need to do support a much wider range of applications in the future. Again, the following proposed enhancements are only tentative. Please feel free to contact us through HaCKBee or Nervous Talk.
Service discovery
In the context of backend development, service discovery typically refers to the process of discovering backend nodes that support specific services. As CKB grows stronger, we also need to differentiate CKB nodes by the services they support. A simple example is that how do we find out all the nodes that support light client or lightning network.
Although determining the capabilities is supported by reading the Flags parameter in the identify process, this alone is not enough. We need to make this service discovery process usable to external hosts, allowing them to learn about CKB’s capacity without establishing any connection… This is particularly useful in cases where light clients randomly sample nodes supporting light client from the whole network.
Again, structured networks like DHT would make life a lot easier. We can publish our peer info (basically the public key and public addresses) through DHT, and anyone interested in a certain service can learn our address information from the open DHT network. With the current unstructured architecture, it would be inefficient—if possible—to create a way to discover services.
Topic based gossip protocol
As service discovery, it is sometimes useful to create a broadcast for a specific topic. An example of this is that we may only want to broadcast payment channel updates to nodes within the same payment network. Currently, the audience of broadcast in CKB is always the whole network. We need to create a more flexible pub/sub protocol.
Peer routing
Directly connecting to a specific peer (This is called peer routing in libp2p.) is highly desirable, in the use cases such as opening/closing a payment channel between two peers.
Currently, CKB cannot connect to a peer with specific peer ID, unless we know its public address in advance. This is because all the nodes are learned from sending GetNodes messages. We have no way to learn the connectable addresses of a peer from its peer ID.
There are a few ways to establish connectivity between two nodes. A crucial step is to discover the peer’s addresses. In a more traditional settings, we can make both peers connect to the same centralized server. And then let the intermediate server coordinate the connection establishment process. In a decentralized applications like CKB, using DHT to discover dialable peer addresses is more appropriate. Another way to establish connectivity is to wrap and peel packets through intermediate servers like onion routers. Although this would incur an obviously higher latency, it can also provide enhanced security.
Supporting more transport protocols
Currently Tentacle only supports TCP and WebSocket as transport protocol. We need to catch up with some of the recent innovations. Many of QUIC’s innovative features (e.g. native multiplexing, fewer round-trips) would make it a great candidate for CKB’s transport protocol. Protocols like TOR are still going strong. They can make CKB more private and more secure. All in all, it would make CKB even better if we can have more protocols plugged in.
Running in more restricted environment
In order to support a critical mass of the whole population, we need to support more constrained environments, like web browser and NAT network.
- Web browsers support: For example, we may want a browser light client directly to talk to a full node. Two things are crucial to port part of CKB to web assembly and use webRTC for connection. Currently, Tentacle supports wasm and can connect to WebSocket peers. It would be better if we can support webRTC to circumvent the https certificate requirement for websocket/webtransport as what libp2p did.
- NAT traversal: A large portion of CKB nodes have no stable public IPs. We may need to use some Hole punching techniques to establish connections between two nodes behind NAT.
Battle-tested cryptography
As mentioned above, Tentacle’s cryptography library is deprecated. We need to replace it with a bespoken, battle-tested one protocol like Noise or TLS.
P2P Networks Status Quo
| node discovery | standard cryptography | multiplexing | TOR/i2p | peer routing | nat connectivity | UPnP IGD/NAT-PMP | topic based gossip | webrtc | wasm | |
|---|---|---|---|---|---|---|---|---|---|---|
| ckb | gossip | no (secio) | yamux | no | no | no | yes | no | no | yes |
| bitcoin | gossip | no encryption 1 | no | yes | no | with tor relay 2 | yes | no | no | no |
| etheruem | dht (discv5) 3 | no (rlpx) 4 | somewhat | no | yes | no | no | maybe 5 | no | no |
| filecoin | dht (libp2p) | yes | yamux | no | no | yes | yes | yes | maybe 6 | maybe |
| trin 7 | dht (discv5) | no (as ethereum) | no | no | yes | wip 8 | no | somewhat 9 | no | no |
- bips/bip-0324.mediawiki at master · bitcoin/bips
- kind of support nat traversal with TOR, see bitcoin/doc/tor.md at master · bitcoin/bitcoin
- devp2p/discv5/discv5.md at master · ethereum/devp2p
- devp2p/rlpx.md at master · ethereum/devp2p
- there exists topic based gossip in spec, but not implemented, see devp2p/discv5/discv5-theory.md at master · ethereum/devp2p
- It is possible with libp2p. I don’t know if it is actually implemented. One libp2p consumer known actively trying to use libp2p to communicate between nodes and browser is smoldot. See Add support for WebRTC in the non-browser node · Issue #2802 · paritytech/smoldot
- ethereum/trin: An Ethereum portal client: a json-rpc server with nearly instant sync, and low CPU & storage usage is an implementation of ethereum Portal Network. Ethereum Portal Network is a lightweight network protocol stack for constraint devices to access Ethereum network. Trin has many much needed features.
- NAT traversal · Issue #596 · ethereum/trin.
- Trin has different networks for different Light Ethereum Subprotocols (LES)
