1 Synopsis
The CKB Open Transaction (OTX) allows users to create transactions that are open to change. Users send these open transactions to a network, where many autonomous agents provide services for assembling CKB transactions from open transactions.
This report will recap the accomplishments and insights we gained in designing a OTX proposal and building a prototype. The purpose of this prototype is to test a possible design, understand its pros and cons, find unexpected obstacles and potential improvements to the core protocol. Most of the OTX prototyping work is done by Ethan Yuan and myself. Credits go to Jiandong for helping us design and implement the first version of OTX Lock based on the Instructions List.
The OTX Prototype project can be divided into four areas, each of which will be discussed in a separate chapter later in the text. Please note that all the work here is for prototype purpose, the OTX protocol could be very different from this prototype in future iterations. Here is a summary of these areas:
- OTX Format: An extensible transaction format used to describe a CKB transaction along with its attached metadata.
- OTX Streaming Pattern: A pattern that involves placing open transactions in a stream and dividing the process of constructing transactions into smaller, reusable components.
- OTX Lock: A partial signing lock script that excludes certain transaction properties from the signature, allowing the signature to remain valid even if those properties are altered.
- OTX Framework: A framework for designing and developing dApps utilizing OTX Lock, OTX Streaming Pattern, and OTX format.
2 OTX Format
Constructing a CKB transaction can be complex. Even a basic CKB transfer transaction requires gathering live cells, forming an output for the recipient, allocating sufficient fees, and creating the change output for the remaining CKB.
To address the issue of transaction construction, several teams have formulated step-by-step approaches and developed proprietary formats to store intermediate results. For instance, ckb-cli incorporates additional fields for signing informations and stores them into a JSON file within the tx sub-command. Similarly, Lumos offers the TransactionSkeleton interface to construct transactions. However, the use of proprietary formats prevents collaboration across different tools. For example, itās impossible to construct a transaction in Lumos first, then sign it using ckb-cli.
To address this, the OTX Format has been introduced as a transaction format specification standard that promotes collaboration and components reuse, enabling seamless movement of transactions between components, processes, and machines across the network. The proposed specification, submitted as PR#406, is available for review in the nervosnetwork/rfcs repository.
The OTX Format stores metadata as key-value pairs, with keys as 32-bit integers and values as opaque data types. The key determines the encoding of the corresponding value. Applications serialize and deserialize values of interest, while treating others as raw byte arrays. This modular design decouples dependencies and enables the creation of simple, reusable, and composable components.
To avoid key conflicts, the community should collectively determine the allocation of keys via the RFC process, where the implication and usage of the keys are defined, specified, and reserved in the RFC repository. The RFC repository serves as a registry for sharing information among transaction construction applications. Proprietary keys can be used by applications without community consensus if they are not transmitted over the public network. A rich library of shared keys is essential for facilitating successful collaboration among applications using the OTX Format.
A typical pattern to use the keys is to store commands that instruct a service to construct transactions. For instance, a service may have a key named TRANSFER_CKB. The code in Listing 1 below shows an example of the value. From the code snippet, it is easy to derive that the user āalice@ckbā intends to transfer 1,000 CKB to ābob@ckbā, where the fee rate must be within the range of 0.0001 to 0.00011. As depicted in the example, the sender and recipient addresses use an email-like format, and the service can lookup the corresponding CKB lock script in the name registries. Clients who wish to transfer CKB can initiate an open transaction by setting TRANSFER_CKB without needing to know the underlying details. The service will use the TRANSFER_CKB key to generate a valid CKB transfer transaction, much like calling a method transferCKB. This pattern has evolved into the OTX Streaming Pattern, which is discussed in Chapter 3.
Listing 1: Example value of the key TRANSFER_CKB
{
"from": "alice@ckb",
"to": "bob@ckb",
"amount": "1000",
"minFeeRate": "0.0001",
"maxFeeRate": "0.00011",
}
3 OTX Streaming Pattern
The Streaming Pattern is an architecture that enables software to react and operate as events occur. This pattern allows for software components to work together in a real-time, decoupled, and scalable fashion. It is well suited for the development of modern real-time distributed systems, such as dApps. Confluent offers an excellent introduction to the Streaming Pattern; and ReactiveX is an example of the Streaming Pattern framework that supports numerous languages.
The OTX Streaming Pattern uses the OTX Format as the event payload. Autonomous OTX Agents subscribe to the OTX Stream to receive notifications of new open transactions and then process them selectively based on their criteria. The agents also emit modified or new transactions, which are merged into the OTX Stream.
Figure 1: OTX Streaming Pattern
As demonstrated in Figure 1, one of the component is optional. For instance, an agent that receives transactions from RPC will only require an OTX Source, while an agent that sends finished open transactions to a CKB node only needs an OTX Processor.
To further illustrate the pattern, here are some examples of agents.
- RPC Emitter receives transactions from an RPC endpoint and forwards them to the OTX Stream.
- CKB Sender sends completed transactions to the CKB RPC endpoint.
- Signer identifies signing requests in the stream via a registered OTX Format key and stores them locally. A wallet application retrieves the pending signing requests from the Signer agent for users. These requests are presented to users as a list in the applicationās UI. After reviewing the requests, users authorize the wallet application to sign the transactions using their private keys. The application then submits the signed transactions to the RPC Emitter agent. Once the Signer detects the presence of the signed transactions in the stream, it removes the corresponding local unsigned versions.
- Atomic Swap Matcher functions as an order book for the atomic swap proposals. It indexes the swap proposals locally and offers an RPC to search for these proposals. Clients have the option to search their own proposals to check their status or search proposals made by others to take the orders. The agent also tries to merge matched proposals and emit the merged open transaction to the OTX Stream. Therefore, the Atomic Swap Matcher acts the roles of both OTX Processor and OTX Source.
4 OTX Lock
CKB lock scripts typically require users to sign the entire transaction, which involves two steps to complete an open transaction. First, users need to send the initial open transaction. Then, they must wait for its completion before signing it. In case of failure, users have to wait for a new completed transaction to sign again.
The OTX Lock offers partial signing mechanisms that protect specific properties of a transaction while leaving the other parts free to change. The signature remains valid as long as the signed properties remain unchanged. This enables users to pre-sign open transactions using OTX Lock, facilitating a fire-and-forget style similar to typical CKB transaction processes.
In the following sections, we will discuss two different ways to design the Lock: the Instruction List Lock and the Sighash Lock. Furthermore, Chapter 6 will present several new directions for further exploration.
4.1 Instructions List Lock
The Instructions List Lock enables users to specify the content for inclusion in the digest message for signing through an instructions list. Each instruction adds transaction properties to the digest, such as a full input cell or the number of output cells. We previously presented a design proposal in the RFC: Composable Open Transaction Lock Script.
We have tried to use the lock in different scenarios; two of the most studied ones are Atomic Swap and Unilateral Payment.
- In Atomic Swap, users send swap proposals as open transactions. These proposals describe the assets and quantities users wish to obtain, as well as the assets and quantities they want to pay. Each proposal is unbalanced, and it is the responsibility of the Atomic Swap OTX Agent to merge the matched proposals into a balanced CKB transaction.
- In Unilateral Payment, we want to design a mechanism based on open transactions to pay assets without interactions from payees. The requirement is from the scenario that users pay small amounts of CKB or user-defined tokens (UDT) but donāt want to incur the storage costs of creating a cell for the payee.
The design of the transaction structures and the instruction lists can be found in an older revision of the repository EthanYuan/open-transaction-pool.
During the experiments to design those scenarios, weāve experienced significant drawbacks of the proposal. Itās challenging to create a secure instructions list, and the field-by-field inclusion algorithm is not powerful enough for most complex dApps.
The security issues originate from the uncertainty of uncovered transaction fields, making it difficult to predict how open transaction will be consumed. In subsequent paragraphs, we will recapitulate two categories of security issues.
The first category is the Replay Attack, which occurs when an attacker reuses the signature created from a specific instruction list to gain access to other assets owned by the same user. This can occur when the instructions list does not include a field unique to the current transaction. To prevent this, users must avoid signing any unknown digest messages. Additionally, instructions lists must include at least one input cell out point to ensure uniqueness.
The second category of security issues arises from the absence of support for Cell Grouping. One of the key features of OTX is the ability to merge open transactions. However, when multiple transactions are merged, it becomes impossible to reference cells based on their absolute locations. To address this, the proposal introduces the method of referencing a cell using the relative index, which is a number relative to a base value. While anyone is allowed to modify the base values, it is typically the responsibility of the Agent who merges open transactions to set the appropriate values without compromising the existing signatures. See how Listing 3 sets the base values for the Open Transaction 2 after merging two atomic swap proposals.
Listing 3: Atomic Swap Merging Example
Open Transaction 1:
# Alice wants to get 20 SUDT X by paying 200 CKB
inputs:
0: CKB 200 owned by Alice
outputs:
0: SUDT X 20 owned by Alice
inputWitnesses:
0:
# Initially, both base values are set to 0
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 2:
# Bob wants to get 200 CKB by paying 20 SUDT X
inputs:
0: SUDT X 20 owned by Bob
outputs:
0: CKB 200 owned by Bob
inputWitnesses:
0:
# Initially, both base values are set to 0
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 1 + 2:
# Make proposals from Alice and Bob
inputs:
0: CKB 200 owned by Alice
1: SUDT X 20 owned by Bob
outputs:
0: SUDT X 20 owned by Alice
1: CKB 200 owned by Bob
inputWitnesses:
0:
# Transaction 1 comes first, so the base values are zeros.
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
1:
# Shift the base values by one to reference the correct cells.
inputBase: 1
outputBase: 1
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Allowing anyone to set the base values is dangerous, as adversaries can exploit this by reusing a cell in different open transactions. This is possible because output cells in the CKB transaction lack unique identifications. Letās revise the swap example by splitting Bobās proposal into two identical proposals as in Listing 4. By reusing the last output, Bob only receives 100 CKB instead of the intended 200 CKB.
Listing 4: Cell Output Reusing
Open Transaction 1:
# Alice wants to get 20 SUDT X by paying 200 CKB
inputs:
0: CKB 200 owned by Alice
outputs:
0: SUDT X 20 owned by Alice
inputWitnesses:
0:
inputBase: 0
outputBase: 0
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 2:
# Bob wants to get 100 CKB by paying 10 SUDT X
inputs:
0: SUDT X 10 owned by Bob
outputs:
0: CKB 100 owned by Bob
inputWitnesses:
0:
inputBase: 0
outputBase: 0
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 3:
inputs:
0: SUDT X 10 owned by Bob
outputs:
# This output is identical to the one in Transaction 2
0: CKB 100 owned by Bob
inputWitnesses:
0:
inputBase: 0
outputBase: 0
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 1 + 2 + 3:
# Make proposals from Alice and Bob
inputs:
0: CKB 200 owned by Alice
1: SUDT X 10 owned by Bob
2: SUDT X 10 owned by Bob
outputs:
0: SUDT X 20 owned by Alice
1: CKB 100 owned by Bob
# The adversary removes the output from Open Transaction 3
inputWitnesses:
0:
inputBase: 0
outputBase: 0
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
1:
inputBase: 1
outputBase: 1
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
2:
inputBase: 2
# The adversary sets this to 1 to reuse the output in Transaction 2
outputBase: 1
instructions:
- "inputs[inputBase + 0]"
- "outputs[outputBase + 0]"
signature: "0x..."
In conclusion, a cells-grouping mechanism that prevents tampering by adversaries is crucial for merging open transactions. Implementing the mechanism in CKB is much simpler compared to using contracts, but it requires careful design to minimize overhead and ensure compatibility.
There is a concept called script group, which is similar to cell grouping. It involves grouping cells based on scripts and arguments, and then running the script once for each group. But cells in an open transaction often do not align with a script group. Whatās even worse is that there are often overlaps between open transactions and script groups. If a lock script fails to iterate through all the cells in the group or limit the number of cells, it becomes vulnerable to an attack where adversaries can append input cells that belong to the same script group. These additionally inputs get unlocked along with existing ones without requiring a new signature from the user. This clearly violates the intentions of the user. Listing 5 depicts an OTX Lock implementation that is vulnerable, as it only verifies the first input witness in the group. Listing 6 demonstrates an attack that exploits this vulnerability. Pay attention to input 2 of the merged transaction. Specifying the number of input cells when signing can solve this issue, but would prevent merging open transactions from the same user into a single CKB transaction. This is a common scenario in merchant apps where a buyer purchases multiple items in a single transaction. Restricting the number of input cells would result in a complex design where goods belonging to the same seller cannot be combined in the same transaction.
Listing 5: A vulnerable OTX Lock which verifies only the first input witness in the group
function main() {
const { instructions, signature } = getInputWitnessInGroup(0);
const digest = generateDigest(instructions);
verifySignature(digest, getPubkeyFromArgs(), signature);
}
Listing 6: An attack that appends cells in the same script group
Open Transaction 1:
# Alice wants to get 20 SUDT X by paying 200 CKB
inputs:
0: CKB 200 owned by Alice
outputs:
0: SUDT X 20 owned by Alice
inputWitnesses:
0:
# Initially, both base values are set to 0
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 2:
# Bob wants to get 200 CKB by paying 20 SUDT X
inputs:
0: SUDT X 20 owned by Bob
outputs:
0: CKB 200 owned by Bob
inputWitnesses:
0:
# Initially, both base values are set to 0
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Open Transaction 1 + 2:
# Make proposals from Alice and Bob
inputs:
0: CKB 200 owned by Alice
1: SUDT X 20 owned by Bob
# Adversaries append the cell which is in the same script group as input 0
2: CKB 1000 owned by Alice
outputs:
0: SUDT X 20 owned by Alice
1: CKB 200 owned by Bob
inputWitnesses:
0:
# Transaction 1 comes first, so the base values are zeros.
inputBase: 0
outputBase: 0
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
1:
# Shift the base values by one to reference the correct cells.
inputBase: 1
outputBase: 1
instructions:
# The first input starting from inputBase
- "inputs[inputBase + 0]"
# The first output starting from outputBase
- "outputs[outputBase + 0]"
signature: "0x..."
Another drawback that was mentioned earlier is that the Instruction List Lock is not powerful enough. For instance, it is not possible to replicate the logic of Anyone-Can-Pay (ACP) Lock using an instruction list, because there are no commands available for performing arithmetic calculations and comparisons. We have been cautious about adding new instructions, due to the possibility of requiring endless additional instructions. The sustainable approch would be through script composition, where dApps expose verified assertions for signing. There are two threads in the CKB GitHub repository relevant to this topic.
4.2 Sighash Lock
To simplify the design of the instructions list, why not limit users to a set of established patterns for signing transactions? This is where the concept of OTX Sighash Lock comes into play.
The OTX Sighash Lock design is modeled on the Bitcoin Sighash pattern. If you want to learn more about the Bitcoin Sighash, check out the tutorial from saylor.org: CS120: Bitcoin for Developers I, Elliptic Curve Signatures. For those interested in implementing OTX Sighash Lock, refer to the following Github repository: EthanYuan/otx-sighash-lock.
Although the Sighash pattern provides only six options, it does not simplify the issue. Rather, the complexity is shifted elsewhere. Sighash pattern requires an elaborate transaction layout design, as seen in the Atomic Swap demo, where users need to prepare a dedicated cell for the open transaction instead of using existing ones. For a reference, see the atomic swap documentation. By contrast, the older Instructions List Lock design gave users more freedom to choose from existing cells.
5 OTX Framework
OTX Framework combines OTX Format, OTX Streaming Pattern, and OTX Lock together to provide an easy-to-use instance for developers.
The central component of the framework is the implementation of the OTX Streaming Pattern, referred to as the open transaction pool. A Rust Proof of Concept (PoC) is available on GitHub at EthanYuan/open-transaction-pool.
Using the open transaction pool as a foundation, we can develop various agents, such as Atomic Swap and Signer. In our vision, there will be a public marketplace where agents can be shared. Developers can easily incorporate agents from this marketplace and construct complex transactions through agent composition. Certain agents, such as the one designed to collect live cells based on the specific criterion, can significantly facilitate the transaction construction process.
The SDK acts as the interface of the framework for developers. We already have code snippets to work with Open Transactions in Rust and Javascript, but these are far from an SDK for the framework.
6 Future Works
6.1 New Directions of OTX Lock
Both the Instruction List Lock and Sighash Lock require a redesign of the Cell Grouping mechanism. In the short term, it is necessary to integrate a solution into the OTX Lock. In the long term, further research is needed to determine how to support cell grouping in the CKB transaction structure.
There are also other mechanisms of partial signing, such as signing the user intent rather than specific fields. An intent is a message indicating the operation that user wants to perform, such as the example in Listing 7.
Listing 7: An intent to swap 10 CKB with 50 SUDT
{
"app": "0x...",
"nonce": 1,
"command": "swap",
"from": {
"assets": "ckb",
"balance": "10",
},
"to": {
"assets": "sudt",
"id": "0x...",
"balance": "50",
}
}
The dApp checks the intent has been successfully carried out. The transaction properties that do not affect the intent execution are free to change. For example, the intent above does not care which inputs have been collected to provide the 10 CKB balance; it only need to check whether the userās CKB balance has decreased by exactly 10 CKB.
The dApp verifies successful execution of intent. The transaction properties that do not affect execution are free to alter. For instance, the specific inputs used to attain the 10 CKB balance do not matter - only the userās CKB balance being reduced by precisely 10 CKB matters.
To prevent repay attacks, itās crucial to implement a mechanism to make the intent unique, such as the nonce field in the example.
Intent functions as the instruction for constructing a transaction as well. This means that dApps can use the same logic for both constructing and verifying transactions. The verification code rebuilds the transaction using the intent and checks that it matches the target transaction being verified.
6.2 Others
- A public P2P network to exchange open transactions is crucial for the adoption in dApps. We could incorporate an opt-in protocol in CKB to relay open transactions.
- Create a uniform interface for transaction construction across SDKs, for both CKB and open transactions. Developers can experience a seamless transition with turn to the open transactions solution.
- Improve tools for better developer experience.
- We already have code snippets to work with Open Transactions in Rust and Javascript. We could improve the JavaScript SDK further. Besides working with the OTX Format, the SDK must hide the complexity of the OTX Lock and support a mechanism to seamlessly integrate the features provided by agents.
- We have a Rust PoC for the Streaming Pattern available on GitHub at EthanYuan/open-transaction-pool. We could implement the Streaming Pattern and all agents in JavaScript.