Extended ckb-indexer

Background

The current ckb-indexer only accepts one single scirpt to index, but with the addition of ACPL (anyone-can-pay lock) and SUDT (simple user defined token), new features need to be realized in ckb-indexer to accommodate more scenarios.

1. ex_get_cells

Expand filters: two scripts(lock script and type script), data, block range and cell capacity sorting.

Parameters

search_lock_key:  // optional 
        script -  Script 
        args_len - maximal prefix search args len, optional
search_type_key:  // optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   // optional
order: enum, asc | desc | size_asc | size_desc
range: [fromBlock, toBlock]    // optional e.g.[0x253b40, 0x253f28] ["0x" + 2440000n.toString(16), "0x" + 2441000n.toString(16)]
needed_capacity: needed capacity, optional
limit: result size limit
after_cursor: pagination parameter, optional

Returns

objects - live cells
last_cursor - pagination parameter

Explanation

The difference between empty field and the field filled with null:

*Parameters where the field is not filled means no filtering, when the value of the field in parameter is: Script means the part of the data has to satisfy the Script format, null means the part of the data is empty, occupied means the part of the data is non-empty.

In addition, script args should also be filtered: specify args | null (==all), when script args is null, the search returns cells that meet the specified code_hash condition. *Refer to ckb-lumos-indexer: if script_args fills “”, args_len = N, prefix search will satisfy this function.

needed_capacity: Based on the capacity of CKBytes requested, return the cells that meet the demand.

Example: When a user needs to transfer 1000 CKB, when requesting cells, instead of returning all cells, just return the cells that meet the 1000 CKB transfer requirement according to the order policy.

  • For cells with lock script and without type script and without data (i.e., the available CKB in an address). Provide the needed_capacity parameter to return cells based on the required amount of capacity.

Order:

  • asc | desc: sort by the formation time of a cell
  • size_asc | size_desc:sort by the capacity of a cell

Block search range:

  • [fromBlock, toBlock]

Refer to ckb-lumos-indexer: [0x253b40, 0x253f28] is [“0x” + 2440000n.toString(16), “0x” + 2441000n.toString(16)]

2.ex_get_transactions

Returns the transactions by the lock script, type script and data

Parameters

search_lock_key:  // optional 
		script -  Script 
		args_len - maximal prefix search args len, optional
search_type_key:  // optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   // optional
order: enum, asc | desc
range: [fromBlock, toBlock]    // optional e.g.[0x253b40, 0x253f28] ["0x" + 2440000n.toString(16), "0x" + 2441000n.toString(16)]
limit: result size limit
after_cursor: pagination parameter, optional

Returns

objects - transactions
last_cursor - pagination parameter

3.ex_get_cells_capacity

Returns the live cells‘ capacity by the lock script, type script and data

Parameters

search_lock_key:  //optional 
		script -  Script 
		args_len - maximal prefix search args len, optional
search_type_key:  //optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   //optional

Returns

capacity - total capacity
block_hash - indexed tip block hash
block_number - indexed tip block number

4.get_sudt_cells_balance

Returns the live sudt cells‘ sudt balance and capacity by the lock script and type script.

Parameters

search_lock_key: //optional
		script -  Script
		args_len - maximal prefix search args len, optional
search_type_key:
		script -  Script
				args: <SUDT args>
				code_hash: <SUDT code_hash>
				hash_type: type
		args_len - maximal prefix search args len, optional

Returns

balance - total balance
capacity - total capacity
block_hash - indexed tip block hash
block_number - indexed tip block number

Description of some practical examples

Get the available CKB balance of an address

  • For cells that with lock script and without type script and without data (i.e., the available CKB in an address), returns the capacity of available CKBytes in that address

ex_get_cells_capacity (search_lock_key : Scirpt, search_type_key : null, search_data_key : null)

Get the capacity of CKB occupied in an address (without type script)

  • For cells that with lock script and without type script and with data (i.e., unoccupied CKB in an address), returns the capacity of CKBytes that are currently occupied (without type script) in that address.

ex_get_cells_capacity (search_lock_key : Scirpt, search_type_key : null, search_data_key: occupied)

Get the capacity of CKB occupied in an address (with type script)

  • For cells that with lock script and with type script (i.e., CKBs with type script in an address), returns the capacity of CKBytes that are currently occupied (with type script) in that address.

ex_get_cells_capacity (search_lock_key : Scirpt, search_type_key : script)

Get the balance of a SUDT in an address and the capacity of occupied CKB (also suitable for Nervos DAO).

  • For cells that with lock script and with type script and code_hash of type script is the SUDT contract and args is one kind of SUDT (i.e. a SUDT cell in an address), provide the balance function to return the balance of the SUDT in that address; provide the capacity function to return the capacity of CKBytes occupied by the SUDT in that address.

get_sudt_cells_balance (search_lock_key : Scirpt, search_type_key : sudt_ID_script)

Get the available amount of a SUDT and the total capacity of CKB occupied (also suitable for Nervos DAO).

  • For cells that without lock script and with type script and code_hash of type script is the SUDT contract and args is one kind of SUDT (i.e., one kind of SUDT), provide the balance function to return the available amount of the SUDT; provide the capacity function to return the total capacity of CKBytes occupied by the SUDT.

get_sudt_cells_balance (search_type_key : sudt_ID_script)

3 Likes

背景

当前的 ckb-indexer 查询只接受单一 scirpt 查询,而随着 ACPL (anyone-can-pay lock) 和 SUDT (simple user defined token) 的加入,为了适应更多的应用场景,ckb-indexer 需要加入新的功能。

1. ex_get_cells

拓展筛选条件:双 scripts (lock script and type script)、data、区块范围筛选条件及 cell 的 capacity 大小排序检索。

Parameters

search_lock_key:  // optional 
		script -  Script 
		args_len - maximal prefix search args len, optional
search_type_key:  // optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   // optional
order: enum, asc | desc | size_asc | size_desc
range: [fromBlock, toBlock]    // optional e.g.[0x253b40, 0x253f28] ["0x" + 2440000n.toString(16), "0x" + 2441000n.toString(16)]
needed_capacity: needed capacity, optional
limit: result size limit
after_cursor: pagination parameter, optional

Returns

objects - live cells
last_cursor - pagination parameter

部分参数说明

不填字段与字段值为 null 的区别:

*parameter 中字段不进行填写则指不进行筛选,当 parameter 中字段值为:Script 指该部分数据需满足 Script 格式,null 指该部分数据为,occupied 指该部分数据为非空

此外,script args 也应有筛选检索 :指定 args | null (==all),当 script args 为 null 时,检索返回满足指定 codehash 条件的 cells。*参考 ckb-lumos-indexer: 是否 script_args 填写 “”,args_len=N,prefix search 前缀搜索即满足此功能。

needed_capacity: 根据请求的 CKB 数量,返回满足需求的 cells:

示例:某用户需要转账 1000 CKB 时,在请求 cells 时,不需要返回全部 cells,而是需要根据某些 order 策略返回满足 1000 CKB 转账需求的 cells。

  • 对于with lock scriptwithout type scriptwithout data的 cells(即为某地址中可用的 CKB)。提供所需 capacity 参数,根据所需的 capacity 数量返回 cells

order 排序:

  • asc | desc: 按 cell 形成的时间新旧进行排序
  • size_asc | size_desc:增加 size_asc | size_desc,按 cell 的 capacity 大小进行排序

筛选 Block 查询范围:

  • [fromBlock, toBlock]

参考 ckb-lumos-indexer: [0x253b40, 0x253f28] 即 [“0x” + 2440000n.toString(16), “0x” + 2441000n.toString(16)]

2.ex_get_transactions

Returns the transactions by the lock script, type script and data

Parameters

search_lock_key:  // optional 
		script -  Script 
		args_len - maximal prefix search args len, optional
search_type_key:  // optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   // optional
order: enum, asc | desc
range: [fromBlock, toBlock]    // optional e.g.[0x253b40, 0x253f28] ["0x" + 2440000n.toString(16), "0x" + 2441000n.toString(16)]
limit: result size limit
after_cursor: pagination parameter, optional

Returns

objects - transactions
last_cursor - pagination parameter

3.ex_get_cells_capacity

Returns the live cells‘ capacity by the lock script, type script and data

Parameters

search_lock_key:  //optional 
		script -  Script 
		args_len - maximal prefix search args len, optional
search_type_key:  //optional
		script -  Script | null
		args_len - maximal prefix search args len, optional
search_data_key: occupied | null   //optional

Returns

capacity - total capacity
block_hash - indexed tip block hash
block_number - indexed tip block number

4.get_sudt_cells_balance

Returns the live sudt cells‘ sudt balance and capacity by the lock script and type script.

Parameters

search_lock_key: //optional
		script -  Script
		args_len - maximal prefix search args len, optional
search_type_key:
		script -  Script
				args: <SUDT args>
				code_hash: <SUDT code_hash>
				hash_type: type
		args_len - maximal prefix search args len, optional

Returns

balance - total balance
capacity - total capacity
block_hash - indexed tip block hash
block_number - indexed tip block number

一些实际用例说明

查询某地址中可用 CKB 余额

  • 对于with lock scriptwithout type scriptwithout data的 cells(即为某地址中可用的 CKB),返回该地址的可用 CKB 余额

ex_get_cells_capacity(search_lock_key : Scirpt, search_type_key : null, search_data_key: null)

查询某地址中已占用 CKB 余额(无 type script)

  • 对于with lock scriptwithout type scriptwith data的 cells(即为某地址中无 type 占用的 CKB),返回该地址的正在被占用的 CKB 余额(无 type script)

ex_get_cells_capacity (search_lock_key : Scirpt, search_type_key : null, search_data_key: occupied)

查询某地址中已占用 CKB 余额(有 type script)

  • 对于with lock scriptwith type script的 cells(即为某地址中有 type 占用的 CKB),返回该地址的正在被占用的 CKB 余额(有 type)

ex_get_cells_capacity (search_lock_key : Scirpt, search_type_key : script)

查询某地址中某种 SUDT 的余额,和占用 CKB 的数量(适用于 Nervos DAO)

  • 对于with lock scriptwith type scripttype script 的 code_hash 为 SUDT 合约,args 为某 SUDT的 cells(即为某地址的 SUDT cell),提供 balance 函数,返回该地址的某种 SUDT 的余额;提供 capacity 函数,返回该地址的某种 SUDT 占用的 CKB 数量

get_sudt_cells_balance (search_lock_key : Scirpt, search_type_key : sudt_ID_script)

查询某种 SUDT 的可流通量,和占用 CKB 的总量(适用于 Nervos DAO)

  • 对于without lock scriptwith type scripttype script 的 code_hash 为 SUDT 合约,args 为某 SUDT的 cells(即为某种 SUDT cell),提供 balance 函数,返回该 SUDT 的可流通量;提供 capacity 函数,返回某种 SUDT 占用的 CKB 总数量

get_sudt_cells_balance (search_type_key : sudt_ID_script)

2 Likes

现有的数据结构下,ex_get_transactions 是没有办法根据 output data 是否被占用来查询的,需要这个查询的实际需求是对应到什么样的业务场景?

嗯,确实,在目前的业务场景中,只有 simplest dapp(直接往 data 中写数据)涉及到了 output data 是否被占用来查询的情况。而这种交易应该是属于比较特殊的情况,现阶段可以由开发者自行过滤,可以暂不需要加入到 ex_get_transactions 中。

看起来未来 NFT 也会遇到一样的问题

用不同的参数设计来尝试满足你提到的这些需求,请 review 这个 PR

3 Likes

get_cellsget_cells_capacity 已经可以满足设计需求(膜拜膜拜)。在设计文案中我们增加了一个按照 cell capacity 大小排序的新需求,但讨论后,目前的按照 cell 形成的新旧时间排序结合 output_capacity_range 已经可以满足实际使用需求,暂无增加必要。

get_transactions 部分可能需要再增加 lock / type 的相关筛选。

尝试过用给 get_transactions 部分添加 lock/type 筛选,因为采用的是 N + 1 查询方式,性能很差,我想了解一下这个过滤对应的实际场景是什么,我再看看有没有其他的实现方式可以满足这个需求

嗯嗯。比如最常见的,钱包这边需要显示某个地址下某种代币的历史交易,就需要添加 lock 和 type 两个参数进行筛选。

PR 已经更新,添加了 get_transactions rpc filter option,麻烦你测试看看各种场景,目前可能在 lock 和 type 两个参数对应的数据量都很大的情况下,还是会有一些性能问题,但是大部分场景应该是足够好的

2 Likes

:+1:
最近在准备pw版本更新,过了这段时间我们试一下

我看了一下,测试需要我们自己编译一份 multi-search-key 分支的二进制是吗

对的,目前还没有合并,需要你们用这个分支自己编译

1 Like