A potential contributor (Thao) I got in contact with through the foundation’s hiring pipeline had a couple inquiries, appreciate any information anyone can share pertaining to them:
“About this issue https://github.com/nervosnetwork/ckb/issues/3940, I had
some experiences with LMDB and I also checked the source code of that part.
I think the
KeyValueBackendtrait should be improved, like an LMDB
Wrapper. And that trait should be Send & Sync (because Rust automatically
derived, methodless & marker traits) to guarantee thread safety, as well as
atomic transactions to ensure database integrity.”
"As I can see, we’re still using JSON-RPC.
We could use gRPC instead of JSON-RPC for the APIs because it is better fit
for transferring binary data, faster, more compact and can reuse the
proto-described structures transparently to fit many languages (Python, Go,
Rust) without manually implementing and maintaining
serialization/deserialization for each one.
Besides that, Protobuf or “proto” is a schema-based binary
serialization/deserialization system. It is designed to be fast and compact.
gRPC & Protobuf support streaming and versioning as well, which I think
also contributed an important part in high performance.
(Continued) IMHO, for the APIs, gRPC and Protobuf seems to be the most mature ecosystem.
Actually, we can use a mix of Protobuf and custom serialization in the gRPC
endpoint. Switching to full protobuf is not mandatory and could be kept
partially to support the new chosen CKB core serialization format.
In addition, gRPC supports transcoding Protobuf to JSON via proxies
(internal or external) and transport over HTTP/1 and WebSockets via proxies
(internal or external). So here is the answer if someone is worried about
gRPC not matching frontend libs.
To summarize, using gRPC together with Protobuf will bring huge benefits in
terms of performance, as well as simplify the data ser/deser process. There
is also no need to write any additional client code because the protoc
compiler will autogenerate client libraries - including wire serialization
and deserialization code - for just about every major language. The
compiler writes thousands of lines of code so we don’t have to. Also, No
more problems streaming data from a server. gRPC supports both 1-to-1 RPCs
and 1-to-many streaming requests, no more awkward REST / WebSocket combo