Spark Program | Ckb-probe: Deep Observability Tool for CKB Nodes Based on Aya Kernel eBPF/ckb-probe：基于 Aya 内核 eBPF 的 CKB 节点深度可观测性工具

[clair]

Week 3 Report: eBPF Feasibility Validation Passed + check Subcommand Shipped

Period: 2026-03-30 ~ 2026-04-05
Author: ckb-probe Development Team
Project: ckb-probe — eBPF-Based Deep Observability Tool for CKB Full Nodes

---

1. Goals for This Week

Per the project plan, Week 3 required completing four eBPF feasibility validations and reaching Milestone 1 by end of week:

1. RocksDB uprobe latency measurement — entry/return pair attachment, measuring function execution duration
2. Multi-function uprobe simultaneous attachment — all 19 Tier 1 symbols tested individually
3. TCP kprobe — attaching to kernel tcp_sendmsg / tcp_recvmsg, capturing P2P network events
4. sys_enter tracepoint — attaching to raw_syscalls/sys_enter, profiling syscall distribution
5. Implement ckb-probe check subcommand — one-command environment detection + eBPF probe validation

---

2. Milestone 1 Completion Status

Milestone 1 (End of Week 3): Feasibility validation complete; three BPF program types verified against the CKB process. check and symbols subcommands delivered.

Deliverable	Status	Notes
uprobe/uretprobe validation		4 RocksDB function entry/return pairs attached successfully
Multi-function uprobe validation		All 19 Tier 1 symbols tested individually; 15 confirmed attachable
kprobe/kretprobe validation		tcp_sendmsg / tcp_recvmsg 4/4 attached, real-time event capture
tracepoint validation		raw_syscalls/sys_enter attached successfully, real-time syscall capture
ckb-probe check subcommand		8 environment checks + eBPF probe validation
ckb-probe symbols subcommand		Delivered in Week 2, no changes this week
eBPF kernel-side programs		Built ckb-probe-ebpf from scratch: 6 uprobe pairs + 2 kprobe pairs + 1 tracepoint
xtask build system		Dual-target eBPF compilation management

Milestone 1 achieved.

---

3. Feature Demos

3.1 ckb-probe check — One-Command Environment Detection + eBPF Validation

When invoked without --pid, only environment checks are performed (8 items). When --pid is provided, actual attachment tests for all three eBPF probe types are appended:

$ ckb-probe check --binary ./ckb --pid $(pgrep -x ckb)

╔══════════════════════════════════════════════════════════════╗
║  ckb-probe environment check                               ║
╠══════════════════════════════════════════════════════════════╣
  ✅ Kernel version            6.8.0-106-generic (need >= 5.8)
  ✅ BPF config                BPF=y SYSCALL=y JIT=y
  ✅ BTF support               /sys/kernel/btf/vmlinux exists
  ✅ Permissions               running as root
  ✅ bpf() syscall             available
  ✅ uprobe support            /sys/kernel/debug/tracing/uprobe_events exists
  ✅ CKB process               1 instance(s), pid=3127545
  ✅ CKB symbols               2/3 key symbols found (symtab)
╚══════════════════════════════════════════════════════════════╝

  Result: 8/8 checks passed
  🎉 All checks passed!

3.2 uprobe Validation — RocksDB Function Attachment

A full scan of all 19 Tier 1 trace targets confirmed that 15 are uprobe-attachable in the official CKB v0.205.0 binary:

╔══════════════════════════════════════════════════════════════╗
║  ckb-probe eBPF validation                                 ║
╠══════════════════════════════════════════════════════════════╣
  ✅ ── uprobe latency ──      entry/return pair attach test
  ✅   rocksdb_get_pinned_cf   entry + return attached
  ✅   rocksdb_put             entry + return attached
  ✅   rocksdb_write           entry + return attached
  ❌   rocksdb_delete          symbol not in binary (expected)
  ✅   rocksdb_create_iterator_cf  entry + return attached
  ❌   rocksdb_multi_get_cf    symbol not in binary (expected)
  ✅ ── uprobe Tier 1 ──       all 19 Tier 1 symbol attach test
  ✅   rocksdb_get             symbol found, uprobe-attachable
  ✅   rocksdb_get_pinned      symbol found, uprobe-attachable
  ✅   rocksdb_get_pinned_cf   symbol found, uprobe-attachable
  ✅   rocksdb_put             symbol found, uprobe-attachable
  ✅   rocksdb_put_cf          symbol found, uprobe-attachable
  ✅   rocksdb_write           symbol found, uprobe-attachable
  ❌   rocksdb_delete          not found in binary
  ❌   rocksdb_delete_cf       not found in binary
  ❌   rocksdb_multi_get_cf    not found in binary
  ✅   rocksdb_transaction_put_cf  symbol found, uprobe-attachable
  ✅   rocksdb_transaction_delete_cf  symbol found, uprobe-attachable
  ❌   rocksdb_transaction_get_cf  not found in binary
  ✅   rocksdb_transaction_commit  symbol found, uprobe-attachable
  ✅   rocksdb_optimistictransaction_begin  symbol found, uprobe-attachable
  ✅   rocksdb_create_iterator_cf  symbol found, uprobe-attachable
  ✅   rocksdb_iter_seek       symbol found, uprobe-attachable
  ✅   rocksdb_iter_seek_to_first  symbol found, uprobe-attachable
  ✅   rocksdb_iter_next       symbol found, uprobe-attachable
  ✅   rocksdb_iter_destroy    symbol found, uprobe-attachable
  ✅ uprobe summary            latency pairs: 4/6, Tier 1 symbols: 15/19
╚══════════════════════════════════════════════════════════════╝

The 4 entries are RocksDB functions not called by CKB (removed by the linker’s dead code elimination). These are expected absences and do not affect core monitoring capabilities.

3.3 Real-Time Event Capture — uprobe / kprobe / tracepoint

After successful attachment, ckb-probe check automatically collects 3 seconds of live event data. Below is actual output from a CKB v0.204.0 node (pid=3127545):

  ⏳ Collecting live events for 3 seconds...

  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=84.7μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=67.0μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=53.6μs
  [uprobe] pid=3127545 tid=3310904 func=write                    latency=44.9μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=5783.8μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=3848.8μs
  [uprobe] pid=3127545 tid=3310904 func=create_iterator_cf       latency=23.2μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=82.1μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=59.0μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=54.6μs
  [syscall] pid=3127545 tid=3322756 nr=232 (epoll_wait)
  [uprobe] pid=3127545 tid=3310899 func=get_pinned_cf            latency=3293.9μs
  [syscall] pid=3127545 tid=3310527 nr=1 (write)
  [uprobe] pid=3127545 tid=3310899 func=get_pinned_cf            latency=8292.1μs
  [uprobe] pid=3127545 tid=3310899 func=get_pinned_cf            latency=3363.4μs
  [syscall] pid=3127545 tid=3322756 nr=232 (epoll_wait)
  [uprobe] pid=3127545 tid=3310904 func=write                    latency=51.6μs
  [syscall] pid=3127545 tid=3310904 nr=1 (write)
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=6467.0μs
  [uprobe] pid=3127545 tid=3310904 func=get_pinned_cf            latency=9306.0μs
  [uprobe] pid=3127545 tid=3310904 func=create_iterator_cf       latency=25.4μs
  [uprobe] pid=3127545 tid=3310902 func=get_pinned_cf            latency=6416.9μs
  [tcp] pid=3127545 tid=3322756 dir=TX bytes=1471
  [tcp] pid=3127545 tid=3322757 dir=RX bytes=469

  📊 Captured 1873 uprobe, 2 tcp, 621 syscall events in 3s

Key observations:

• uprobe latency measurement is working — get_pinned_cf (CKB’s primary read path) latency ranges from 41.8μs to 9306.0μs; write runs at ~44–51μs; create_iterator_cf at ~23–25μs
• kprobe captured P2P network TCP send/receive events (TX 1471 bytes / RX 469 bytes)
• tracepoint captured syscall distribution (predominantly epoll_wait and write)
• 3-second capture volume: 1873 uprobe + 2 tcp + 621 syscall = 2496 total events

---

4. Validation Results Summary

#	Probe Type	Target	Attachment Result	Event Capture
1	uprobe/uretprobe	RocksDB latency measurement (entry/return pairs)	4/6 pairs	1873 events in 3s
2	uprobe symbol scan	All 19 Tier 1 targets tested individually	15/19	Attachable
3	kprobe/kretprobe	tcp_sendmsg / tcp_recvmsg	4/4	2 events in 3s
4	tracepoint	raw_syscalls/sys_enter	1/1	621 events in 3s

All three BPF program types (uprobe / kprobe / tracepoint) validated successfully.

---

5. Currently Supported Commands

$ ckb-probe --help

Commands:
  check     Check environment and validate eBPF probes
  symbols   Analyse a CKB binary for uprobe-attachable symbols

Usage	Description
ckb-probe check	Environment detection (8 items)
ckb-probe check --binary ./ckb	+ CKB symbol check
ckb-probe check --binary ./ckb --pid 1234	+ eBPF three-probe-type validation
ckb-probe check --binary ./ckb --pid 1234 --probe uprobe	Validate uprobe only
ckb-probe symbols ./ckb	Symbol analysis (Tier 1/2/3)
ckb-probe symbols ./ckb --json	JSON format output
ckb-probe symbols ./ckb --tier 1 --filter get	Filter by tier and keyword

---

6. Next Week’s Plan

Entering Phase 2: RocksDB Core Probe Development.

Per the project plan, Week 4 will build on the 15 Tier 1 symbols validated this week to implement full RocksDB storage layer deep tracing:

1. uprobe/uretprobe for five core operations — get_pinned_cf (primary read), transaction_put_cf (primary write), write (batch write), create_iterator_cf (iterator creation), transaction_commit (transaction commit)
2. Three types of BPF Maps — operation count statistics, latency distribution histograms (log2 buckets), slow-operation events exceeding thresholds
3. RocksDbCollector — user-space data collector that periodically polls Map data and formats output

The goal is to reach Milestone 2 by end of Week 5: ckb-probe rocksdb running on a testnet node, outputting meaningful RocksDB performance data with basic anomaly detection.