[DIS] FNN Safeguard — Recovery Verification and Pre-Upgrade Qualification for Fiber Nodes

DAOs & Funding CKB Community Fund DAO
1

Proposal summary

Proposal type: Budget Proposal
Applicant and project owner: Dapps over Apps
Project lead: Abdulkareem Oyeneye
Technical lead: Michael Dean
Funding requested: 8,263,000 CKB — fixed amount
Approximate USD reference: $9,800
Reference conversion: $0.001186 per CKB, captured on September 14, 2026
Project duration: 10 weeks
Contact email: [email protected]
Receiving address: ckb1qyqwy03adjsaajsdftkwddjqpdtdt52789ds8tau3j
Repository: https://github.com/intelliDean/fnn_safeguard
License: MIT or Apache-2.0, at the user’s option

Executive summary

FNN Safeguard is an open-source, self-hosted operator tool for checking whether a Fiber Network Node backup can be recovered and whether a target FNN release can process a clone of an operator’s state before the operator modifies the live node.

FNN already provides native backup, restore, and migration capabilities. FNN Safeguard does not replace those capabilities. It provides an independent operator-controlled verification layer around them.

The completed MVP can:

  • inspect a running FNN node through JSON-RPC;

  • trigger FNN’s native backup RPC;

  • detect the newly created backup directory;

  • validate the backup’s required database and key files;

  • create and verify a checksummed recovery manifest;

  • attempt recovery in a temporary environment;

  • run fnn --check-validate;

  • verify that the recovered Fiber identity matches the source;

  • start the recovered node state;

  • compare source and recovered channel and payment inventories;

  • generate machine-readable evidence describing the outcome.

FNN’s official admin RPC can create an immediate online backup of the Fiber store and node key files, while the official node documentation warns operators to back up state and treat upgrades carefully because channel state is operationally sensitive.

The grant will not reimburse the team for the completed MVP. Funding will support the next phase:

  1. evidence-grade isolated recovery;

  2. accurate recovery-result classification;

  3. state-specific pre-upgrade qualification;

  4. encrypted off-host recovery-point replication;

  5. public Fiber testnet validation;

  6. packaging, operator documentation, and bounded maintenance.

Applicant and prior Fiber work

Dapps over Apps is the applicant, accountable project owner, and proposed grant recipient.

Members of the Dapps over Apps team previously built FiberGuard Session, which placed third in Category 1 — Wallet and Payment UX Infrastructure in the Gone in 60ms Fiber Network Infrastructure Hackathon.

The official results highlighted FiberGuard Session’s least-privilege model, modular policy engine, gateway, SDK, approval console, audit logging, and working settlement path.

FNN Safeguard is a separate project addressing a different infrastructure boundary:

FiberGuard Session:
External application → Fiber node permission boundary

FNN Safeguard:
Fiber node → backup, recovery, and upgrade lifecycle

1. Problem

A backup directory existing on disk does not prove that an operator can recover from it.

A recovery point can still fail operationally because:

  • required files are missing or unreadable;

  • the database checkpoint cannot be opened;

  • the recovered node identity differs from the source;

  • the required configuration or password is unavailable;

  • the only backup copy exists on the same disk as the live node;

  • channel or payment records differ after recovery;

  • a target FNN release cannot open or migrate the operator’s state;

  • a cloned node accidentally reaches live Fiber peers;

  • the operator discovers these problems only after a failure or failed upgrade.

A related problem exists during upgrades. A tagged FNN release may work correctly in general, but an operator still needs to know whether that binary can open and migrate a clone of their own node state without detected identity changes, missing records, or database failures.

The project is based on a narrow operating principle:

A backup is not a verified recovery point until it has been restored and checked. A target release is not qualified for an operator’s node until it has been tested against a clone of that operator’s state.

FNN Safeguard will not claim to guarantee fund recovery or certify that an FNN release is universally defect-free. It will produce evidence about a specific backup, node identity, state inventory, and target binary.

2. Current MVP

The current public MVP provides three main commands:

fnn-safeguard inspect
fnn-safeguard backup
fnn-safeguard drill

2.1 Node inspection

inspect connects to a running FNN node and records a non-secret operational inventory, including:

  • node public key;

  • FNN version and commit;

  • network identifier;

  • channel count and state distribution;

  • payment count and status distribution;

  • channel-ID digest;

  • payment-hash digest;

  • sanitized configuration checksum.

2.2 Backup verification

backup can:

  • trigger the native FNN backup RPC;

  • capture the successful JSON-RPC response;

  • record directories that existed before the request;

  • detect the newly created timestamped backup directory;

  • wait for the backup files to stabilize;

  • verify the database checkpoint, Fiber identity key, and CKB key;

  • calculate file-level checksums;

  • generate a composite bundle checksum;

  • record source channel and payment fingerprints;

  • generate and independently verify a checksummed recovery manifest.

The repository includes the captured successful backup RPC response and the evidence showing the newly created backup directory and its detection time.

2.3 Recovery drill

drill currently:

  • creates a temporary recovery environment;

  • attempts the FNN restore path;

  • runs fnn --check-validate;

  • verifies the recovered node identity;

  • starts the recovered node state;

  • queries the recovered node through JSON-RPC;

  • compares source and recovered channel counts;

  • compares source and recovered channel-ID digests;

  • compares source and recovered payment counts;

  • compares source and recovered payment-hash digests;

  • writes machine-readable evidence.

The MVP was exercised against two distinct FNN nodes in the same local Fiber devnet environment.

The first node contained:

Node identity: distinct node 1 identity
ChannelReady channels: 7
Successful payments: 6

The second node contained:

Node identity: distinct node 2 identity
ChannelReady channels: 7
Successful payments: 5

The two recovery points contain different node identities, key material, configurations, RocksDB files, bundle checksums, and payment histories. For each recovery point, the recovered identity, channel count, channel-ID digest, payment count, and payment-hash digest matched the corresponding source state.

The repository’s current CI run is green and covers formatting, Clippy, tests, and use of the clean official FNN v0.9.0 release binary.

3. Current MVP boundaries

The proposal intentionally separates what the MVP already demonstrates from what remains to be delivered.

3.1 Existing evidence is local devnet evidence

The two completed evidence runs were produced using distinct FNN nodes inside the same local Fiber devnet environment.

They demonstrate real FNN channel state, payments, backup creation, database recovery, identity comparison, and inventory comparison.

They are not presented as:

  • public Pudge testnet evidence;

  • two independent external operator pilots;

  • production-mainnet recovery proof.

Public testnet and separate-host validation are included in the funded scope.

3.2 The official FNN v0.9.0 restore command failed in both runs

In both current evidence runs, the official FNN v0.9.0 restore command exited with a missing-SST error.

FNN Safeguard then used a direct-checkpoint recovery fallback. The resulting database:

  • passed fnn --check-validate;

  • started successfully;

  • retained the expected Fiber identity;

  • returned matching channel and payment inventories.

The public restore logs record the failed official restore attempt and the subsequent fallback.

The current result should therefore be understood as:

Official FNN restore attempt: FAILED
Direct-checkpoint recovery fallback: PASSED
fnn --check-validate: PASSED
Recovered identity comparison: MATCHED
Recovered channel inventory: MATCHED
Recovered payment inventory: MATCHED

The proposal does not claim that the native restore command succeeded.

The funded implementation will preserve the official command’s real exit status and report any fallback as a separate recovery method.

3.3 Process-mode egress isolation is not yet proven

The current restored-node inventory comparison runs with loopback binding, but the committed process-mode evidence does not prove that outbound network connections were blocked.

The current evidence labels this honestly as:

ISOLATION_NOT_PROVEN

The funded phase will add an environment where:

  • external egress is attempted;

  • the attempt must fail;

  • the failure is recorded;

  • the recovered node remains locally queryable.

3.4 Current secret scanning is limited

The current MVP includes a generic marker scan for common credential and private-key labels.

It does not yet prove detection of every raw, hexadecimal, or base64 representation of the actual controlled test keys, passwords, or tokens.

The funded phase will add known-secret comparison against controlled test credentials before the strongest recovery status can be issued.

4. Proposed funded work

Workstream 1 — Evidence-grade isolated recovery

FNN Safeguard will add an isolation backend in which:

  • the cloned Fiber identity cannot reach public Fiber peers;

  • the recovered node remains locally queryable;

  • a real outbound connection attempt is performed;

  • the outbound connection attempt must fail;

  • the isolation evidence is recorded;

  • official restore and fallback recovery results remain separate;

  • the strongest verified status requires proven isolation;

  • no live node state is modified automatically.

The implementation may use:

  • an internal-only Docker network;

  • a Linux network namespace;

  • rootless Podman;

  • or another reproducible mechanism with equivalent guarantees.

Recovery result taxonomy

The funded version will use explicit results such as:

STRUCTURAL_CHECK_PASSED
RESTORE_VALIDATED
RESTORE_AND_INVENTORY_VERIFIED
ISOLATED_RECOVERY_POINT_VERIFIED
VERIFIED_WITH_FALLBACK
FAILED

An official restore command that exits unsuccessfully will never be reported as an official restore success.

Workstream 2 — Pre-upgrade qualification

Add:

fnn-safeguard qualify --target <binary-or-image>

The command will:

  1. select a verified recovery point;

  2. create a separate clone;

  3. pin and record the target binary or container-image digest;

  4. run migration only against the clone;

  5. start the target FNN version in the isolated environment;

  6. query the migrated state;

  7. compare the source and migrated identity and inventories;

  8. produce an evidence-backed PASS, WARN, or BLOCK result.

The initial implementation will test at least one qualification path across two tagged FNN v0.9.x releases.

Qualification checks

The comparison will include:

  • node public key;

  • database validation;

  • migration exit status;

  • startup result;

  • channel count;

  • channel-state distribution;

  • channel-ID digest;

  • payment count;

  • payment-status distribution;

  • payment-hash digest;

  • unexpected missing or additional records.

Decision semantics

PASS
The target binary starts successfully against the clone, identity matches,
and all required state fingerprints match.

WARN
The target starts and preserves the required records, but a documented
non-fatal operational difference requires operator review.

BLOCK
Migration fails, startup fails, identity changes, required records are
missing, or state fingerprints differ unexpectedly.

This is state-specific qualification. It will not be described as universal certification of an FNN release.

Workstream 3 — Encrypted off-host recovery points

FNN Safeguard will add operator-controlled encrypted replication using established backup software rather than custom cryptography.

The planned implementation will use Restic or an equivalent mature tool and support:

  • local encrypted repositories;

  • SFTP repositories;

  • S3-compatible repositories;

  • retention policies;

  • snapshot verification;

  • remote download;

  • checksum revalidation;

  • recovery drills from downloaded snapshots.

Example usage:

fnn-safeguard replicate \
  --backup latest \
  --repository <restic-repository>

fnn-safeguard verify-remote \
  --snapshot latest

fnn-safeguard drill \
  --remote latest

FNN Safeguard will remain self-hosted.

Dapps over Apps will not:

  • operate a hosted backup service;

  • receive operator private keys;

  • take custody of operator backups;

  • retain operator passwords;

  • implement custom encryption algorithms.

Workstream 4 — Packaging and validation

The funded phase will also deliver:

  • versioned Linux binary releases;

  • versioned Docker packaging;

  • systemd service and timer examples;

  • JSON and human-readable reports;

  • public Fiber testnet evidence;

  • validation on at least two separate hosts;

  • an operator recovery runbook;

  • a pre-upgrade qualification runbook;

  • a threat model;

  • a supported-version compatibility matrix;

  • a final completion report;

  • a demonstration video;

  • 90 days of bounded compatibility maintenance.

5. Scope exclusions

The project will not implement:

  • automatic live rollback;

  • automatic production upgrades;

  • automatic replacement of active channel state;

  • automatic force-closing of channels;

  • hosted custody of backups or keys;

  • custom cryptography;

  • a general Fiber node-management dashboard;

  • wallet payment or channel-management interfaces;

  • liquidity automation;

  • LSP operation;

  • CCH orchestration;

  • application permission sessions;

  • universal guarantees of fund recovery.

These exclusions are on purpose.

Restoring older payment-channel state after a live node has processed newer channel activity can create fund-safety risk. FNN Safeguard will verify recovery and produce operator-facing evidence, but it will not autonomously overwrite a live funded node.

6. Non-duplication

FNN native backup and restore

FNN remains responsible for producing native backups and implementing its own database recovery and migration behavior.

FNN Safeguard adds an independent operator layer for:

  • integrity manifests;

  • recovery-point verification;

  • isolated drills;

  • source-versus-recovered state comparison;

  • off-host replication;

  • target-release qualification;

  • evidence generation.

Its usefulness remains even if the currently observed restore-path behavior is fixed upstream.

Fiber Studio

Fiber Studio is a graphical desktop application for managing a local Fiber node, payments, channels, and node updates.

FNN Safeguard is a headless Linux and Docker operator tool for server deployments, backup verification, recovery drills, and release qualification.

It will not reproduce Fiber Studio’s payment, channel, or desktop-management interfaces.

Fiber testing projects

Fiber testing environments create controlled development networks and failure scenarios.

FNN Safeguard takes an operator-specific recovery point as input and answers:

Can this recovery point be opened?
Does the recovered identity match?
Does the recovered channel and payment state match?
Can a target FNN release process this cloned state?

FiberGuard Session and Clasp

FiberGuard Session and Clasp address app-to-node or app-to-wallet permission boundaries.

FNN Safeguard addresses the backup, recovery, and upgrade lifecycle of persistent FNN deployments.

The scopes do not overlap.

7. Why the Community Fund DAO should fund this

FNN Safeguard is shared ecosystem infrastructure rather than a commercial end-user application.

Its outputs will be:

  • open source;

  • self-hosted;

  • operator-controlled;

  • usable without a hosted dependency;

  • usable without transferring keys or backups to the project team;

  • measurable through public acceptance criteria;

  • valuable without requiring an upstream merge.

The team has already funded and completed the core MVP.

The grant funds the next production phase:

Completed before funding:
- FNN inspection
- Native backup triggering
- Backup-directory detection
- Checksummed manifests
- Recovery attempt and checkpoint fallback
- Database validation
- Identity verification
- Recovered channel and payment inventory comparison

Grant-funded:
- Proven network isolation
- Accurate recovery-result classification
- Pre-upgrade qualification
- Encrypted off-host replication
- Public-testnet validation
- Separate-host pilots
- Packaging, documentation, and maintenance

8. Milestones

Milestone 1 — Production isolation and recovery semantics

Timeline: Weeks 1–3
Budget: 2,361,000 CKB

Deliverables

  • evidence-grade isolation backend;

  • locally queryable recovered node with public egress blocked;

  • tested failed-egress probe;

  • separate official-restore and fallback-recovery reports;

  • revised result taxonomy;

  • known-secret comparison tests;

  • sanitized upstream report for the observed restore-path failure;

  • updated threat model and security documentation.

Acceptance criteria

  • the cloned node cannot reach a configured external endpoint;

  • local RPC inspection remains possible;

  • the official restore exit code is preserved;

  • fallback recovery is separately labelled;

  • missing isolation proof prevents the strongest verified result;

  • raw, hex, and base64 representations of controlled test secrets are checked;

  • the final report is written only after all checks complete.

Milestone 2 — Pre-upgrade qualification

Timeline: Weeks 4–6
Budget: 2,782,000 CKB

Deliverables

fnn-safeguard qualify --target <binary-or-image>

  • qualification against a cloned recovery point;

  • pinned binary and image digests;

  • migration and startup capture;

  • pre/post identity comparison;

  • pre/post channel comparison;

  • pre/post payment comparison;

  • PASS, WARN, and BLOCK decisions;

  • negative and destructive tests;

  • supported-version compatibility matrix;

  • upgrade-planning report.

Acceptance criteria

  • live node files are never modified;

  • migration or startup failure produces BLOCK;

  • identity mismatch produces BLOCK;

  • missing records or digest differences produce BLOCK;

  • documented non-fatal operational differences produce WARN;

  • a matching qualification produces PASS;

  • every decision links to machine-readable evidence.

Milestone 3 — Encrypted off-host recovery

Timeline: Weeks 7–8
Budget: 1,940,000 CKB

Deliverables

  • encrypted Restic-based replication;

  • local repository support;

  • SFTP repository support;

  • S3-compatible repository support;

  • retention policy support;

  • remote snapshot verification;

  • download and checksum verification;

  • recovery drill from a downloaded snapshot.

Acceptance criteria

  • a recovery point is encrypted outside the live node directory;

  • the remote snapshot is downloaded and independently reverified;

  • the downloaded recovery point passes an isolated drill;

  • no operator secret appears in public evidence;

  • the project team does not receive or host the operator’s backup.

Milestone 4 — Packaging, public validation, and documentation

Timeline: Weeks 9–10
Budget: 1,180,000 CKB

Deliverables

  • versioned Linux release;

  • versioned Docker release;

  • systemd service and timer examples;

  • public Fiber testnet report;

  • validation on at least two separate hosts;

  • recovery runbook;

  • upgrade-qualification runbook;

  • final demonstration video;

  • completion report;

  • 90-day compatibility-fix commitment.

Acceptance criteria

  • a fresh Linux environment can reproduce the documented workflow;

  • public-testnet evidence is clearly distinguished from local-devnet evidence;

  • binaries and images are pinned by version and digest;

  • all known limitations are documented;

  • final deliverables remain public under MIT or Apache-2.0.

9. Budget

Work package CKB
Production isolation and recovery hardening 2,361,000
Pre-upgrade qualification 2,782,000
Encrypted off-host recovery 1,940,000
Packaging, validation, documentation, and maintenance 1,180,000
Total 8,263,000 CKB

The fixed CKB request is approximately $9,800 at the stated reference rate.

The budget covers engineering, testing, public evidence, packaging, documentation, separate-host validation, and 90 days of compatibility maintenance.

It does not include:

  • retroactive payment for the completed MVP;

  • travel;

  • hardware purchases;

  • marketing campaigns;

  • hosted infrastructure sold as a service;

  • custody of operator funds or backups;

  • a formal third-party security audit.

Disbursement may follow the Community Fund DAO’s applicable milestone-review process.

10. Team

Dapps over Apps — Applicant and project owner

Dapps over Apps is responsible for:

  • grant accountability;

  • milestone delivery;

  • financial reporting;

  • public communication;

  • project ownership;

  • final-deliverable coordination.

Official project communication:

[email protected]

Abdulkareem Oyeneye — Project Lead

Responsibilities:

  • project planning;

  • milestone and scope management;

  • DAO communication;

  • budget accountability;

  • public progress reporting;

  • operator outreach;

  • coordination between engineering and community review.

Michael Dean — Technical Lead

GitHub:

https://github.com/intelliDean

Responsibilities:

  • Rust implementation;

  • FNN RPC integration;

  • backup and recovery engine;

  • identity and state verification;

  • isolation backends;

  • release qualification;

  • compatibility testing;

  • technical evidence;

  • release engineering.

Steven Hert — Engineering Contributor

Planned responsibilities:

  • deployment and packaging support;

  • workflow reproduction;

  • integration and regression testing;

  • operator-environment validation;

  • documentation review;

  • milestone evidence review.

No additional qualifications or prior deliverables are being claimed for Steven Hert beyond his assigned work on this project.

11. Risks and mitigations

FNN release or RPC changes

Risk: FNN interfaces or storage behavior change during implementation.

Mitigation: Pin every tested binary and container image, record digests, publish a compatibility matrix, and limit support claims to explicitly tested versions.

False confidence from incomplete verification

Risk: A structural check is interpreted as full recovery verification.

Mitigation: Use explicit result levels and fail closed when identity, inventory, isolation, binary provenance, or secret checks have not completed.

Cloned identity reaches live peers

Risk: A recovered node connects to the Fiber network using copied identity material.

Mitigation: Require a measured failed-egress test before issuing the strongest recovery status.

Secret leakage

Risk: Keys or credentials appear in evidence files.

Mitigation: Keep processing local, sanitize configuration, compare evidence against controlled test-secret representations, and generate the final report only after the scan completes.

FNN restore behavior changes

Risk: A later FNN release changes or fixes the restore behavior currently observed.

Mitigation: Keep native restore and fallback recovery separate. If the native path succeeds, the fallback is unnecessary. Recovery verification, replication, and upgrade qualification remain useful.

State-specific results are mistaken for universal certification

Risk: Operators interpret a passing qualification as proof that the release contains no defects.

Mitigation: Every report will state the exact source recovery point, target binary, environment, checks executed, and limits of the result.

12. Maintenance

Dapps over Apps commits to 90 days of maintenance after the final delivery, covering:

  • reproducible defect fixes;

  • compatibility corrections for supported FNN versions;

  • documentation corrections;

  • issue triage;

  • fixes required to satisfy the agreed acceptance criteria.

The maintenance commitment does not include:

  • indefinite feature development;

  • hosted operations;

  • unrelated future FNN functionality;

  • permanent support for every historic FNN version.

13. Public deliverables

All funded code, documentation, acceptance tests, evidence schemas, and completion reports will remain public under MIT or Apache-2.0.

FNN Safeguard repository

https://github.com/intelliDean/fnn_safeguard

Fiber Network repository

https://github.com/nervosnetwork/fiber

Fiber hackathon results

https://talk.nervos.org/t/gone-in-60ms-fiber-network-infrastructure-hackathon-results/10671

14. Accountability statement

Dapps over Apps is the applicant, project owner, and accountable grant recipient for FNN Safeguard.

Michael Dean serves as Technical Lead. Steven Hert serves as an Engineering Contributor.

All official grant communication, milestone reporting, financial accountability, and project correspondence will be managed by Dapps over Apps through:

[email protected]

Grant receiving address:

ckb1qyqwy03adjsaajsdftkwddjqpdtdt52789ds8tau3j
5 Likes
CKB Ecosystem Biweekly Update
CKB Ecosystem Biweekly Update
2

@zz_tovarishch @quake, could you please advise on the appropriate next step for moving this proposal forward? Thank you!

3

Hi Dappsoverapps, I suggest you share your ideas on Telegram: View @NervosNation and Telegram: View @NervosNetwork to engage more with community members and gather their feedback.

1 Like
4

While I appreciate the effort behind this proposal, much of the scope appears to involve relatively straightforward integration of existing tools. It’s not yet clear to me what distinct value it adds beyond that, or whether that value is sufficient to justify a dedicated DAO grant.

5

Thanks for the feedback @quake The main value here isn’t just connecting existing backup tools. It’s making sure an FNN backup can actually restore the same node identity and channel/payment state, and letting operators test a new FNN version against a copy of their real node state before touching production.

That Fiber-specific recovery and upgrade verification is already the core of the proposal; the existing backup integrations are only supporting pieces.

6

@phroi @knmo @Yeti @ArthurZhang @Ayoub_Lesfer . We would appreciate it if you could take a look at this proposal when you have a chance. Any thoughts would be appreciated. Thank you.

7

Just to clarify my perspective, I wasn’t previously familiar with this project and haven’t been involved in its design or development, so I can only comment as an outside observer based on the proposal and the discussion here.

Also I broadly agree with quake’s concern. The problem being addressed is legitimate, but much of the proposed work currently looks like integration and operational tooling around capabilities that already exist, rather than a sufficiently distinct technical contribution in its own right.

I also think the design could be specified more rigorously. For a system concerned with recovery verification and pre-upgrade qualification, it would help to define the relevant safety invariants, failure model, state-equivalence criteria, and upgrade/rollback semantics more explicitly, and then derive the implementation from those requirements.

So my concern is not that the problem is unimportant, but that I do not yet see enough technical differentiation or design rigour to justify treating this as a distinct safeguard system rather than an integration layer around existing FNN recovery and operational tooling.

2 Likes
8

Thanks for the tag. I don’t run a Fiber node so I can’t judge the recovery side properly, but on quake’s point, what would help me as a reader is one concrete case where a native backup looked fine and then failed to restore, or came back with different channel state. If you’ve hit that on testnet I’d put it right at the top.

9

Isn’t it true that once files have been copied or extracted, security assumptions become obsolete anyway?

To me, this sounds like a “reproducible build” approach. If even Quake doesn’t see the point of your idea, I’m probably not qualified enough to comment here. In any case, thanks for the post—a forum is meant for discussion.

2 Likes