@ArthurZhang @Ayoub_Lesfer @quake, thank you for reviewing the proposal. We have updated the MVP and published the recovery evidence, safety specification, and implementation.
The forum is restricting links, so I have included the repository name and file paths below.
GitHub repository: intelliDean/fnn_safeguard
Revision: 89badd585a2cea7de71664aa8f06f01cba23a698
The concrete recovery case
On two distinct nodes in our local Fiber devnet, the native FNN backup RPC completed and created backups, but the subsequent FNN v0.9.0 restore attempts failed with missing SST-file errors.
A separately recorded checkpoint-copy recovery then passed fnn --check-validate. The recovered nodes matched their respective source inventories on node identity, channel counts, channel-ID digests, payment counts, and payment-hash digests.
Node 1: 7 channels and 6 successful payments.
Native restore log:
evidence/devnet-node1/restore-output.log
Recovery report:
evidence/devnet-node1/final-report.json
Node 2: 7 channels and 5 successful payments.
Native restore log:
evidence/devnet-node2/restore-output.log
Recovery report:
evidence/devnet-node2/final-report.json
The native restore failure and subsequent recovery are recorded separately, so the reports show which operation failed and which recovery path worked.
Safety model and implementation
To address the design questions, we have added a safety specification covering the failure model, recovery requirements, result definitions, and upgrade/rollback boundaries:
docs/SAFETY_MODEL.md
The comparison implementation checks:
- Recovery-point integrity: file checksums and whether the backup directory changed during the restore operation.
- Identity and inventory matching: node identity, channel counts and identifier digests, channel-state distributions, payment counts and hash digests, and payment-status distributions.
- Recovery outcomes: separate native and fallback results, database validation, and explicit isolation status.
Comparison implementation:
src/commands/drill.rs
Decision engine:
src/core/invariants.rs
The decision engine derives the recovery verdict from these checks. The tests cover decision logic for identity and inventory mismatches, database-validation failure, recovery-point mutation, detected secret leakage, and isolation failure.
Tests:
tests/test_invariants.rs
CI for this revision passed formatting, Clippy, and the test suite. The run is available under the repository’s Actions tab, run ID 38831337194.
This is the FNN-specific contribution: checking recovery outcomes against defined requirements, rather than treating a successful backup operation as evidence that recovery will work. The proposed pre-upgrade work extends that approach to testing a target FNN release against a cloned recovery point before changing the live node.