I built a working Phase-0 reference implementation of this RFC and put it up as a public lab, so the storage model isn't just on paper:

→ https://github.com/xbizzybone/flipper-os-lab

It reproduces the whole Phase-0 storage stack on loop-backed disk images on any Linux host (no RK3576 board, no QEMU) — every mechanism is the genuine kernel feature, just on files instead of eMMC:

• Immutable base = squashfs + dm-verity (read-only lower); a tamper test flips a byte and watches verity reject it.
• Profiles = Btrfs subvolumes used as the OverlayFS upperdir.
• clone = btrfs subvolume snapshot (copy-on-write, O(metadata)); reset = restore from a read-only pristine snapshot — no re-flash.
• /data = separate partition, bind-mounted, survives both profile reset and base swap.
• base_min_version guard refuses to mount a profile against an incompatible base.
• Anti-drift lint implements the build-time MUST: it fails the profile if its upper shadows any base file outside a whitelisted drop-in dir. The demo plants an illegal shadow of etc/flipper/base.conf and the lint rejects it.

initramfs/flipper-overlay.sh is the reference local-bottom hook showing how the lab's boot <id> maps onto a real device — the only thing that changes is where the profile id comes from (lab CLI → kernel cmdline flipper.profile= → MCU BOOT_SELECTION over Interconnect, exactly as the RFC describes).

CI proof: GitHub Actions runs the full 8-step demo on ubuntu-22.04 with real dm-verity on every push — root hash, tamper rejection, lint pass+reject, and /data survival, exit 0: https://github.com/xbizzybone/flipper-os-lab/actions/workflows/demo.yml

Scope is honest about phase boundaries: A/B + RAUC + MCU watchdog (Phases 1–2) are referenced, not implemented yet — that's the roadmap. Two minor deltas vs. the RFC: the lab's /data is ext4 (RFC says Btrfs±LUKS — LUKS is Phase 3) and verity hashes live in a sidecar file rather than appended (cosmetic for the lab).

Happy to evolve this toward Phase 1 (QEMU aarch64 + U-Boot A/B slot selection + RAUC bootcount rollback) if useful for the discussion.

We’ve been thinking along similar lines, but without squashfs, which is rather limiting in our context, as it forces a single base image and precludes storage-level data deduplication if more than one are used.

Right now we are experimenting with a stack fully based around Btrfs snapshots, which allows multiple base profiles to coexist space-efficiently, with instantaneous cloning/branching via native CoW functionality. We haven’t yet considered the introduction of verity into the chain though (or discussed if it’s vital for our envisaged usage scenarios and target audience)

Really nice work!

A couple of things I keep wondering about:

• Base swap vs profile: when A/B replaces the base and base_min_version no longer matches, does the profile get migrated, refused, or reset?
• Backup: any restore/export path for /data or for a profile itself (send/receive), or is that for later?
• Packages per profile: full userland in each overlay, or shared base + deltas? And does anything dedup across them?

Mainly trying to get my head around the day-2 side: updates, rollback, recovery, not just the layout.

@alchark you're right on the substance, and I want to concede it cleanly: a squashfs base is an opaque compressed blob, so it forces a single active base per slot and can't extent-share with a second base — there's no storage-level dedup across bases. The PoC shows exactly that shape: one base.squashfs, with CoW sharing happening only among profiles on the Btrfs pool, never touching the base. So this isn't hypothetical.

I'll also own a gap in the RFC's framing. We listed "three options" and rejected full-image-per-profile on storage-scaling grounds ("N × image × 2"), but we never evaluated your design — multiple coexisting Btrfs base subvolumes that reflink-share. That's a distinct fourth point in the space, it answers the same scaling objection without giving up apt or waiting on deb-ostree, and it deserved an explicit weighing rather than silence. So I've gone ahead and added it: the latest revision on this PR now has Alternative D — All-Btrfs base (multiple reflink-sharing base subvolumes), credited to this review.

Worth being clear about what we already share: profiles are Btrfs subvolumes today, and the PoC demonstrates the CoW snapshot clone/branch/reset you describe (clone = btrfs subvolume snapshot, reset = restore from a read-only pristine snapshot — O(metadata) by construction, though I haven't benchmarked wall-clock time). So the profile layer is already the Btrfs-CoW half of what you're describing; where we diverge is putting the base itself on Btrfs.

The crux is verity, and it's genuinely a fork rather than a disagreement. Classic whole-device dm-verity needs a static read-only hash-tree device and is mutually exclusive with a writable Btrfs-snapshot base — that's exactly why the RFC pushed the base onto squashfs (it also keeps RAUC's A/B slots and the MCU/U-Boot early-boot path simple). What the PoC actually proves here is narrow but concrete: a byte-flipped base image is rejected against its sealed dm-verity root hash in CI. I don't want to overstate it — that's offline image verification, and RAUC signature verification / fs-verity / Btrfs-checksum integrity are not built yet.

You're asking the right question — is verity vital for your audience? For a tool whose owner deliberately runs untrusted code with physical access, base verity buys a narrow thing: continuous at-rest tamper-evidence (rootkit persistence, bad flash, bit-rot) plus a hash-pinned rollback target. It's an anti-persistence story, not load-bearing for the profile/clone/reset UX. So in the same revision I also reframed base integrity as an explicit pluggable axis — dm-verity / fs-verity / Btrfs-checksums / none-plus-signed-update — selected by threat model, rather than baking squashfs+dm-verity into the partition layout. dm-verity stays the default; choosing fs-verity or signed-update-only is precisely what unlocks Alternative D.

Net: if verity is a hard requirement, squashfs buys it cheaply at the cost of multi-base dedup. If it isn't, your all-Btrfs multi-base stack wins on storage and coexisting-bases UX essentially by construction. I'd love to chase a hybrid — sealed read-only Btrfs base snapshots + fs-verity — so we keep native CoW/branching and an integrity story. To write Alternative D up fairly I'd rather use your real numbers than assume: if you can share the multi-base layout you're experimenting with, and how you're handling atomic A/B update + rollback on a single Btrfs pool (RAUC's discrete-slot model doesn't map cleanly onto one shared pool — I'd expect snapshot-swap orchestration instead), I'll flesh it out with actual tradeoffs.

@Yury-MonZon thanks — these are exactly the day-2 issues the layout has to earn, so let me be precise about what's designed vs. what actually runs today.

base_min_version mismatch after a base swap → refused, not migrated or reset. The manifest's base_min_version is an anti-corruption guard: on an incompatible base the profile MUST refuse to mount and offer a rebuild, rather than silently overlaying onto a base it wasn't built against. The Phase-0 PoC enforces the refusal in the boot path. Two honest caveats: the "rebuild" path is named in the RFC but not yet specified (what it regenerates, and from what), and the PoC currently does an exact base-version match rather than a real "≥ minimum" comparison — so the "min version" semantics are aspirational at the moment, not implemented.

Packages per profile & dedup → shared base + deltas, never a full userland per profile. Userland is served read-only from the single immutable base (squashfs + dm-verity) as the OverlayFS lower; each profile's Btrfs subvolume is just the upper, holding only its delta — drop-in config, packages baked at profile build time (the profile.toml [packages] manifest), and runtime writes. One scope caveat: the PoC's profiles currently carry only drop-in config — build-time package baking is Phase 3 and isn't implemented yet, so I'm describing that path from the design, not a running demo.

On dedup, to be precise rather than overclaim: there's dedup in exactly one place today — profiles live in a Btrfs pool, so clone is a reflink snapshot sharing extents with its parent via CoW (only the diff costs space, O(metadata)); the PoC proves that. What does not dedup: (1) two independently-built profiles baking the same package — no content-addressed store, so they get separate copies; and (2) anything across base images — the base is deliberately squashfs outside Btrfs, so multiple bases share no blocks. That second point is exactly what @alchark raised above, and it's now an open tradeoff in the RFC (Alternative D / unresolved question #4). The content-addressed answer that would dedup all of it (ostree, Option C) is acknowledged but deferred.

Backup / export → honest split. Profile export is on the roadmap but undesigned — btrfs send/receive is a clean fit since profiles are already CoW subvolumes, but there's no committed artifact format, no signing/verification story yet (the overlay isn't verity-protected, so an imported profile is user-trusted — a real trust gap for a "profile store"), and no cross-base-version compatibility on import. And /data backup/restore is a genuine gap — honestly "for later"; the RFC neither designs nor mentions it and the PoC doesn't touch it. Happy to spec the export artifact + trust model if there's appetite.

Day-2 overall (updates / rollback / recovery) → designed, not yet demonstrated. Rollback is a U-Boot boot-counter plus a flipper-health.service that only commits a slot after it proves itself, with the MCU watchdog as a hardware backstop that can force a rollback even if Linux wedges. But to be straight about scope: the PoC proves Phase 0 (the storage model) only — A/B + RAUC + watchdog are Phases 1–2 and aren't built yet. The natural next step is a Phase-1 demo (QEMU aarch64 + U-Boot A/B slot selection + RAUC boot-count rollback) to make the rollback story real rather than asserted.