| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for the chunkah build invocation in priv-integration.sh. This step converts the FCOS test image into a non-ostree image and previously had no visibility into podman or chunkah internals when something went wrong here (as with the host-namespace-escape hang fixed in the prior commit). Cheap to leave on permanently since chunkah's own progress output is otherwise sparse, and useful for any future regressions in this step regardless of cause. This was originally proposed as part of PR bootc-dev#2392 alongside a (different, since superseded) fix attempt for the same underlying hang; keeping it here since it's independently useful. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for the chunkah build invocation in priv-integration.sh. This step converts the FCOS test image into a non-ostree image and previously had no visibility into podman or chunkah internals when something went wrong here (as with the host-namespace-escape hang fixed in the prior commit). Cheap to leave on permanently since chunkah's own progress output is otherwise sparse, and useful for any future regressions in this step regardless of cause. This was originally proposed as part of PR bootc-dev#2392 alongside a (different, since superseded) fix attempt for the same underlying hang; keeping it here since it's independently useful. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for the chunkah build invocation in priv-integration.sh. This step converts the FCOS test image into a non-ostree image and previously had no visibility into podman or chunkah internals when something went wrong here (as with the host-namespace-escape hang fixed in the prior commit). Cheap to leave on permanently since chunkah's own progress output is otherwise sparse, and useful for any future regressions in this step regardless of cause. This was originally proposed as part of PR bootc-dev#2392 alongside a (different, since superseded) fix attempt for the same underlying hang; keeping it here since it's independently useful. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
The "Test install" job's `priv-integration.sh` step hangs intermittently at the chunkah build (a nested `podman run` invoked via `systemd-run -dP --wait`). The root cause is not fuse-overlayfs performance or the chunkah build itself: it's that the outer `sudo podman run --privileged --pid=host ...` invocation bind-mounts the HOST's real systemd and dbus sockets in (`-v /run/dbus:/run/dbus -v /run/systemd:/run/systemd`), so every `systemd-run` call inside the container talks to the host's actual systemd manager rather than anything scoped to the container. `systemd-run --wait` asks systemd to fork and supervise the wrapped command, and with the host's real systemd manager on the other end of that dbus call, the forked process ends up living in the HOST's mount and IPC namespaces instead of the container's -- despite having been launched from inside the privileged container. When that escaped process is itself a `podman run` (as with the chunkah build), it starts contending on host-level podman locks (e.g. `/dev/shm/libpod_lock`) against the *actual* host-level `sudo podman run --privileged` process that is itself still running the CI job, which deadlocks forever. This was tracked down empirically: comparing `/proc/<pid>/ns/mnt` and `/proc/<pid>/ns/ipc` of the hung process against host PID 1 confirmed the escape, and reproduced the hang reliably in isolation by stripping priv-integration.sh down to just the chunkah step. None of the four `systemd-run -dP --wait` call sites here actually need systemd supervision: they're synchronous builds/copies (`podman build`, two `skopeo copy` calls, and the chunkah `podman run`) that work fine run directly, and were apparently only wrapped in systemd-run to get a cgroup for the nested container. Fixing this isn't just a matter of deleting those wrappers, though. `/run/dbus` can go entirely -- nothing here ever dials it once systemd-run is gone -- but `/run/systemd` has to stay mounted, because the `nsenter -m -t 1 journalctl _COMM=bootc` check a few lines above this depends on it: ostree-ext's structured logging silently skips writing to the journal at all unless `libsystemd::daemon::booted()` (a check for `/run/systemd/system`) is true, and this test's own "layers already present" assertion relies on that write happening. But leaving /run/systemd mounted for the whole script also breaks cgroup delegation for the nested podman invocations below (podman's cgroup-manager auto-detection only produces a working, fully delegated `cgroupfs` setup when it can't see a live systemd socket at all) -- and that turns out to hold regardless of `--cgroup-manager` flags at any level, since it's decided dynamically per invocation rather than once at container-creation time. So instead: keep /run/systemd mounted at container creation for the journal check, then have priv-integration.sh itself unmount it right after that check, before any of the systemd-run-free podman/build work that follows. Verified with the same namespace comparison that the nested chunkah process's mnt/ipc namespaces match the outer container's own entrypoint rather than the host's, and with a full, unmodified end-to-end run of priv-integration.sh that both the journal check and the final "ok privileged integration" line pass. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org> Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for the chunkah build invocation in priv-integration.sh. This step converts the FCOS test image into a non-ostree image and previously had no visibility into podman or chunkah internals when something went wrong here (as with the host-namespace-escape hang fixed in the prior commit). Cheap to leave on permanently since chunkah's own progress output is otherwise sparse, and useful for any future regressions in this step regardless of cause. This was originally proposed as part of PR bootc-dev#2392 alongside a (different, since superseded) fix attempt for the same underlying hang; keeping it here since it's independently useful. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
There was a problem hiding this comment.
lgtm, can't approve my own PR though
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
priv-integration.sh was intermittently hanging at the chunkah build step (a nested podman run invoked via systemd-run -dP --wait).
The root cause isn't fuse-overlayfs performance -- it's that the outer sudo podman run --privileged --pid=host ... invocation bind-mounts the host's real /run/systemd and /run/dbus into the container. Every systemd-run call inside therefore talks to the host's actual systemd manager, which forks the wrapped process into the host's mount/IPC namespaces instead of the container's. When that escaped process is itself a podman run (the chunkah build), it contends forever on host-level podman locks (/dev/shm/libpod_lock) against the real host-level sudo podman run --privileged process that's still running the CI job -- a deadlock.
This was confirmed empirically by comparing /proc/<pid>/ns/mnt/ns/ipc of the hung process against host PID 1, and reproduced reliably in isolation.
Update: an earlier version of this PR simply dropped both /run/systemd and /run/dbus and deleted the systemd-run wrappers. That fixes the hang, but a full end-to-end run turned up a second, unrelated issue it introduced: a few lines above the chunkah step, the script asserts on a "layers already present" message via nsenter -m -t 1 journalctl _COMM=bootc. ostree-ext's structured logging silently skips writing to the journal at all unless libsystemd::daemon::booted() (a check for /run/systemd/system) is true, so without /run/systemd that check has nothing to find and fails -- not a hang, just a different, real regression.
The fix now keeps /run/systemd mounted (only /run/dbus is dropped -- nothing here ever needed it) so that check keeps working, and has priv-integration.sh itself umount /run/systemd right after that check, before any of the systemd-run-free podman/build work later in the script. Cgroup delegation for nested podman run turns out to be re-evaluated dynamically on each invocation based on whatever systemd sockets are currently visible, not fixed once at outer-container-creation time, so this mid-script unmount is enough: with no live systemd socket at that point, podman's own cgroup-manager detection falls back to cgroupfs automatically and the nested chunkah container stays properly inside the outer container's own namespaces. None of the four systemd-run -dP --wait call sites need systemd supervision -- they're synchronous builds/copies that were only ever wrapped to get a cgroup for the nested container, which podman now arranges on its own.
Also keeps the --log-level=debug/RUST_LOG=chunkah=debug logging from the original version of this PR as a separate commit, since it's independently useful regardless of this fix.
Assisted-by: AI
I reviewed this investigation and the resulting patch; the root cause (and this later correction to it) was confirmed via repeated CI reproduction in a sandbox repo, including a full unmodified end-to-end run of priv-integration.sh with the exact patch applied, before landing here.