systrap: fail-stop sentry on stuck context by NahumLitvin · Pull Request #14201 · google/gvisor · GitHub

A systrap task can remain in sleepOnState forever when its stub stops responding. The existing path sends a single interrupt and logs repeated warnings, but it has no retry and no escape. Container teardown can then block indefinitely in WaitExited, leaving the sandbox and its control RPCs stuck.

We hit this in a production Kubernetes cluster. Multiple gVisor pods remained Terminating for days, on both release-20260427.0 and release-20260714.0. In one affected sandbox, both runsc kill and runsc debug --stacks hung while the sentry stayed alive. Systrap [exe] stubs and node load continued to accumulate even though CPU use remained low. Killing the sentry process tree cleared the pod and returned node load to normal immediately. Full forensics in #14408.

#14405 is an independent production report of the same hang, with the sentry stacks we could not capture: a task goroutine parked 7,818 minutes in sleepOnState on the ThreadContext.State futex after one missed SIGUSR1, Kernel.Pause() blocked behind it so the URPC handler for runsc kill --all never returns. On the shim side 45,646 goroutines piled up on Init.mu behind the hung kill, growing containerd and kubelet memory until node OOM.

The fix, staged from least to most disruptive:

Because systrap: sleepOnState() never returns when a stub thread is unresponsive, permanently hangs Kernel.Pause() #14405 shows a single interrupt can simply be lost, the interrupt is resent on every checkup wakeup instead of once. A lost signal recovers the workload with no visible damage.
If the stub stays unresponsive through repeated interrupts for the full stuck-context deadline (raised from 30s to 3 minutes, matching the watchdog's stuck-task default, so transient host pressure has time to recover), all goroutine stacks are dumped via log.TracebackAll. Stacks are unobtainable from outside a wedged sandbox, so the escalation self-documents what systrap: sleepOnState() never returns when a stub thread is unresponsive, permanently hangs Kernel.Pause() #14405 had to capture by hand.
Only the stuck subprocess is then killed, through the same path NotifyInterrupt already uses when a stub is gone (mark dead, ContextStateUnexpectedDeath, kill the syscall thread). Healthy subprocesses in the sandbox are unaffected, and the task goroutine returns instead of blocking Kernel.Pause() forever.

sleepOnStateWithTimeout returns a typed error and stays side-effect free; sleepOnState does the traceback and the kill. A stub that is already gone takes the quiet NotifyInterrupt path (warn + kill subprocess, no stack dump). Resent interrupts do not extend the stuck deadline; a regression test covers that.

Tests:

bazel test //pkg/sentry/platform/systrap:systrap_test
bazel build //runsc:runsc
Stuck, stub-gone, repeated-interrupt, and recovered-context cases covered on the typed error value.

Fixes #14408. Related to #14405 and #12209.

Assisted-by: Claude Code

A systrap task can wait forever for ThreadContext.State when its stub stops responding. The current checkup sends an interrupt, but if that does not recover the context it only logs and continues waiting. Container teardown can then block indefinitely in WaitExited. After the existing 30-second stuck-context deadline and a complete checkup interval, terminate the sentry. Unexpected systrap stub death is already treated as fatal, and fail-stop avoids partial cleanup of shared systrap state. Recheck the state after every futex return so a context that recovered during the wait is not terminated. Deadline and liveness checks also run after EINTR and EAGAIN wakeups. Add tests for the stuck timeout, missing-stub ESRCH path, and recovery race. Related to google#12209. Assisted-by: Claude Code

@EtiennePerot @konstantin-s-bogom gentle ping. we hit this exact freeze in production again today (sentry unresponsive, kubelet KillPodSandbox DeadlineExceeded, pod stuck Terminating ~1.5h until manual kill -9 on the node). if the fail-stop direction is wrong say so and il rework it, can also add tests if that helps

Filed #14408 with the full production forensics for the freeze this PR addresses (two releases affected, node-side signature, why stacks are unobtainable).

Similar to issue #14405 , Sentry hang causes containerd and kubelet memory to grow unbounded, eventually the whole node OOM.

…ail-stop A single interrupt to a stuck context can be lost (google#14405 captured a task goroutine wedged forever after one missed SIGUSR1). Resend the interrupt on every checkup wakeup so a lost signal recovers the workload, and fail-stop only when repeated interrupts prove the stub is unresponsive. Resent interrupts do not extend the stuck deadline. Before terminating the sentry, dump all goroutine stacks via log.TracebackAll: stacks are unobtainable from outside a wedged sandbox (control RPCs hang, google#14408), so the fail-stop path self-documents. Updates google#14405. Fixes google#14408.

@tanyifeng im sure this is the same incident as #14405. your dump shows the task goroutine parked in sleepOnState on the ThreadContext.State futex after a single missed SIGUSR1, with Kernel.Pause() wedged behind it so runsc kill --all never returns. thats exactly what we see from the outside in #14408: sentry alive, control RPCs hang (runsc debug --stacks included, which is why we never got the stacks you did), kubelet KillPodSandbox DeadlineExceeded forever, and later kills queueing behind the shims Init.mu, same as your 45k blocked goroutines. same loop, observed from two sides.

your dump also changed my fix: a single lost SIGUSR1 is recoverable, so im updating this PR to resend the interrupt on every checkup wakeup and only fail-stop when repeated interrupts prove the stub is gone, plus dumping all goroutine stacks before killing the sentry so the next occurrence self-documents what you had to capture by hand. would love your review on it once i push, hopefully today

Thank you for updating the fix based on the evidence from #14405. Resending SIGUSR1 and dumping the stacks before exiting are both helpful improvements.

I’m not a reviewer, but I’m personally concerned that terminating the entire sentry after 30 seconds may be too aggressive. A context remaining unchanged for 30 seconds does not necessarily mean that the stub is unrecoverable; it could also be caused by host performance issues or CPU scheduling delays.

I would prefer a staged approach: first retry SIGUSR1 and collect diagnostic information, then terminate the sentry only if the stub is confirmed to be gone, repeated interrupts fail to make progress, or the sandbox is already being torn down. Would it also be possible to add separate timeouts for Stats and Kill in the shim to prevent unbounded goroutine and memory growth?

We have enabled --panic-signal=12 by default in our environment. If this happens again, we can send the configured signal to the sentry to trigger a panic and capture the full stack dump, even if runsc debug --stacks is unavailable.

This issue is rare, but its impact is severe when it occurs. @ayushr2 , could you please take a look and help move the fix forward?

@tanyifeng good points, thanks for the push. on the staged approach, the updated diff already does most of it: SIGUSR1 is resent on every 5s checkup, state is rechecked after every futex wakeup so a stub that makes any progress escapes, all goroutine stacks are dumped before the kill, and ESRCH fail-stops immediately since there the stub is confirmed gone. the one stage i cant implement is "confirm the stub is gone" beyond ESRCH.. thats exactly what #14408 shows is unobservable from outside a wedged sandbox, the sentry is alive and every control RPC hangs.

on 30s being too aggressive, fair. the gate is stricter than it looks, fail-stop needs zero state change through the whole window with repeated interrupts already sent, and even a hard cfs-throttled stub gets scheduled every 100ms period so it escapes on the next wakeup. but its still a heuristic, so il decouple the fail-stop deadline from the 30s stuck-warning deadline and give it more headroom, say a few minutes like the watchdog's stuck-task default. can also put it behind a flag if maintainers prefer opt-in.

shim-side timeouts on Stats/Kill i agree with as damage control but its a different layer, il file a follow-up issue rather than grow this PR. one caveat there: a timed-out Kill returning an error to kubelet is the same DeadlineExceeded loop we already see, so the win is bounded shim memory, not recovery.

the --panic-signal tip is good, thats how you got the stacks manual kill never gave us. the TracebackAll in this PR is basically that, automated at the moment it matters.

does a longer fail-stop default address your concern, or do you think sentry termination should be opt-in?

does a longer fail-stop default address your concern, or do you think sentry termination should be opt-in?

Thanks, that addresses most of my concern.

I would prefer a longer, configurable fail-stop deadline that remains enabled by default, rather than making termination opt-in. A few minutes, similar to the watchdog’s stuck-task default, gives transient host pressure more time to recover while still preventing the multi-day failure and node-level memory growth we observed.

Instead of having this killSentry callback within a callback, how about just having sleepOnStateWithTimeout return a new err type, and have sleepOnState call call sighandling.KillItself along with the logging.

Wrote this before my req to change the KillItself, but point still stands, remove the callbacks.

Done. sleepOnStateWithTimeout now returns errStuckContext and sleepOnState does the traceback + kill, no callbacks.

Done, callbacks removed.

Useless comment

Done, removed.

This log is basically dead code after this change

Done, removed.

Lost in refactoring? This continue should still be here in case we got an interrupt.

Done, restored. The escalation paths also recheck the state so a context that recovered during the futex wait is never treated as stuck (a test caught exactly that race).

Return an error, do a TracebackAll in sleepOnStateWithTimeout in case we see it.

Done. interruptStub returns errNoStubThread / errStubThreadGone; the loop logs and continues on the first, escalates on the second.

Return an error, do a TracebackAll in sleepOnStateWithTimeout in case we see it.

Done, same as above.

I think calling KillItself here is an over-reaction. If there's one stuck subproc but the others are fine, we'd end up killing everything. Like in NotifyInterrupt, the first action should still be to kill the subprocess.

Done. Escalation now kills only the stuck subprocess through the same path NotifyInterrupt uses (mark dead, ContextStateUnexpectedDeath, kill syscall thread), extracted into killSubprocess and shared by both.

Please review comments, we don't need to redescribe the code in comment form.

Done, dropped them.

Ditto

Done.

Address review: sleepOnStateWithTimeout returns errStuckContext instead of taking a kill callback; sleepOnState does the TracebackAll and kills only the stuck subprocess via the same path NotifyInterrupt uses for a dead stub, instead of terminating the whole sentry. interruptStub returns typed errors and NotifyInterrupt shares killSubprocess. Restore the continue on futex wakeup and recheck state before escalating so a context that recovers during the wait is never killed. Raise the stuck deadline to 3 minutes, matching the watchdog stuck-task default. Drop redundant comments and the dead warning log.

@konstantin-s-bogom all comments addressed, pushed as abd26a0. escalation now kills only the stuck subprocess via the NotifyInterrupt path, no callbacks, typed errors, comments dropped. also raised the stuck deadline to 3 minutes to match the watchdog stuck-task default, per @tanyifeng's concern that 30s is too aggressive under transient host pressure. systrap_test passes 20/20 runs and runsc builds.

A dead stub is quasi-normal and mirrors NotifyInterrupt (warn + kill subprocess, no stack dump); only a stub unresponsive through repeated interrupts dumps all stacks. The no-stub-thread case tracebacks once and backs off to the checkup interval instead of warning every preempt tick.

github-actions Bot requested review from EtiennePerot and konstantin-s-bogom August 19, 2026 19:35

NahumLitvin mentioned this pull request Aug 25, 2026

systrap: sentry unresponsive with stuck contexts; pod wedged in Terminating, only SIGKILL of sentry tree recovers #14408

Open

NahumLitvin requested a review from nixprime as a code owner August 26, 2026 05:47

konstantin-s-bogom requested changes Aug 26, 2026

View reviewed changes

+              	timeout := unix.NsecToTimespec(contextPreemptTimeout.Nanoseconds())
+              	interruptsSent := 0
+              	deadline := time.Now().Add(stuckTimeout)
+              	failStop := func() error {

               	// shared memory between reads.
               	threadID := atomic.LoadUint32(&sc.shared.ThreadID)
               	if threadID == invalidThreadID {
-              		return

               	if !ok {
               		// This is either an invalidThreadID or another garbage value; either way we
               		// don't know which thread to interrupt; best we can do is mark the context.
-              		return

-              	timeout := unix.Timespec{
-              		Sec:  0,
-              		Nsec: contextPreemptTimeoutNsec,
+              	return sc.sleepOnStateWithTimeout(state, stuckContextTimeout, contextCheckupTimeout, sighandling.KillItself)

+              		// A single interrupt can be lost: #14405 captured a task goroutine
+              		// wedged forever after one missed SIGUSR1. Resend on every checkup
+              		// wakeup until the context recovers or the deadline expires, and
+              		// fail-stop only when repeated interrupts prove the stub is gone.

+              		// Dump all goroutine stacks first: once the sentry is gone there is
+              		// nothing left to debug, and stacks are unobtainable from outside a
+              		// wedged sandbox (control RPCs hang, see #14408).

+              			if interruptsSent > 0 {
+              				return failStop()
+              			}
               			log.Warningf("Systrap task goroutine has been waiting on ThreadContext.State futex too long. ThreadContext: %v", sc)

-              		if errno == 0 {
-              			continue
-              		}

+              		if err := killSentry(); err != nil {
+              			panic(fmt.Sprintf("failed to kill sentry with stuck systrap context: %v", err))
+              		}
+              		// KillItself doesn't return on success. This return keeps the path testable.

systrap: fail-stop sentry on stuck context by NahumLitvin · Pull Request #14201 · google/gvisor · GitHub

Uh oh!

systrap: fail-stop sentry on stuck context - #14201

systrap: fail-stop sentry on stuck context#14201NahumLitvin wants to merge 4 commits intogoogle:mastergoogle/gvisor:masterfrom NahumLitvin:fix/systrap-stuck-contextNahumLitvin/gvisor:fix/systrap-stuck-contextCopy head branch name to clipboard

Conversation

NahumLitvin commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NahumLitvin commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NahumLitvin commented Aug 25, 2026

Uh oh!

tanyifeng commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NahumLitvin commented Aug 26, 2026

Uh oh!

tanyifeng commented Aug 26, 2026

Uh oh!

NahumLitvin commented Aug 26, 2026

Uh oh!

tanyifeng commented Aug 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NahumLitvin commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

systrap: fail-stop sentry on stuck context#14201
NahumLitvin wants to merge 4 commits into
google:masterfrom
NahumLitvin:fix/systrap-stuck-context

NahumLitvin commented Aug 19, 2026 •
edited
Loading

NahumLitvin commented Aug 25, 2026 •
edited
Loading

tanyifeng commented Aug 26, 2026 •
edited
Loading