Control-plane read failures during sandbox SSH-session cleanup; delete acknowledgment can precede durable cleanup

Summary

OpenShell 0.0.109 intermittently fails control-plane reads while deleting sandbox-owned SSH-session records. Separately, sandbox delete can report success before durable sandbox/policy records have disappeared, so an immediately following workspace delete can fail.

These observations came from real Docker testing of the OpenClaw OpenShell integration, including concurrent file operations and commands in both mirror and remote modes. They are independent of the host-gateway policy fixture correction in openclaw/openclaw#130465.

Environment

• OpenShell CLI/gateway 0.0.109, upstream commit 8d67250a5d17348eb96c4fa46226b06d8041f2ba.
• Linux aarch64 inside an isolated privileged Docker container; nested Docker uses the vfs storage driver. No host repository, home, or Docker socket is mounted.
• One gateway, Docker compute driver, mTLS enabled, one managed Docker network, separate non-default test workspaces.
• File-backed SQLite on the container's overlay filesystem, ?mode=rwc; no journal, busy-timeout, pool, or schema override. The observed journal mode was delete; SQLx 0.8.6 defaults to a five-second busy timeout.
• The same gateway state/network/dependency versions were retained across controlled diagnostic runs. Each run started without task sandboxes/workspaces; the default workspace was retained.

A. Traced SSH-session cleanup write burst and failed reads

A full mirror-then-remote integration run passed all five foreground tests, including strict workspace deletion and final inventory checks. However, the gateway trace recorded 542 individual store.delete operations for object_type=ssh_session during the approximately 43-second remote cleanup window. These overlapped 22 slow SQL operations and six GetSandboxConfig/GetInferenceBundle INTERNAL responses. Their failed SELECTs returned zero rows and marked the Store span as ERROR after approximately five seconds; the affected RPCs took 5.1–9.9 seconds.

The competing write owner is visible in the parameterized SQL trace. In the pinned source, cleanup_sandbox_ssh_sessions reads up to 1,000 workspace sessions, decodes them, and awaits one DELETE per matching session. Each DELETE is a separate SQLx execution. This identifies the serial cleanup write burst; it does not yet establish the exact SQLite connection-level cause of every failed read.

The Store ERROR field is independently recorded on the nested store span: store_dispatch_traced! marks that span only for unexpected persistence errors, not for an ordinary missing row. GetInferenceBundle maps a missing sandbox to NOT_FOUND, and a missing inference route to Ok(None); persistence errors become INTERNAL. For example, the traced inference-route SELECT at 01:51:10.190813 UTC marked its own store.get_by_name span ERROR after 5.0965 seconds, followed by an INTERNAL RPC response. The trace does not capture the underlying SQLite error text for those six RPCs, so this report does not label them all SQLITE_BUSY. Only C below captured literal code 5.

The same run's fcntl/fsync trace showed repeated short process-wide locks: maximum observed PENDING interval 0.379 seconds, EXCLUSIVE 0.378 seconds, and SHARED 0.117 seconds. Individual sync calls were below 0.22 seconds. There was no single observed five-second kernel lock. Repeated-write starvation is a hypothesis, not proof of a leaked transaction; SQLite's same-process connection locking can also occur without a kernel call.

Source:

• OpenShell/crates/openshell-server/src/compute/mod.rs

  Lines 2850 to 2875 in 8d67250

  	.store
  	.list(SshSession::object_type(), workspace, 1000, 0)
  	.await
  	.map_err(|e| format!("list SSH sessions: {e}"))?;
  	for record in records {
  	if let Ok(session) = SshSession::decode(record.payload.as_slice())
  	&& session.sandbox_id == sandbox_id
  	{
  	self.store
  	.delete(SshSession::object_type(), session.object_id())
  	.await
  	.map_err(|e| format!("delete SSH session {}: {e}", session.object_id()))?;
  	}
  	}
  	Ok(())
  	}
  	async fn cleanup_stopped_sandbox_sessions(&self, sandbox: &Sandbox) -> Result<(), String> {
  	// Disconnect first so a store failure cannot leave the stopped
  	// sandbox reachable through an existing supervisor stream. Both
  	// operations are idempotent and are retried for durable Stopped
  	// records during explicit stop requests and startup recovery.
  	self.supervisor_sessions.disconnect(sandbox.object_id());
  	self.cleanup_sandbox_ssh_sessions(sandbox.object_id(), sandbox.object_workspace())

• OpenShell/crates/openshell-server/src/persistence/sqlite.rs

  Lines 366 to 380 in 8d67250

  	pub async fn delete(&self, object_type: &str, id: &str) -> PersistenceResult<bool> {
  	let result = sqlx::query(
  	r#"
  	DELETE FROM "objects"
  	WHERE "object_type" = ?1 AND "id" = ?2
  	"#,
  	)
  	.bind(object_type)
  	.bind(id)
  	.execute(&self.pool)
  	.await
  	.map_err(|e| map_db_error(&e))?;
  	Ok(result.rows_affected() > 0)
  	}

• https://github.com/launchbadge/sqlx/blob/v0.8.6/sqlx-core/src/logger.rs

• OpenShell/crates/openshell-server/src/persistence/mod.rs

  Lines 178 to 192 in 8d67250

  	Self::Sqlite(s) => s.$method($($arg),*).await,
  	}
  	};
  	}
  	/// [`store_dispatch`] for methods carrying a span, marking that span failed
  	/// unless the error is one the caller is expected to act on.
  	macro_rules! store_dispatch_traced {
  	($self:ident . $method:ident ( $($arg:expr),* )) => {{
  	let result = store_dispatch!($self.$method($($arg),*));
  	if let Err(err) = &result
  	&& !err.is_expected()
  	{
  	crate::otel_tracing::mark_error(&tracing::Span::current());
  	}

• OpenShell/crates/openshell-server/src/inference.rs

  Lines 67 to 91 in 8d67250

  	async fn get_inference_bundle(
  	&self,
  	request: Request<GetInferenceBundleRequest>,
  	) -> Result<Response<GetInferenceBundleResponse>, Status> {
  	let sandbox_id = authorize_inference_bundle(
  	request
  	.extensions()
  	.get::<crate::auth::principal::Principal>(),
  	)?;
  	let sandbox: Sandbox = self
  	.state
  	.store
  	.get_message::<Sandbox>(&sandbox_id)
  	.await
  	.map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?
  	.ok_or_else(|| Status::not_found(format!("sandbox '{sandbox_id}' not found")))?;
  	let workspace = sandbox.object_workspace();
  	resolve_inference_bundle_with_credentials(
  	self.state.store.as_ref(),
  	workspace,
  	Some(&self.state.credentials),
  	)
  	.await
  	.map(Response::new)
  	}

• OpenShell/crates/openshell-server/src/inference.rs

  Lines 1108 to 1128 in 8d67250

  	resolve_route_by_name_with_credentials(store, workspace, None, route_name).await
  	}
  	async fn resolve_route_by_name_with_credentials(
  	store: &Store,
  	workspace: &str,
  	credentials: Option<&crate::credentials::CredentialRuntime>,
  	route_name: &str,
  	) -> Result<Option<ResolvedRoute>, Status> {
  	let route = store
  	.get_message_by_name::<InferenceRoute>(workspace, route_name)
  	.await
  	.map_err(|e| Status::internal(format!("fetch route failed: {e}")))?;
  	let Some(route) = route else {
  	return Ok(None);
  	};
  	let Some(config) = route.config.as_ref() else {
  	return Ok(None);
  	};

B. Separate early deletion acknowledgement

A small CLI-only sequence created three sandboxes, exercised deny/allow networking and 32 serial SSH commands, generated 578 additional ssh-config files without executing them, then deleted the three sandboxes followed immediately by the workspace. Those extra config generations did not allocate sessions: the actual CreateSshSession RPC count remained 37. All three sandbox-delete commands returned exit 0 and printed that the sandbox was deleted. Workspace deletion then returned exit 1 because sandbox and sandbox-policy resources remained. The next all-workspaces sandbox inventory was empty; the workspace remained Terminating. There were no slow SQL warnings in this run.

The source matches this race: after the driver returns deleted=true, delete_sandbox_inner calls cleanup_local_state_if_sandbox_absent. That function clears local state only if the durable row is already absent; it returns success when the row still exists. Watcher reconciliation performs the durable cleanup separately. Workspace deletion, meanwhile, synchronously rejects remaining records.

Please clarify whether DeleteSandbox success is intended to mean compute removal only or completed durable cleanup, and which supported completion signal callers should use before deleting the workspace. The API proto and Go SDK docs currently just describe sandbox deletion.

Source:

• OpenShell/crates/openshell-server/src/compute/mod.rs

  Lines 1589 to 1607 in 8d67250

  	}
  	},
  	)
  	.await;
  	match result {
  	Ok(response) => {
  	let deleted = response.into_inner().deleted;
  	if deleted {
  	self.cleanup_local_state_if_sandbox_absent(&delete_guard, &target.sandbox_id)
  	.await?;
  	} else if !self
  	.remove_deleting_sandbox_record(&delete_guard, &target.sandbox_id)
  	.await
  	{
  	return Err(Status::internal(
  	"compute resource was absent, but gateway cleanup did not complete",
  	));
  	}

• OpenShell/crates/openshell-server/src/compute/mod.rs

  Lines 2910 to 2924 in 8d67250

  	async fn cleanup_local_state_if_sandbox_absent(
  	&self,
  	delete_guard: &SandboxLifecycleGuard,
  	sandbox_id: &str,
  	) -> Result<(), Status> {
  	let _guard = self.lock_global_for_lifecycle(delete_guard).await;
  	let record = self
  	.store
  	.get(Sandbox::object_type(), sandbox_id)
  	.await
  	.map_err(|err| Status::internal(format!("fetch sandbox failed: {err}")))?;
  	if record.is_none() {
  	self.cleanup_removed_sandbox_state(sandbox_id);
  	}
  	Ok(())

• OpenShell/crates/openshell-server/src/grpc/workspace.rs

  Lines 285 to 448 in 8d67250

  	}
  	pub(super) async fn handle_delete_workspace(
  	state: &Arc<ServerState>,
  	request: Request<DeleteWorkspaceRequest>,
  	) -> Result<Response<DeleteWorkspaceResponse>, Status> {
  	let name = request.into_inner().name;
  	if name.is_empty() {
  	return Err(Status::invalid_argument("name is required"));
  	}
  	if name == DEFAULT_WORKSPACE_NAME {
  	return Err(Status::failed_precondition(
  	"the default workspace cannot be deleted",
  	));
  	}
  	let ws: Workspace = state
  	.store
  	.get_message_by_name("", &name)
  	.await
  	.map_err(|e| Status::internal(format!("fetch workspace failed: {e}")))?
  	.ok_or_else(|| Status::not_found(format!("workspace '{name}' not found")))?;
  	let ws_id = ws
  	.metadata
  	.as_ref()
  	.map(|m| m.id.clone())
  	.unwrap_or_default();
  	let already_terminating = ws
  	.metadata
  	.as_ref()
  	.is_some_and(|m| m.deletion_timestamp_ms != 0);
  	// Track the resource_version so the final delete targets exactly this
  	// workspace instance (prevents ABA if a same-name workspace is recreated
  	// between the blocker scan and the delete).
  	let mut delete_version = ws.metadata.as_ref().map_or(0, |m| m.resource_version);
  	if !already_terminating {
  	let cas_result = state
  	.store
  	.update_message_cas::<Workspace, _>(&ws_id, 0, |w| {
  	let now_ms = current_time_ms();
  	if let Some(meta) = w.metadata.as_mut() {
  	meta.deletion_timestamp_ms = now_ms;
  	}
  	w.status = Some(WorkspaceStatus {
  	phase: WorkspacePhase::Terminating.into(),
  	});
  	})
  	.await;
  	match cas_result {
  	Ok(updated) => {
  	delete_version = updated.metadata.as_ref().map_or(0, |m| m.resource_version);
  	}
  	Err(e) => {
  	if matches!(e, crate::persistence::PersistenceError::Conflict { .. }) {
  	let refreshed: Option<Workspace> = state
  	.store
  	.get_message_by_name("", &name)
  	.await
  	.map_err(|e| Status::internal(format!("workspace re-fetch failed: {e}")))?;
  	let refreshed = refreshed.ok_or_else(|| {
  	Status::not_found(format!("workspace '{name}' not found"))
  	})?;
  	let now_terminating = refreshed
  	.metadata
  	.as_ref()
  	.is_some_and(|m| m.deletion_timestamp_ms != 0);
  	if !now_terminating {
  	return Err(Status::aborted(
  	"workspace was concurrently modified, please retry",
  	));
  	}
  	delete_version = refreshed
  	.metadata
  	.as_ref()
  	.map_or(0, |m| m.resource_version);
  	} else {
  	return Err(Status::internal(format!(
  	"mark workspace terminating failed: {e}"
  	)));
  	}
  	}
  	}
  	}
  	// The workspace is now Terminating — concurrent create-path operations
  	// will be rejected by resolve_workspace + ensure_active.
  	let mut blocking = Vec::new();
  	for (object_type, label) in [
  	(Sandbox::object_type(), "sandbox"),
  	(Provider::object_type(), "provider"),
  	(StoredProviderProfile::object_type(), "provider profile"),
  	(ServiceEndpoint::object_type(), "service"),
  	(SshSession::object_type(), "ssh session"),
  	(
  	super::policy::SANDBOX_SETTINGS_OBJECT_TYPE,
  	"sandbox settings",
  	),
  	(POLICY_OBJECT_TYPE, "sandbox policy"),
  	(DRAFT_CHUNK_OBJECT_TYPE, "draft policy chunk"),
  	(
  	StoredProviderCredentialRefreshState::object_type(),
  	"credential refresh state",
  	),
  	] {
  	let records = state
  	.store
  	.list(object_type, &name, 1, 0)
  	.await
  	.map_err(|e| Status::internal(format!("resource check failed: {e}")))?;
  	if !records.is_empty() {
  	blocking.push(label);
  	}
  	}
  	if !blocking.is_empty() {
  	return Err(Status::failed_precondition(format!(
  	"workspace '{}' still contains resources: {}",
  	name,
  	blocking.join(", ")
  	)));
  	}
  	// Cascade-delete non-blocking resources before the final CAS delete.
  	// This is safe without a transaction: the workspace is Terminating, so
  	// ensure_active rejects new resource creation. If delete_if conflicts
  	// below, the retry will find no routes/members to delete and succeed.
  	state
  	.store
  	.delete_all_in_workspace(InferenceRoute::object_type(), &name)
  	.await
  	.map_err(|e| Status::internal(format!("delete inference routes failed: {e}")))?;
  	state
  	.store
  	.delete_all_in_workspace(WorkspaceMember::object_type(), &name)
  	.await
  	.map_err(|e| Status::internal(format!("delete workspace members failed: {e}")))?;
  	// Keep the terminating workspace durable until platform cleanup has been
  	// accepted. A failed cleanup can then be retried through this same path.
  	state.compute.delete_workspace(&name).await.map_err(|e| {
  	Status::new(
  	e.code(),
  	format!("delete workspace platform resources failed: {e}"),
  	)
  	})?;
  	let deleted = state
  	.store
  	.delete_if(Workspace::object_type(), &ws_id, delete_version)
  	.await
  	.map_err(|e| {
  	if matches!(e, crate::persistence::PersistenceError::Conflict { .. }) {
  	Status::aborted("workspace was concurrently modified, please retry")
  	} else {
  	Status::internal(format!("delete workspace failed: {e}"))
  	}
  	})?;
  	Ok(Response::new(DeleteWorkspaceResponse { deleted }))
  	}

C. Original foreground cleanup failure, not syscall-traced

An earlier full integration run completed both stress assertion sets, but failed both real cases during strict cleanup. Mirror workspace deletion returned:

fetch workspace failed: database error: error returned from database: (code: 5) database is locked

The remote workspace deletion reported remaining sandbox, SSH-session, and policy resources. Before stopping the gateway, inventory showed two Ready sandboxes and their Active/Terminating test workspaces. Their sandbox deletions had failed before the workspace deletion. This run captured gateway/CLI logs but not syscall or parameterized SQL DEBUG traces, so it must not be presented as the same traced event as A.

Controlled reductions and limits

• A short CLI-only flow with 37 actual CreateSshSession RPCs cleaned up successfully.
• Adding 90 seconds of idle time also succeeded.
• Adding enough actual SSH executions to reach 615 session RPCs also succeeded, without slow queries. Session volume alone did not reproduce the database failure.
• Generating extra ssh-config files does not create sessions; only actual SSH ProxyCommand execution does. A config-generation-only run exposed B, not a session-volume failure.
• An exact mirror-only integration run passed strict cleanup but still showed six slow database reads.
• The full traced run in A had 128 stress workflows, 64 command executions, 16 intentional command failures, and 144 verified data files plus the mirror count/ledger. Its task sandboxes/workspaces, nested containers, and temporary directories were absent after automatic cleanup; the default workspace remained.

No WAL, busy-timeout, pool, schema, retry, or production-policy changes were used to make these tests pass. Raw databases, SSH configurations, session tokens, and unredacted logs will not be attached.

Current-source check

The relevant compute functions are byte-identical in v0.0.115: delete_sandbox_inner, cleanup_local_state_if_sandbox_absent, cleanup_sandbox_ssh_sessions, cleanup_sandbox_owned_records, and apply_deleted_locked. SQLite connection setup is also unchanged. This is source evidence only; v0.0.115 has not been live-tested for this report.

Exact test and diagnostic command

The reviewed test candidate is now published at OpenClaw commit 6953ce5; the full test and stress helper include the strict cleanup oracle. The held PR evidence records both the original failure and later traced pass. This is a full integration reproduction, not a reduced deterministic reproducer.

In an isolated Linux container with Docker, Node, pnpm, SSH, and the pinned OpenShell CLI/gateway installed, the tested setup uses a task-owned nested dockerd --storage-driver=vfs, a dedicated Docker network, and an explicit mTLS gateway. Generate certificates with openshell-gateway generate-certs; use the same OPENSHELL_LOCAL_TLS_DIR for certificate generation, the server, and openshell gateway add --local. The gateway config explicitly enables mtls_auth, uses that network and namespace for the Docker driver, points its guest endpoint at https://host.openshell.internal:17680, and supplies the generated guest CA/client certificate/key. The supervisor image is pinned to ghcr.io/nvidia/openshell/supervisor:8d67250a5d17348eb96c4fa46226b06d8041f2ba. No installer-managed gateway is running alongside it.

The source was installed with frozen dependencies and built before running the focused E2E:

git checkout 6953ce5cd4f77df0eef5c9fa8ebfe2b003f43caf
pnpm install --frozen-lockfile
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build

# Start only the isolated gateway with its existing generated TLS files/config.
# Set RUST_LOG=info,sqlx::query=debug on that gateway process for diagnostic SQL.
# Register/select it using the same isolated XDG_CONFIG_HOME and TLS directory.
openshell gateway select cleanup-proof
openshell --gateway cleanup-proof whoami

OPENCLAW_E2E_OPENSHELL_CONFIG_HOME="$XDG_CONFIG_HOME" \
OPENCLAW_E2E_WORKERS=1 OPENCLAW_VITEST_MAX_WORKERS=1 \
OPENCLAW_E2E_SKIP_BUILD=1 OPENCLAW_E2E_VERBOSE=1 \
pnpm test:e2e:openshell

# Before stopping the gateway, distinguish task resources from the default workspace.
openshell --gateway cleanup-proof sandbox list --all-workspaces -o json
openshell --gateway cleanup-proof workspace list -o json
docker ps -a

OPENCLAW_E2E_SKIP_BUILD=1 uses the successful preceding build; it does not skip either real test. Both tests must run. Capture the test exit and automatic inventory before any explicit recovery or environment teardown. For lock diagnostics we attached strace -f -ttt -T -yy -e trace=fcntl,fsync,fdatasync to only the gateway process; no read/write payloads or bound SQL values were collected.