Summary

KubernetesClientFactory hard-wires fabric8's VertxHttpClientFactory. fabric8's
Vert.x client ignores Config#websocketPingInterval (only the OkHttp client applies
it), so the operator's watch WebSockets carry no traffic while the watched
resources are quiet. On a managed Kubernetes service whose API endpoint closes
connections that are idle for 50 seconds, every informer watch is closed every
~50s. The informers go UNHEALTHY, events are missed, and new applications fail
with DriverStartTimedOut even though the driver pod was created and started.

Environment

• spark-kubernetes-operator 0.8.0 (fabric8 7.6.1). Also reproduced with 0.8.0
  rebuilt against fabric8 7.8.0, the version 1.0.0 ships.
• Managed Kubernetes v1.30, API server reached through a provider load balancer
  that closes connections idle for 50s.
• spark-operator/build.gradle (0.8.0 and 1.0.0):
  implementation(libs.kubernetes.httpclient.vertx)

Symptoms

Operator log, repeating every ~50s for all informers at once:

DEBUG i.f.k.c.d.i.AbstractWatchManager Watching https://10.96.0.1:443/api/v1/pods?allowWatchBookmarks=true&labelSelector=spark.operator%2Fname%3Dspark-kubernetes-operator&resourceVersion=...&timeoutSeconds=600&watch=true...
DEBUG i.f.k.c.d.i.WatcherWebSocketListener WebSocket successfully opened
... 50 seconds, no events ...
DEBUG i.f.k.c.d.i.AbstractWatchManager Watch connection error, will reconnect if possible
java.io.IOException: io.vertx.core.http.HttpClosedException: Connection was closed
INFO  i.f.k.c.d.i.AbstractWatchManager Watch connection error received 335 times without progress, will reconnect if possible
ERROR o.a.s.k.o.p.HealthProbe Controller: sparkappreconciler, Event Source: ..., Informer: UNHEALTHY is in JOSDK_ALL_NAMESPACES, not a healthy state

The resourceVersion in the reconnect URL never advances: each watch lives 50s
without receiving a single event or bookmark (the API server sends bookmarks
roughly once a minute, i.e. later than the 50s idle cutoff).

Effect on applications:

KubernetesClientTimeoutException: Timed out waiting for [60000] milliseconds for [Pod] with name:[<app>-0-driver] in namespace [<ns>].
ERROR ... SparkAppReconciler Failed attempt: 0, last attempt: false
WARN  ... AppCleanUpStep Application is configured to restart, resources created in current attempt would be force released.

and eventually the application ends in DriverStartTimedOut. The failure is
intermittent: it depends on whether the driver pod's events arrive while a watch
is connected.

Root cause

Measured from a pod on the same node, watching a ConfigMap that does not change:

So the connection is dropped only when idle, and a client-side WebSocket ping
keeps it open. fabric8 defaults websocketPingInterval to 30s, but:

• httpclient-okhttp applies it (OkHttpClientFactory: pingInterval(...)).
• httpclient-vertx does not reference it at all (checked 7.6.1, 7.8.0 and main).
• httpclient-jdk documents it as unused.

Because the operator constructs VertxHttpClientFactory explicitly, users cannot
switch to a client that sends pings without patching the operator.

Suggested fixes

Any of these would resolve it:

1. Use (or allow configuring) the OkHttp HTTP client, which honours
   websocketPingInterval.
2. Keep Vert.x but send periodic WebSocket pings on watch connections (or get this
   fixed in fabric8's Vert.x client, see below).
3. Make the HTTP client factory pluggable so operators on such platforms can pick
   one.

Separately, the liveness probe does not restart the operator while informers stay
UNHEALTHY, so the operator can run for hours with a stale cache.

Related

• StandardHttpClient.shouldRetry() does not retry on Vert.x HttpClosedException fabric8io/kubernetes-client#7450: Vert.x HttpClosedException was not retried
  (closed 2026-03-20). With 7.8.0 the watches do reconnect, but the idle drops
  continue.
• A matching fabric8 issue for the missing ping support in httpclient-vertx
  should be filed alongside this one.