What is this PR for?

After restarting an interpreter from the setting page there is no way to ask whether it is reachable. The badge on that page is the dependency download status (InterpreterSetting.Status), which the code that starts and stops processes never touches, so it stays green whether the process is gone, stuck, or was never started. The only remaining way to find out is to run a paragraph, and for Spark that means the check itself takes cluster resources.

This adds a probe that a user asks for explicitly:

POST /api/interpreter/setting/{settingId}/healthcheck

{
  "settingId": "2CH2VMY7B",
  "settingName": "spark",
  "groups": [
    { "groupId": "2CH2VMY7B-shared_process", "healthy": true,  "reason": "OK",
      "alive": true, "running": true,  "probeTookMs": 3 },
    { "groupId": "2CH2VMY7B-user2",         "healthy": false, "reason": "NOT_REACHABLE",
      "alive": true, "running": false, "probeTookMs": 1004 },
    { "groupId": "2CH2VMY7B-user3",         "healthy": false, "reason": "PROBE_TIMEOUT",
      "probeTookMs": 3000 }
  ]
}

Review points:

• The deadline has to come from the caller. isRunning() takes no timeout, and its cost is not uniform: local and yarn read memory, docker opens a socket with a hardcoded one second timeout, and the k8s launcher asks the kube-apiserver through a client built with library defaults (10s per request, up to 10 retries with backoff). The thrift path is worse, since the socket is created without a timeout and the call is retried three times. So the probes run on another thread, and the request stops waiting after a budget shared by the whole setting. That budget is a 3 second constant. The number itself is arbitrary, so please suggest a more meaningful one and I will change it.
• Bounding the response does not reclaim the probe. Cancelling a future only interrupts, and neither a blocking socket connect nor an HTTP client call ends on an interrupt, so a probe thread can stay busy after its result was given up on. The pool is therefore limited to 4 daemon threads. Daemon matters because normal shutdown does not call System.exit, so a stuck non daemon thread would delay it.
• It never starts an interpreter. Restart only closes the processes and the next use creates them again, so a setting with nothing running is reported with the reason NOT_RUNNING rather than as a failure. The frontend sub-tasks need that distinction to render something other than a red badge.
• One setting can own many processes. An isolated setting has a group per user or note, so the response carries an entry per group. Probes are submitted together and collected against the one deadline, so an interpreter that does not answer costs the request its budget once instead of delaying the groups that would have answered right away.
• Two places where this differs from the ticket, both deliberate. The ticket names isAlive, but the interface documents isRunning as "the interpreter can communicate with server" while isAlive only says a process exists, so healthy follows isRunning and both values are reported. The ticket also names two outcomes, healthy and unhealthy, which the response keeps as the healthy flag; the reason next to it is what lets a caller tell a setting that was never started from one that timed out. Happy to reduce it if you prefer the literal reading.
• Known limit. For the local and yarn launchers, isRunning() reads in process state, so an interpreter that is up but no longer answering is still reported healthy there. Only docker, k8s, and existing process check from outside. A thrift level ping would answer that properly, but it needs a schema change and has no bound of its own, so it is left out.

Out of scope, on purpose: GET /api/session calls isRunning() per session on a timer and has the same cost problem, but fixing it is a separate change.

What type of PR is it?

Feature

Todos

• - Add InterpreterHealthCheck, the response model, built through factory methods so healthy cannot disagree with reason
• - Add InterpreterHealthChecker, which probes the groups of a setting under one deadline
• - Add the POST /api/interpreter/setting/{settingId}/healthcheck endpoint with the note based authorization that restart already uses
• - Add unit and REST tests
• - Document the endpoint in docs/usage/rest_api/interpreter.md

What is the Jira issue?

• ZEPPELIN-6576

How should this be tested?

Included tests, all passing on current master:

export JAVA_HOME=<your JDK 11 home>
./mvnw package -pl zeppelin-server --am \
  -Dtest=InterpreterHealthCheckerTest,InterpreterRestApiTest -DfailIfNoTests=false

• InterpreterHealthCheckerTest, 7 tests. Covers the deadline, a slow group not hiding a fast one, group order, not probing a launching group, not starting an interpreter to answer, and a single remote call in the healthy case. (7 passed, 0 failed)
• InterpreterRestApiTest, 2 added. A health check of an unused setting returns NOT_RUNNING with no groups and leaves the setting without one, and an unknown setting returns 404. (13 passed, 0 failed)

License check: ./mvnw clean org.apache.rat:apache-rat-plugin:check -Prat -pl zeppelin-server reports 0 unapproved.

Manually, on a running server:

curl -u <admin> -X POST http://localhost:8080/api/interpreter/setting/<settingId>/healthcheck

Right after a restart this answers NOT_RUNNING without starting anything. Run a paragraph first and the same call reports the group as healthy.

Screenshots (if appropriate)

N/A, REST only.

Questions:

• Does the license files need to update? No. The four new files carry the ASF header and the RAT check reports no unapproved files.
• Is there breaking changes for older versions? No. The endpoint is new and nothing existing changes behavior.
• Does this needs documentation? Yes, and it is included in docs/usage/rest_api/interpreter.md.