Description

A Kubernetes cluster in Alert can now change its size, and have worker nodes removed with nodeids. Previously scaleKubernetesCluster refused any request in Alert. So one stopped node VM blocked scaling, and blocked the CloudStack cluster-autoscaler, which scales through the same API. It also blocked removing that stopped node.

Fixes #14603

What changes

• State check (KubernetesClusterManagerImpl.validateKubernetesClusterStateForScale, moved out of validateKubernetesClusterScaleParameters). In Alert it accepts a size change, or removing worker nodes by nodeids. It refuses:
  • a service offering or autoscaling change, since both act on every node, including the one that is down. Sending the offerings and autoscaling setting the cluster already has is accepted;
  • a request that changes neither the size nor the nodes. Such a request would do nothing and then move the cluster from Alert to Running;
  • removing control, etcd or external nodes by nodeids. Scaling never acts on external nodes; they are removed with removeVirtualMachinesFromKubernetesCluster;
  • any request while the first control node VM isn't running. Scaling reaches the cluster over SSH to that node;
  • any request while the cluster has more or fewer VMs than nodes, for example VMs left by a failed scale-up, which a scale-up would only add to.
  • any request while a control or etcd node VM is down. etcd may then have lost quorum, and new nodes can't join, so the control plane has to be recovered first;
  • any request while none of the cluster's worker VMs is down (see "Down" below), leaving out external nodes, which scaling doesn't act on. The cluster is then in Alert for another reason that scaling won't fix, such as a failed start or upgrade, or a VM on a disconnected host.
• What the worker gets in Alert:
  • No offerings, so it only changes the size, and can't apply the cluster offering to a node type whose VMs use another one.
  • The cluster record that was validated is passed on as is, rather than read again.
  • The worker fails the request if the cluster's state has changed since it was validated, for example to Alert, or because another operation has started.
• State machine (KubernetesCluster): Alert -> Scaling on ScaleUpRequested and ScaleDownRequested. Scaling -> Running on success and Scaling -> Alert on failure are unchanged, so the status scanner puts the cluster back in Alert if a node is still down.
• Scale-up (KubernetesClusterScaleWorker):
  • After provisioning, the worker waits for the nodes whose VMs aren't down to be Ready. Before, it waited for every node, and a stopped node, which is never Ready, made every scale-up time out.
  • The new nodes must all be Ready. The expected count is read again on every check, in case a VM is started or stopped meanwhile. This goes through a new KubernetesClusterUtil.validateKubernetesClusterReadyNodesCount overload; the existing signature behaves as before.
  • Once the count matches, the worker also waits for each new node to be Ready by name (isKubernetesClusterNodeReady, as the upgrade worker does), within the same scale timeout, before it detaches the binaries ISO. Kubernetes keeps reporting the node of a VM that was just stopped as Ready for 40-50 s (node-monitor-grace-period), and in that window it can make up the count before a new node has joined. The ISO was then detached and the new node could never join.
  • The capacity check plans only the new VMs in Alert, as it does in Running. Before, it planned capacity for the whole cluster again.
• Scale-down by size: workers whose VMs are down are removed first. Otherwise the order is unchanged.
• Removing a node whose VM is down: the worker no longer runs kubectl drain, and deletes the Kubernetes node object directly.
  • The drain waits for pod deletions that a NotReady node can never confirm, so it failed, and the scale-down with it.
  • A down VM runs no pods. Once its node object is deleted, Kubernetes garbage-collects the pods and their controllers recreate them elsewhere.
• "Down" means Stopped, Shutdown, Error, Destroyed or Expunging. Migrating, Unknown (a disconnected host) and the other states may still be running the VM, so those nodes are treated as before.
• UI: the Scale action is shown for a cluster in Alert that doesn't autoscale. The dialog then shows only the size field, and sends only the size. An autoscaling cluster in Alert is scaled by the cluster-autoscaler itself. The per-node delete button in the cluster's instances tab is unchanged, so removing a specific node in Alert is done through the API (nodeids) for now.

Running, Created and Stopped clusters behave as before. The exception is a cluster that is still Running while one of its VMs is already down, before the status scanner has moved it to Alert. There, the down node is now removed first, isn't drained, and isn't expected to become Ready.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

• Unit tests, all in plugins/integrations/kubernetes-service (mvn -pl plugins/integrations/kubernetes-service -am test -Dtest='KubernetesCluster*Test'), pass. Checkstyle on api and the plugin is clean.
• New tests in KubernetesClusterManagerImplTest:
  • accepted in Alert: a size change; nodeids for a worker; a size change sent with the cluster's current offerings and autoscaling setting;
  • refused in Alert: a cluster or node-type offering change; an autoscaling change, including a different minsize/maxsize; a request with no size or node change; a request with the current size; removing a control node; the first control node VM stopped; more VMs than nodes; no worker VM down (all running, or only Unknown); an etcd VM down; only an external node down;
  • an offering change is still accepted in Running, and Upgrading is still refused.
• New tests in KubernetesClusterScaleWorkerTest: a stopped worker is picked first for a scale-down by size; the down-node count leaves out etcd nodes and newly provisioned VMs; which VM states count as down; after a scale-up, the ready-node count leaves out down nodes, each new node is then waited for by its lower-case name, and the scale-up fails if the count matches while a new node isn't Ready.
• Live test: scale-up, scale-down and nodeids removal were run against a real CKS cluster in Alert (see the lab report in the comments). The first run found a scale-up reporting success before the new node joined, which the second commit fixes. The UI change was not tested in a browser.

How did you try to break this feature and the system with this change?

• Other states: for Created, Running and Stopped clusters, the state check, the offering map and the existing ready-node check are unchanged. The only difference is the down-VM handling for a Running cluster with a down VM, as described above.

• Restarts during a scale-up: the expected Ready count is read again on every check. A stopped VM that HA or a user starts again is then expected to rejoin, and the scale-up can't report success before the new nodes are Ready.

• Hosts disconnected or VMs migrating: Unknown and Migrating don't count as down, so those nodes are still drained, and still expected to be Ready.

• Concurrent requests: a second scale request is refused at validation once the first has moved the cluster to Scaling. If both are validated before that, the second now fails when its worker starts, because the state has changed. Before, it ran alongside the first. The check and the transition are still separate steps, so a very narrow window remains.

• Not covered here:

  • Removing control nodes by nodeids in Running records the wrong worker count. That is unchanged by this PR, and refused in Alert.
  • removeKubernetesClusterNode increments its retry counter twice per pass, so it makes fewer attempts than intended.
  • A request that sends only the cluster offering in Running still re-applies it to node types whose VMs use another offering.
  • A worker count change without a new offering records the cluster's cores and memory using the cluster offering, not the worker offering, which new worker VMs get. A size change in Alert is always such a change, as is a size-only request in any state (the cluster-autoscaler's). Where the two offerings differ, the recorded capacity is wrong. This is a known follow-up.
  • A Running VM whose node never joined, or is NotReady, is still drained, and the drain fails. Scaling up a cluster with such a node still times out, as it does in Running.

  These are existing issues and are left for separate changes.