Description

On KVM, growing a running VM's ROOT volume by giving it a larger disk offering was refused with "Cannot resize ROOT volume [...] when VM is not on Stopped State", although a size-only resizeVolume grows the same volume in place.

Fixes #14601

Why it breaks

• The size-only path already works live on KVM. resizeVolume id=<root> size=<n> reaches orchestrateResizeVolume, which handles a running KVM VM (it checks that the host is Up and Enabled). The agent grows the disk with virsh blockresize (LibvirtResizeVolumeCommandWrapper, resizevolume.sh).
• The disk-offering paths refuse it. resizeVolume with diskofferingid and changeOfferingForVolume both call checkIfVolumeIsRootAndVmIsRunning. So does scaleVirtualMachine of a running VM when allow.diskoffering.change.during.scale.vm is on, through changeOfferingForVolume. That refuses any size change of a ROOT volume unless the VM is Stopped, on every hypervisor. It was added in Allow to configure root disk size via Service Offering (diskoffering of type Service). #4341, when ROOT volumes were first allowed to change disk offering, and was never relaxed for KVM.
• When the ROOT volume's offering fixes its size, the size-only path is refused as well (isNotPossibleToResize), so the only remaining way to grow the root disk was to stop the VM.

How it is fixed

checkIfVolumeIsRootAndVmIsRunning now lets the change through when all of these hold:

• the VM's hypervisor is KVM;
• it is a user VM (orchestrateResizeVolume only looks up the running VM and its host for user VMs);
• the VM is Running;
• the new size is larger than the current one.

Everything else is refused as before:

• shrinking a ROOT volume of a running VM;
• other hypervisors;
• system VMs;
• VMs in other non-Stopped states (Starting, Migrating, ...).

Nothing below the check changes. The resize then takes the same route as a size-only resize of a running VM: the same job queue, orchestrateResizeVolume and its host checks, and the same agent command. The existing checks that apply to every running KVM VM still apply, for example a managed storage pool still refuses a resize while the VM is running (validateVolumeResizeWithSize).

Not in scope / existing behaviour now reachable for ROOT

• With allow.diskoffering.change.during.scale.vm on, scaling a running KVM VM to an offering with a larger root disk now grows the root disk instead of failing. The setting's description already says it covers running VMs. The root disk is changed before CPU and memory are reconfigured, which is the existing order in upgradeRunningVirtualMachine.
• Changing the offering of a running VM's volume updates IOPS / bytes-rate limits in the database, but the running domain keeps its current iotune until the VM is restarted. This is how a data disk's offering change already behaves.

Growing the partition and filesystem inside the guest (growpart, resize2fs) is up to the guest, as it already is for a size-only resize. cloud-init only does it at boot.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

• New unit tests in VolumeApiServiceImplTest for checkIfVolumeIsRootAndVmIsRunning:
  • growing a ROOT volume of a Running KVM user VM is allowed;
  • each of these is refused: shrinking it, the same grow on a Running VMware VM, on a Running KVM virtual router, and on a KVM VM that is Migrating;
  • a Stopped VM and a data disk are unaffected.
• The whole VolumeApiServiceImplTest class passes (152 tests), with checkstyle, on the 4.22 branch.
• Not tested on a live system. I have not run this against a KVM host. The live grow itself is not new code: it is the path a size-only resizeVolume of a running KVM VM already takes. That was confirmed by reading the code, not by running it.

How did you try to break this feature and the system with this change?

• Shrink on a running KVM VM: still refused, by this check, before anything reaches the agent.
• Other hypervisors: the new condition requires KVM, so XenServer, VMware and the rest behave exactly as before. VMware's own ROOT-resize checks (resizeVolumeInternal, validateVolumeReadyStateAndHypervisorChecks) are unchanged.
• Transitional VM states: only Running is let through. Starting, Stopping and Migrating are refused as before, and the orchestration's host checks only cover Running.
• Storage that can't be resized live: the size-only path already lets the same request through on every pool type for a running KVM VM. Managed pools other than PowerFlex and FiberChannel are refused by validateVolumeResizeWithSize, which both paths call.
• Offering change that also needs a volume migration (different storage tags): unchanged. It goes through migrateVolume exactly as it does for a data disk of a running VM today, with the same restrictions.