Live scaling for VMs with fixed service offerings on KVM by bernardodemarco · Pull Request #12975 · apache/cloudstack · GitHub

Description

Apache CloudStack currently supports live scaling only for VMs with custom constrained and custom unconstrained compute offerings. For VMs with fixed compute offerings, users must stop the VM, change its service offering through the scaleVirtualMachine API and then start it again. Depending on the criticality of the applications running on the VM, the downtime caused by this process is highly undesirable.

Therefore, this PR introduces support for live scaling of VMs with fixed service offerings when using the KVM hypervisor in Apache CloudStack. The feature specification is available at #12908.

Fixes #12908 and fixes #6865

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Test environment setup

Created multiple service offerings, including fixed and non-fixed offerings, with and without CPU cap enabled, with and without the dynamic scaling feature enabled:

Active service offerings

Enabled the global setting enable.dynamic.scale.vm.

Definition of the domain XML of the guest VMs

Verified that:

When the enable.dynamic.scale.vm global setting value is false, the domain XML of guest VMs is not prepared for live scaling;
When the Dynamically scalable property of service offerings is false, the domain XML of guest VMs is not prepared for live scaling;
When the Dynamically scalable property of templates is false, the domain XML of guest VMs is not prepared for live scaling;
When the Dynamically scalable property of VMs is false, the domain XML of guest VMs is not prepared for live scaling.

Live scaling

Verified the possibility of live scaling VMs from and to different types of service offerings;
Verified that, when suitable, CPU quota is applied on the fly;
Verified that, when suitable, CPU quota is removed on the fly;
Verified that, when suitable, CPU quota is updated on the fly.

kvm.cpu.dynamic.scaling.capacity and kvm.memory.dynamic.scaling.capacity settings

Verified that when the kvm.cpu.dynamic.scaling.capacity and kvm.memory.dynamic.scaling.capacity settings are equal to or less than 0, the host maximum capacity of CPU and memory is considered to define the upper limits for live scaling.
Verified that when the kvm.cpu.dynamic.scaling.capacity and kvm.memory.dynamic.scaling.capacity settings are greater than 0, the values of the settings are considered to define the upper limits for live scaling.

auto.migrate.vm.on.live.scale.insufficient.capacity setting

Verified that when the auto.migrate.vm.on.live.scale.insufficient.capacity setting is true and the current host lacks sufficient compute capacity to live scale the instance, the VM is automatically migrated to a suitable host;
Verified that when the auto.migrate.vm.on.live.scale.insufficient.capacity setting is false and the current host lacks sufficient compute capacity to live scale the instance, the VM is not migrated to a suitable host and an error message is returned.

Codecov Report

❌ Patch coverage is 72.16495% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.09%. Comparing base (5893ba5) to head (184c46a).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	0.00%	20 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	0.00%	17 Missing ⚠️
.../main/java/com/cloud/agent/api/ScaleVmCommand.java	0.00%	5 Missing ⚠️
.../java/com/cloud/agent/api/to/VirtualMachineTO.java	20.00%	4 Missing ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	94.59%	3 Missing and 1 partial ⚠️
...er/src/main/java/com/cloud/hypervisor/KVMGuru.java	96.61%	0 Missing and 2 partials ⚠️
...ypervisor/kvm/resource/LibvirtDomainXMLParser.java	0.00%	1 Missing ⚠️
...om/cloud/hypervisor/kvm/resource/LibvirtVMDef.java	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #12975   +/-   ##
=========================================
  Coverage     18.09%   18.09%           
- Complexity    16723    16732    +9     
=========================================
  Files          6037     6037           
  Lines        542580   542656   +76     
  Branches      66427    66436    +9     
=========================================
+ Hits          98155    98191   +36     
- Misses       433399   433443   +44     
+ Partials      11026    11022    -4

Flag	Coverage Δ
uitests	3.51% <ø> (+<0.01%)	⬆️
unittests	19.25% <72.16%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan package

@blueorangutan test

@blueorangutan package

@bernardodemarco , this seems like a heavty change/fix. Do you think it makes sense to write an integration test for this?

Hello, @DaanHoogland

Do you think it makes sense to write an integration test for this?

I've checked the existing integration tests in the codebase and identified the test_17_running_vm_scaleup test in test_escalations_instances.py, which already covers the VM live scaling workflow. However, since this feature was not previously available for KVM, the test included a constraint to skip execution when the active hypervisor is KVM. I removed this constraint, so we can now observe how the test behaves.

I believe that this integration test is sufficient. Most of the changes in the PR are on the Agent side and are well covered by unit tests.

I will try to do it later

@erikbocks, thanks!

Meanwhile, could you resolve the conflict?

Sure, conflicts resolved!

@blueorangutan package

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17846

LGTM. Tested locally and everything worked as proposed. Below are described the test scenarios:

Test	Behavior	Expected result
Disabled enable.dynamic.scale.vm and deploy a VM	The VM's XML definition was not ready for scaling.	Yes
Enabled the enable.dynamic.scale.vm configuration and deploy a VM	The VM's XML was ready for dynamic scaling.	Yes
Scaled a fixed offering VM to another offering with more CPU cores	The VM's domain XML was adjusted to the new offering's CPU amount	Yes
Scaled a fixed offering VM to another offering with more memory	The VM's domain XMl was adjusted to the new offering's memory amount	Yes
Scaled a fixed offering VM to another offering with more CPU MHz	The VM's domain XML was adjusted to the new offering's period	Yes
Tried to scale a VM to a offering with CPU cores above kvm.cpu.dynamic.scaling.capacity	An error was thrown. The logs informed that the offering's CPU cores was above permitted	Yes
Tried to scale a VM to a offering with memory above kvm.memory.dynamic.scaling.capacity	An error was thrown. The logs informed that the offering's memory was above permitted	Yes
Tried to scale a VM to a offering with more capacity than the current host when auto.migrate.vm.on.live.scale.insufficient.capacity was enabled	The VM was migrated to another node with sufficient capacity	Yes
Tried to scale a VM to a offering with more capacity than the current host when auto.migrate.vm.on.live.scale.insufficient.capacity was disabled	An error was thrown. The logs informed that the current host did not had enough capacity for the scaling	Yes

@DaanHoogland @weizhouapache can we run the CI here one last time?

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-16148)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52555 seconds
Marvin logs: [archive removed]
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

Add documentation and improved error messages to clarify version-specific constraints for live VM scaling: - CloudStack 4.22.1 and earlier: Live scaling only supported for custom constrained and custom unconstrained compute offerings - CloudStack 4.23.0+: Added support for live scaling with fixed service offerings on KVM hypervisors (see apache/cloudstack#12975) When scaling fails due to unsupported offering types, provide users with a helpful error message explaining the version constraints and suggesting they stop the VM if live scaling is not available.

@blueorangutan sweep

(cherry picked from commit 6b831f5)

bernardodemarco added this to the 4.23.0 milestone Apr 7, 2026

bernardodemarco requested review from GutoVeronezi, JoaoJandre, erikbocks, hsato03 and winterhazel April 7, 2026 18:08

boring-cyborg Bot added component:api component:compute component:kvm component:orchestration component:UI labels Apr 7, 2026

winterhazel added status:needs-testing status:needs-review labels Apr 9, 2026

github-actions Bot added the status:has-conflicts label Apr 10, 2026

bernardodemarco force-pushed the live-scaling-vms-with-fixed-service-offerings branch from 6e93c57 to 6f72aaa Compare April 13, 2026 17:39

bernardodemarco removed the status:has-conflicts label Apr 13, 2026

github-actions Bot added the status:has-conflicts label Apr 14, 2026

bernardodemarco force-pushed the live-scaling-vms-with-fixed-service-offerings branch from 6f72aaa to 23f6e2c Compare April 14, 2026 20:21

boring-cyborg Bot added component:integration-test Python Warning... Python code Ahead! labels Apr 14, 2026

bernardodemarco removed the status:has-conflicts label Apr 14, 2026

DaanHoogland reviewed Apr 15, 2026

View reviewed changes

Comment thread test/integration/component/test_escalations_ipaddresses.py Resolved

DaanHoogland reviewed Apr 15, 2026

View reviewed changes

Comment thread test/integration/component/test_escalations_ipaddresses.py Resolved

DaanHoogland reviewed Apr 15, 2026

View reviewed changes

Comment thread test/integration/component/test_escalations_ipaddresses.py Resolved

move MAX_CPU_QUOTA to CpuTuneDef

184c46a

bernardodemarco force-pushed the live-scaling-vms-with-fixed-service-offerings branch from 01941b6 to 184c46a Compare May 13, 2026 17:58

bernardodemarco removed the status:has-conflicts label May 13, 2026

shwstppr mentioned this pull request May 18, 2026

Unable to Autoscale K8s in CS 4.21.0.0 #11994

Open

erikbocks approved these changes May 20, 2026

View reviewed changes

bernardodemarco removed status:needs-testing status:needs-review labels May 20, 2026

winterhazel added status:ready-for-merge status:waiting-for-smoketest-results labels May 20, 2026

DaanHoogland removed the status:ready-for-merge label May 21, 2026

DaanHoogland added status:ready-for-merge and removed status:waiting-for-smoketest-results labels May 22, 2026

winterhazel removed the status:ready-for-merge label May 22, 2026

winterhazel merged commit 6b831f5 into apache:main May 22, 2026
26 of 28 checks passed

bernardodemarco mentioned this pull request May 22, 2026

Enable dynamic scaling of Kubernetes cluster offerings for KVM #13226

Merged

2 of 9 tasks

github-actions Bot mentioned this pull request Jun 5, 2026

[repo-status] Daily Status Report – June 5, 2026 #13360

Closed

weizhouapache added the type:improvement label Aug 24, 2026

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Sep 11, 2026

Live scaling for VMs with fixed service offerings on KVM (apache#12975 )

32cba6b

(cherry picked from commit 6b831f5)

andrijapanicsb mentioned this pull request Sep 22, 2026

CKS: reconcile node capacity after live resize #14223

Open

Live scaling for VMs with fixed service offerings on KVM by bernardodemarco · Pull Request #12975 · apache/cloudstack · GitHub

Repository navigation

Live scaling for VMs with fixed service offerings on KVM - #12975

Live scaling for VMs with fixed service offerings on KVM#12975winterhazel merged 3 commits intoapache:mainapache/cloudstack:mainfrom scclouds:live-scaling-vms-with-fixed-service-offeringsscclouds/cloudstack:live-scaling-vms-with-fixed-service-offeringsCopy head branch name to clipboard

Conversation

bernardodemarco commented Apr 7, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Test environment setup

Definition of the domain XML of the guest VMs

Live scaling

kvm.cpu.dynamic.scaling.capacity and kvm.memory.dynamic.scaling.capacity settings

auto.migrate.vm.on.live.scale.insufficient.capacity setting

Uh oh!

codecov Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bernardodemarco commented Apr 7, 2026

Uh oh!

weizhouapache commented Apr 7, 2026

Uh oh!

bernardodemarco commented Apr 13, 2026

Uh oh!

DaanHoogland commented Apr 14, 2026

Uh oh!

bernardodemarco commented Apr 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bernardodemarco commented May 13, 2026

Uh oh!

bernardodemarco commented May 13, 2026

Uh oh!

blueorangutan commented May 13, 2026

Uh oh!

blueorangutan commented May 13, 2026

Uh oh!

erikbocks left a comment

Choose a reason for hiding this comment

Uh oh!

winterhazel commented May 20, 2026

Uh oh!

DaanHoogland commented May 21, 2026

Uh oh!

blueorangutan commented May 21, 2026

Uh oh!

blueorangutan commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

DaanHoogland commented Aug 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Live scaling for VMs with fixed service offerings on KVM#12975
winterhazel merged 3 commits into
apache:mainfrom
scclouds:live-scaling-vms-with-fixed-service-offerings

codecov Bot commented Apr 7, 2026 •
edited
Loading

blueorangutan commented May 22, 2026 •
edited
Loading