Fix KVM live volume migration command payload by vanquyen020920 · Pull Request #13319 · apache/cloudstack · GitHub

Description

This PR fixes KVM live volume migration for storage pools where storage motion is supported.

Test evidence: failure before the fix and successful live migration after applying the code changes.

Environment tested

Apache CloudStack: 4.20.0.0
Management server OS: Ubuntu 22.04
KVM host OS: Ubuntu 22.04
Hypervisor: KVM
VM state: Running
Source primary storage: NetworkFilesystem
Destination primary storage: RBD / Ceph
API: migrateVolume with livemigrate=true

Problem

Currently, KVM live volume migration is rejected early in VolumeApiServiceImpl for non-PowerFlex pools with the following error:

KVM does not support volume live migration due to the limited possibility to refresh VM XML domain. Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. Use 'migrateVirtualMachineWithVolumes' instead.

When this early guard is bypassed, AncientDataMotionStrategy still creates MigrateVolumeCommand using the legacy constructor. That constructor does not populate srcData and destData.

As a result, the KVM agent receives a MigrateVolumeCommand with srcData == null, and LibvirtMigrateVolumeCommandWrapper fails with a NullPointerException while accessing srcVolumeObjectTO.getDataStore().

Functional change

This PR changes the migration flow to pass the source and destination data objects to the KVM agent and adds defensive null handling in the KVM wrapper.

Changes included:

Allows the KVM live volume migration flow to continue instead of rejecting it early for non-PowerFlex pools.
Uses the srcData / destData aware MigrateVolumeCommand constructor in AncientDataMotionStrategy.
Adds defensive null handling in LibvirtMigrateVolumeCommandWrapper.execute() to avoid an NPE if a legacy command path still sends a command without srcData.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

Screenshots were added above showing the failure before the fix and successful live migration after applying the code changes.

How Has This Been Tested?

Tested manually on Apache CloudStack 4.20.0.0 with Ubuntu 22.04 management and KVM hosts.

Test scenario:

VM was running on KVM.
Source primary storage was NetworkFilesystem.
Destination primary storage was RBD / Ceph.
Live volume migration was triggered using migrateVolume with livemigrate=true.

Before this change:

migrateVolume with livemigrate=true failed at the API layer with the KVM live volume migration restriction.
After bypassing the API guard, the KVM agent failed with a NullPointerException because srcVolumeObjectTO was null.

After this change:

MigrateVolumeCommand is created with srcData and destData.
The KVM agent receives the source and destination volume objects.
LibvirtMigrateVolumeCommandWrapper no longer throws a NullPointerException.
Live volume migration from NetworkFilesystem to RBD completed successfully for a running VM.

I also verified the bytecode before and after the change:

Before: AncientDataMotionStrategy called the legacy MigrateVolumeCommand(long, String, StoragePool, String, Volume.Type, int, String) constructor.
After: AncientDataMotionStrategy calls the MigrateVolumeCommand(DataTO, DataTO, Map, Map, int) constructor.

How did you try to break this feature and the system with this change?

I tested the failure path and the fixed path by rolling the staging environment back to the original behavior and then applying the fix again step by step.

Validation performed:

Confirmed the original API-level KVM restriction returned before applying the fix.
Confirmed the original KVM agent wrapper dereferenced srcData directly and could fail with srcVolumeObjectTO == null.
Applied the changes and retested the same live volume migration operation.
Verified that the command sent to the agent included source and destination data.
Verified that the migration completed successfully.
Verified that the fallback null guard in the KVM wrapper prevents a NullPointerException if a legacy command path is still used.

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

@vanquyen020920, did you check the final VM XML?

@vanquyen020920, thanks for the PR. I have tested it locally:

Created a VM with multiple disks;

virsh domblklist i-2-11-VM --details
Type   Device   Target   Source
----------------------------------------------------------------------------------------------------------
file   disk     hda      /mnt/2944d4ce-30c9-3c5f-8379-e49054e7c768/19390437-1503-4034-9033-6f72b094384e
file   cdrom    hdc      -
file   disk     vdb      /mnt/e6f01a81-688b-384a-83e6-7fa6d4605d62/e13e4f6b-0a48-4001-9da8-c1525deda620

Migrated the hda device from pri-stor-02-cluster-01-zn-xap-01 (NFS) to pri-stor-01-cluster-01-zn-xap-01 (NFS)

Verified that the VM's domain XML was not updated:

virsh domblklist i-2-11-VM --details
Type   Device   Target   Source
----------------------------------------------------------------------------------------------------------
file   disk     hda      /mnt/2944d4ce-30c9-3c5f-8379-e49054e7c768/19390437-1503-4034-9033-6f72b094384e
file   cdrom    hdc      -
file   disk     vdb      /mnt/e6f01a81-688b-384a-83e6-7fa6d4605d62/e13e4f6b-0a48-4001-9da8-c1525deda620

Migrated the vdb device from pri-stor-01-cluster-01-zn-xap-01 (NFS) to pri-stor-02-cluster-02-zn-xap-01 (NFS)

Verified that the VM's domain XML was not updated:

virsh domblklist i-2-11-VM --details
Type   Device   Target   Source
----------------------------------------------------------------------------------------------------------
file   disk     hda      /mnt/2944d4ce-30c9-3c5f-8379-e49054e7c768/19390437-1503-4034-9033-6f72b094384e
file   cdrom    hdc      -
file   disk     vdb      /mnt/e6f01a81-688b-384a-83e6-7fa6d4605d62/e13e4f6b-0a48-4001-9da8-c1525deda620

During tests, I also verified that only the metadata of the volumes were updated. This happens because, as indicated by the error message below, Apache CloudStack currently does not support live migration of volumes when using KVM with file-based storage pools (NFS or SharedMountPoint) or Ceph RBD.

cloudstack/server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

Lines 3534 to 3536 in 7308dad

    
           throw new InvalidParameterValueException("KVM does not support volume live migration due to the limited possibility to refresh VM XML domain. " + 
        
                   "Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. " + 
        
                   "Use 'migrateVirtualMachineWithVolumes' instead.");

Implementing this feature involves addressing a number of requirements and edge cases, including:

Updating VM domain XML definitions on the fly;
Handling volume backing chains with multiple deltas;
Handling volumes with incremental snapshots;
Handling VMs with incremental snapshots;
Supporting migrations between file-based storage pools and RBD;
And other related scenarios.

I am currently working on designing and implementing this functionality. The current plan is to have it available during the third or fourth quarter of this year.

this will have huge impact

this will have huge impact too

Thanks @bernardodemarco for testing and for the detailed feedback.

You were right. My initial patch was incomplete: it allowed the API/data-motion flow to continue and fixed the srcData == null issue, but it did not update the active libvirt domain disk source. As a result, CloudStack volume metadata could be updated while the running VM was still using the old source disk.

I reworked the KVM agent implementation locally to handle the regular live migration path using libvirt block copy + pivot instead of only relying on copyPhysicalDisk().

The updated flow I tested is:

Detect regular live migration for an attached/running KVM volume.
Prepare the destination disk on the destination storage pool.
Create the destination physical disk if it does not already exist.
Generate the destination libvirt disk XML.
Run libvirt blockCopy.
Wait for the block job to complete.
Pivot the running disk to the destination.
Verify that the active and inactive libvirt domain XML point to the destination disk before returning success.

Test environment:

Apache CloudStack 4.20.0.0
Ubuntu 22.04
KVM
Source primary storage: NFS / file-based primary storage
Destination primary storage: Ceph RBD

Test 1: system VM / virtual router root disk migration

Before migration, the running VM was using the NFS/file-based source:

file disk vda /mnt/0d464e3f-5176-3ba8-8c5f-01b8f4da5a2d/cbe98e70-0ec8-4dfa-8888-42b38a763672

During migration, the agent created the destination RBD volume and completed the live block copy:

Destination disk [cbe98e70-0ec8-4dfa-8888-42b38a763672] does not exist on pool [a15210c6-c858-3174-a390-183f4ed25096]. Creating it before live block copy.
Attempting to create volume cbe98e70-0ec8-4dfa-8888-42b38a763672 (RBD) in pool a15210c6-c858-3174-a390-183f4ed25096 with size (4.88 GB) 5242880000
Block copy has started for regular volume vda : cbe98e70-0ec8-4dfa-8888-42b38a763672
Block copy completed for the volume vda : cbe98e70-0ec8-4dfa-8888-42b38a763672

After migration, virsh domblklist shows that the active disk source was pivoted to Ceph RBD:

network disk vda CMC-CLOUDSTACK/cbe98e70-0ec8-4dfa-8888-42b38a763672

I also verified both active and inactive libvirt XML, and both now point to the RBD destination:

<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source protocol='rbd' name='CMC-CLOUDSTACK/cbe98e70-0ec8-4dfa-8888-42b38a763672'>
    <host name='10.14.5.55'/>
    <host name='10.14.5.56'/>
    <host name='10.14.5.57'/>
    <auth username='admin'>
      <secret type='ceph' uuid='a15210c6-c858-3174-a390-183f4ed25096'/>
    </auth>
  </source>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
</disk>

Test 2: user VM root disk migration

I also tested with a normal user VM.

Before migration, the VM root disk was on NFS/file-based storage:

file disk vda /mnt/0d464e3f-5176-3ba8-8c5f-01b8f4da5a2d/65ca9af4-ccb8-43a2-96f1-c91202cd192f

The guest had a test file before migration:

cat test.txt
Test Migrate Disk

After live migration to Ceph RBD, CloudStack UI shows the root volume on CPM-CEPH.

The agent log shows the destination RBD volume was created and block copy completed:

Using live block copy path for regular volume migration. VM [i-2-213-VM], source path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination pool [a15210c6-c858-3174-a390-183f4ed25096].
Preparing destination disk for regular live volume migration. Destination path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination pool [a15210c6-c858-3174-a390-183f4ed25096].
Destination disk [65ca9af4-ccb8-43a2-96f1-c91202cd192f] was not found on pool [a15210c6-c858-3174-a390-183f4ed25096]. It will be created before live block copy.
Destination disk [65ca9af4-ccb8-43a2-96f1-c91202cd192f] does not exist on pool [a15210c6-c858-3174-a390-183f4ed25096]. Creating it before live block copy.
Attempting to create volume 65ca9af4-ccb8-43a2-96f1-c91202cd192f (RBD) in pool a15210c6-c858-3174-a390-183f4ed25096 with size (10.00 GB) 10737418240
Block copy has started for regular volume vda : 65ca9af4-ccb8-43a2-96f1-c91202cd192f
Block copy completed for the volume vda : 65ca9af4-ccb8-43a2-96f1-c91202cd192f

After migration, I verified the guest remained running and the test file was still readable:

cat test.txt
Test Migrate Disk
Final

This addresses the specific issue you found where only CloudStack metadata changed while the VM domain XML remained unchanged. With the updated implementation, the running libvirt domain is pivoted to the destination storage.

I agree that the broader feature still needs careful validation for additional edge cases, including:

backing chains with multiple deltas;
incremental volume snapshots;
VM snapshots;
file-based to file-based migrations;
RBD to file-based migrations;
multi-disk VMs;
reboot validation after migration.

I will push the updated implementation and include this validation evidence so it can be reviewed and tested further.

How are we with this one? Looks like a great PR, but it has been silent for a while

How are we with this one? Looks like a great PR, but it has been silent for a while

@wido the code here only changes the database, the running vm keeps the old disk (see bernardo's test above). the rework the author mentioned on jun 3 was never pushed. #14426 from @weizhouapache does live volume moves between ceph, nfs, shared mount point and local pools, so should we close this in favour of it?

@Damans227 Got it. With that I’m closing this PR. The author is free to re-open it if that seems like the best thing. But as this one has been silent, closing it seems like the best option for now.

vanquyen020920 added 3 commits June 2, 2026 21:20

Allow KVM live volume migration flow

d786262

Pass source and destination data to migrate volume command

d5070af

Avoid null src data in KVM migrate volume wrapper

a6ec38a

boring-cyborg Bot added component:kvm component:storage labels Jun 2, 2026

bernardodemarco requested changes Jun 2, 2026

View reviewed changes

weizhouapache reviewed Jun 2, 2026

View reviewed changes

vanquyen020920 requested a review from bernardodemarco June 3, 2026 10:07

This was referenced Jun 18, 2026

[aw] Daily Issue Triage failed #13447

Closed

[aw] Daily Issue Triage failed #13458

Closed

[aw] Daily Issue Triage failed #13467

Closed

[aw] Daily Issue Triage failed #13487

Closed

github-actions Bot mentioned this pull request Jun 25, 2026

[aw] Daily Issue Triage failed #13493

Closed

weizhouapache added the component:ceph label Jun 29, 2026

weizhouapache added this to the 4.23.0 milestone Jun 29, 2026

github-actions Bot mentioned this pull request Jun 29, 2026

[aw] Daily Issue Triage failed #13514

Closed

weizhouapache modified the milestones: 4.23.0, 4.22.2 Jun 30, 2026

This was referenced Jul 3, 2026

[aw] Daily Issue Triage failed #13539

Closed

[aw] Daily Issue Triage failed #13550

Closed

[aw] Daily Issue Triage failed #13557

Closed

[aw] Daily Issue Triage failed #13570

Closed

This was referenced Jul 10, 2026

[aw] Daily Issue Triage failed #13590

Closed

[aw] Daily Issue Triage failed #13600

Closed

[aw] Daily Issue Triage failed #13623

Closed

[aw] Daily Issue Triage failed #13633

Closed

github-actions Bot mentioned this pull request Jul 20, 2026

[aw] Daily Issue Triage failed #13653

Closed

This was referenced Jul 22, 2026

[aw] Daily Issue Triage failed #13670

Closed

[aw] Daily Issue Triage failed #13694

Closed

[aw] Daily Issue Triage failed #13722

Closed

[aw] Daily Issue Triage failed #13736

Open

This was referenced Jul 30, 2026

[aw] Daily Issue Triage failed #13751

Open

[aw] Daily Issue Triage failed #13778

Open

[aw] Daily Issue Triage failed #13803

Closed

This was referenced Aug 7, 2026

[aw] Daily Issue Triage failed #13826

Closed

[aw] Daily Issue Triage failed #13841

Closed

[aw] Daily Issue Triage failed #13864

Closed

[aw] Daily Issue Triage failed #13873

Closed

DaanHoogland added this to CloudStack Testing Aug 31, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 31, 2026

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Sep 1, 2026

DaanHoogland assigned Damans227 Sep 1, 2026

wido closed this Oct 16, 2026

github-project-automation Bot moved this from Ready to Done in CloudStack Testing Oct 16, 2026

DaanHoogland removed this from the 4.22.2 milestone Nov 11, 2026

                       StoragePool srcPool = (StoragePool)dataStoreMgr.getDataStore(srcData.getDataStore().getId(), DataStoreRole.Primary);
                       StoragePool destPool = (StoragePool)dataStoreMgr.getDataStore(destData.getDataStore().getId(), DataStoreRole.Primary);
-                      MigrateVolumeCommand command = new MigrateVolumeCommand(volume.getId(), volume.getPath(), destPool, volume.getAttachedVmName(), volume.getVolumeType(), waitInterval, volume.getChainInfo());
+                      MigrateVolumeCommand command = new MigrateVolumeCommand(srcData.getTO(), destData.getTO(), null, null, waitInterval);

Fix KVM live volume migration command payload by vanquyen020920 · Pull Request #13319 · apache/cloudstack · GitHub

Repository navigation

Fix KVM live volume migration command payload - #13319

Fix KVM live volume migration command payload#13319
vanquyen020920 wants to merge 3 commits into
apache:mainfrom
vanquyen020920:main

vanquyen020920 commented Jun 2, 2026 •
edited
Loading

Uh oh!

Uh oh!

boring-cyborg Bot commented Jun 2, 2026

Uh oh!

GutoVeronezi commented Jun 2, 2026

Uh oh!

bernardodemarco left a comment

Uh oh!

weizhouapache Jun 2, 2026

Uh oh!

weizhouapache Jun 2, 2026

Uh oh!

vanquyen020920 commented Jun 3, 2026

Uh oh!

wido commented Oct 16, 2026

Uh oh!

Damans227 commented Oct 16, 2026

Uh oh!

wido commented Oct 16, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	throw new InvalidParameterValueException("KVM does not support volume live migration due to the limited possibility to refresh VM XML domain. " +
	"Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. " +
	"Use 'migrateVirtualMachineWithVolumes' instead.");

Fix KVM live volume migration command payload by vanquyen020920 · Pull Request #13319 · apache/cloudstack · GitHub

Repository navigation

Fix KVM live volume migration command payload - #13319

Fix KVM live volume migration command payload#13319vanquyen020920 wants to merge 3 commits intoapache:mainapache/cloudstack:mainfrom vanquyen020920:mainvanquyen020920/cloudstack:mainCopy head branch name to clipboard

Conversation

vanquyen020920 commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Environment tested

Problem

Functional change

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

boring-cyborg Bot commented Jun 2, 2026

Uh oh!

GutoVeronezi commented Jun 2, 2026

Uh oh!

bernardodemarco left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

weizhouapache Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

vanquyen020920 commented Jun 3, 2026

Test 1: system VM / virtual router root disk migration

Test 2: user VM root disk migration

Uh oh!

wido commented Oct 16, 2026

Uh oh!

Damans227 commented Oct 16, 2026

Uh oh!

wido commented Oct 16, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Fix KVM live volume migration command payload#13319
vanquyen020920 wants to merge 3 commits into
apache:mainfrom
vanquyen020920:main

vanquyen020920 commented Jun 2, 2026 •
edited
Loading