FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[VMware to KVM] Cleanup leftover migrated volumes in case of migration failures by nvazquez · Pull Request #13151 · apache/cloudstack · GitHub

[VMware to KVM] Cleanup leftover migrated volumes in case of migration failures - #13151

Merged
weizhouapache merged 7 commits into
apache:mainfrom
shapeblue:423-vmw-kvm-cleanup-migrated-disks-failure
Jul 8, 2026
Merged

[VMware to KVM] Cleanup leftover migrated volumes in case of migration failures#13151
weizhouapache merged 7 commits into
apache:mainfrom
shapeblue:423-vmw-kvm-cleanup-migrated-disks-failure

Conversation

Copy link
Copy Markdown
Contributor

Description

This PR includes a cleanup mechanism for migrated volumes which are leftover in case the VMware to KVM migrations have failed unexpectedly.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

codecov Bot commented May 13, 2026
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.74144% with 169 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.94%. Comparing base (82986f6) to head (ce87ce4).

Files with missing lines Patch % Lines
...urce/wrapper/LibvirtBaseConvertCommandWrapper.java 47.28% 81 Missing and 16 partials ⚠️
...rtCleanupConvertedInstanceDisksCommandWrapper.java 4.16% 23 Missing ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java 18.51% 20 Missing and 2 partials ⚠️
...gent/api/CleanupConvertedInstanceDisksCommand.java 0.00% 20 Missing ⚠️
.../LibvirtImportConvertedInstanceCommandWrapper.java 12.50% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main   #13151   +/-   ##
=========================================
  Coverage     18.94%   18.94%           
- Complexity    18363    18366    +3     
=========================================
  Files          6192     6195    +3     
  Lines        556361   556451   +90     
  Branches      67908    67916    +8     
=========================================
+ Hits         105397   105421   +24     
- Misses       439393   439455   +62     
- Partials      11571    11575    +4     
Flag Coverage Δ
uitests 3.51% <ø> (ø)
unittests 20.15% <35.74%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17829

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Copy link
Copy Markdown

[SF] Trillian test result (tid-16076)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 4252 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16076-kvm-ol8.zip
Smoke tests completed. 9 look OK, 142 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
runTest Error 0.00 test_2fa.py
runTest Error 0.00 test_account_access.py
runTest Error 0.00 test_accounts.py
runTest Error 0.00 test_affinity_groups_projects.py
runTest Error 0.00 test_affinity_groups.py
runTest Error 0.00 test_annotations.py
runTest Error 0.00 test_async_job.py
runTest Error 0.00 test_attach_multiple_volumes.py
runTest Error 0.00 test_backup_recovery_dummy.py
runTest Error 0.00 test_backup_recovery_nas.py
runTest Error 0.00 test_backup_recovery_veeam.py
runTest Error 0.00 test_certauthority_root.py
runTest Error 0.00 test_cluster_drs.py
runTest Error 0.00 test_console_endpoint.py
runTest Error 0.00 test_create_list_domain_account_project.py
runTest Error 0.00 test_create_network.py
runTest Error 0.00 test_deploy_vgpu_enabled_vm.py
runTest Error 0.00 test_deploy_virtio_scsi_vm.py
runTest Error 0.00 test_deploy_vm_extra_config_data.py
runTest Error 0.00 test_deploy_vm_iso.py
runTest Error 0.00 test_deploy_vm_iso_uefi.py
runTest Error 0.00 test_deploy_vm_root_resize.py
runTest Error 0.00 test_deploy_vms_in_parallel.py
runTest Error 0.00 test_deploy_vms_with_varied_deploymentplanners.py
runTest Error 0.00 test_deploy_vm_with_userdata.py
runTest Error 0.00 test_diagnostics.py
runTest Error 0.00 test_direct_download.py
runTest Error 0.00 test_disk_offerings.py
runTest Error 0.00 test_disk_provisioning_types.py
runTest Error 0.00 test_domain_disk_offerings.py
runTest Error 0.00 test_domain_network_offerings.py
runTest Error 0.00 test_domain_service_offerings.py
runTest Error 0.00 test_domain_vpc_offerings.py
runTest Error 0.00 test_enable_account_settings_for_domain.py
runTest Error 0.00 test_events_resource.py
runTest Error 0.00 test_extension_custom_action_lifecycle.py
runTest Error 0.00 test_extension_custom.py
runTest Error 0.00 test_extension_lifecycle.py
runTest Error 0.00 test_gateway_on_shared_networks.py
runTest Error 0.00 test_global_acls.py
runTest Error 0.00 test_global_settings.py
runTest Error 0.00 test_guest_os.py
runTest Error 0.00 test_guest_vlan_range.py
runTest Error 0.00 test_host_control_state.py
runTest Error 0.00 test_hostha_simulator.py
runTest Error 0.00 test_host_ping.py
runTest Error 0.00 test_image_store_object_migration.py
runTest Error 0.00 test_import_unmanage_volumes.py
runTest Error 0.00 test_internal_lb.py
runTest Error 0.00 test_ipv4_routing.py
runTest Error 0.00 test_ipv6_infra.py
runTest Error 0.00 test_iso.py
runTest Error 0.00 test_kubernetes_clusters.py
runTest Error 0.00 test_kubernetes_supported_versions.py
runTest Error 0.00 test_list_accounts.py
runTest Error 0.00 test_list_disk_offerings.py
runTest Error 0.00 test_list_domains.py
runTest Error 0.00 test_list_hosts.py
runTest Error 0.00 test_login.py
runTest Error 0.00 test_list_ids_parameter.py
runTest Error 0.00 test_list_service_offerings.py
runTest Error 0.00 test_list_storage_pools.py
runTest Error 0.00 test_list_volumes.py
runTest Error 0.00 test_loadbalance.py
runTest Error 0.00 test_metrics_api.py
runTest Error 0.00 test_migration.py
runTest Error 0.00 test_ms_maintenance_and_safe_shutdown.py
runTest Error 0.00 test_multipleips_per_nic.py
runTest Error 0.00 test_nested_virtualization.py
runTest Error 0.00 test_network_acl.py
runTest Error 0.00 test_network_ipv6.py
runTest Error 0.00 test_network_permissions.py
runTest Error 0.00 test_network.py
runTest Error 0.00 test_public_ip_range.py
runTest Error 0.00 test_nic_adapter_type.py
runTest Error 0.00 test_nic.py
runTest Error 0.00 test_non_contigiousvlan.py
runTest Error 0.00 test_nonstrict_affinity_group.py
runTest Error 0.00 test_outofbandmanagement_nestedplugin.py
runTest Error 0.00 test_outofbandmanagement.py
runTest Error 0.00 test_over_provisioning.py
runTest Error 0.00 test_password_server.py
runTest Error 0.00 test_persistent_network.py
runTest Error 0.00 test_portable_publicip.py
runTest Error 0.00 test_portforwardingrules.py
runTest Error 0.00 test_primary_storage.py
runTest Error 0.00 test_primary_storage_scope.py
runTest Error 0.00 test_privategw_acl_ovs_gre.py
runTest Error 0.00 test_privategw_acl.py
runTest Error 0.00 test_projects.py
runTest Error 0.00 test_purge_expunged_vms.py
runTest Error 0.00 test_pvlan.py
runTest Error 0.00 test_quarantined_ips.py
runTest Error 0.00 test_regions.py
runTest Error 0.00 test_register_userdata.py
runTest Error 0.00 test_reset_configuration_settings.py
runTest Error 0.00 test_reset_vm_on_reboot.py
runTest Error 0.00 test_resource_accounting.py
runTest Error 0.00 test_resource_detail.py
runTest Error 0.00 test_resource_names.py
runTest Error 0.00 test_restore_vm.py
runTest Error 0.00 test_router_dhcphosts.py
runTest Error 0.00 test_router_dns.py
runTest Error 0.00 test_router_dnsservice.py
runTest Error 0.00 test_routers_iptables_default_policy.py
runTest Error 0.00 test_routers_network_ops.py
runTest Error 0.00 test_routers.py
runTest Error 0.00 test_scale_vm.py
runTest Error 0.00 test_secondary_storage.py
runTest Error 0.00 test_service_offerings.py
runTest Error 0.00 test_set_sourcenat.py
runTest Error 0.00 test_sharedfs_lifecycle.py
runTest Error 0.00 test_snapshots.py
runTest Error 0.00 test_ssl_offloading.py
runTest Error 0.00 test_ssvm.py
runTest Error 0.00 test_storage_policy.py
runTest Error 0.00 test_systemvm_userdata.py
runTest Error 0.00 test_templates.py
runTest Error 0.00 test_update_security_group.py
runTest Error 0.00 test_usage_events.py
runTest Error 0.00 test_usage.py
runTest Error 0.00 test_vm_autoscaling.py
runTest Error 0.00 test_vm_deployment_planner.py
runTest Error 0.00 test_vm_life_cycle.py
runTest Error 0.00 test_vm_lifecycle_unmanage_import.py
runTest Error 0.00 test_vm_lifecycle_unmanage_kvm_import.py
runTest Error 0.00 test_vm_lifecycle_with_snapshot_or_volume.py
runTest Error 0.00 test_vm_schedule.py
runTest Error 0.00 test_vm_snapshot_kvm.py
runTest Error 0.00 test_vm_snapshots.py
runTest Error 0.00 test_vm_strict_host_tags.py
runTest Error 0.00 test_vnf_templates.py
runTest Error 0.00 test_volumes.py
runTest Error 0.00 test_vpc_conserve_mode.py
runTest Error 0.00 test_vpc_ipv6.py
runTest Error 0.00 test_vpc_redundant.py
runTest Error 0.00 test_vpc_router_nics.py
runTest Error 0.00 test_vpc_vpn.py
runTest Error 0.00 test_webhook_delivery.py
runTest Error 0.00 test_webhook_lifecycle.py
runTest Error 0.00 test_host_maintenance.py
runTest Error 0.00 test_hostha_kvm.py

sureshanaparti requested a review from Copilot May 13, 2026 07:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a best-effort cleanup path for temporary converted volumes when VMware→KVM import fails, and refactors KVM conversion wrappers to share common helper logic.

Changes:

  • Trigger cleanup of temporary converted disks on import failures/timeouts.
  • Introduce CleanupConvertedInstanceDisksCommand and a corresponding KVM resource wrapper to delete leftover conversion artifacts.
  • Refactor LibvirtImportConvertedInstanceCommandWrapper to reuse a new shared LibvirtBaseConvertCommandWrapper.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java Sends a new cleanup command to remove temporary converted disks when import fails.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapper.java Refactors wrapper to inherit shared convert/import helpers and updates cleanup call signature.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtCleanupConvertedInstanceDisksCommandWrapper.java Adds KVM-side handler that locates and deletes temporary conversion disks (and XML when present).
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtBaseConvertCommandWrapper.java Introduces shared helper methods previously embedded in the import wrapper.
core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksCommand.java Adds agent command to request cleanup of converted disks by store + prefix.
core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksAnswer.java Adds a new Answer type for the cleanup command (currently empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

harikrishna-patnala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

code LGTM. @nvazquez please check if we can add some unit tests here.

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17841

Copy link
Copy Markdown
Contributor

@blueorangutan test

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18334

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Copy link
Copy Markdown

[SF] Trillian test result (tid-16395)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52785 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16395-kvm-ol8.zip
Smoke tests completed. 150 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18356

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18367

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

Copy link
Copy Markdown

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18370

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

Copy link
Copy Markdown

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Copy link
Copy Markdown

[SF] Trillian test result (tid-16428)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50929 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13151-t16428-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

Copy link
Copy Markdown
Member

@nvazquez
is this applicable for 4.22 too ?

weizhouapache commented Jul 8, 2026
edited
Loading

Copy link
Copy Markdown
Member

@nvazquez
it looks like this PR can be applied to 4.22 too.

(4.22)# patch -p1 <13151.diff 
patching file core/src/main/java/com/cloud/agent/api/CleanupConvertedInstanceDisksCommand.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtBaseConvertCommandWrapper.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtCleanupConvertedInstanceDisksCommandWrapper.java
patching file plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapper.java
patching file plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtImportConvertedInstanceCommandWrapperTest.java
patching file server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java
Hunk #2 succeeded at 1721 (offset 3 lines).
Hunk #3 succeeded at 2219 (offset 3 lines).

Could you rebase with 4.22 ?

I am moving it to 4.22.2 milestone

weizhouapache modified the milestones: 4.23.0, 4.22.2 Jul 8, 2026

RosiKyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM

Executed the full test plan for FR307: VMWare-2-KVM part II - Cleanup after failure - 19/19 test cases Pass (100%).

Summary:
Testing covered cleanup of temporary artifacts (disks and XML descriptors) after failed VMware-to-KVM imports: temporary locations on the primary pool and NFS secondary storage, XML handling (present / absent / corrupt with fallback to prefix listing), forceConvertToPool positive and negative paths, mid-move multi-disk failure (customer repro), destination-pool orphan check, and correct cleanup scoping (concurrent imports and the convert-stage boundary remain untouched). Negative scenarios confirmed no crashes when there is nothing to clean or when cleanup itself fails, with the original error preserved. Failed conversions are correctly reflected in the UI and in listImportVmTasks, and the successful import regression passed with temporary artifacts fully cleaned.

Executed test cases:

  • Concurrent imports cleanup scoped to failed prefix only
  • Import fails after temporary disks already cleaned nothing to clean no crash (negative)
  • Convert host equals import host VDDK same host cleanup works
  • Import VM tasks UI reflects failed conversion
  • listImportVmTasks shows Failed record on pool selection failure
  • Convert-stage failure not cleaned scope boundary
  • forceConvertToPool success no erroneous cleanup (positive)
  • forceConvertToPool import failure cleans pool disks (negative)
  • Successful import regression temporary fully cleaned (positive)
  • Temporary location primary pool cleaned
  • Temporary location NFS secondary cleaned (positive)
  • Destination-pool orphan check after mid-move failure (negative)
  • Mid-move multi-disk failure temporary fully cleaned customer repro
  • Cleanup with corrupt XML falls back to prefix listing (CS-3416)
  • Cleanup with XML absent removes disks skips XML
  • Cleanup with XML present removes disks and XML (positive)
  • Import failure agent unavailable cleans temporary disks
  • Cleanup failure preserves original error and does not crash (negative)
  • Import failure bad result cleans temporary disks (positive)

Copy link
Copy Markdown
Member

thanks @RosiKyu

discussed with @nvazquez , merging into main/4.23

weizhouapache merged commit 72197fe into apache:main Jul 8, 2026
26 of 27 checks passed
weizhouapache deleted the 423-vmw-kvm-cleanup-migrated-disks-failure branch July 8, 2026 11:07
weizhouapache modified the milestones: 4.22.2, 4.23.0 Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants


Back | FazBrowse Home | New Git URL