VMware 80u2 and 80u3 updates/fixes by sureshanaparti · Pull Request #10586 · apache/cloudstack · GitHub

Description

This PR addresses power on & move datastore issues with VMware 80u2 and 80u3, and fixes tests for restore vm (disk cleanup issue), ssvm/cpvm (wait time to disconnect). Following are the details:

Power On issue
lock issue during PowerOn operation (for ROOT disk, which is cloned from seeded template)

https://knowledge.broadcom.com/external/article/316576/troubleshooting-issues-resulting-from-lo.html

Now, PowerOn operation is being tried after lock issue, and succeeds after few attempts.

Move datastore file issue
access issue with move datastore file operation (when creating volume from snapshot)

Now, Move datastore file operation is being tried after access issue, and succeeds after few attempts.

Restore VM test behavior/issue
test checks for old root disk existence and trying to cleanup.

But for VMware, old root disk is force expunged irrespective of the expunge parameter in restoreVirtualMachine API (which was introduced in PR #8800 - 4.19.1.0), so no need to check for old root disk exists when expunge is false.

cloudstack/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

Lines 1187 to 1191 in 2674da2

    
           // In case of VMware VM will continue to use the old root disk until expunged, so force expunge old root disk 
        
           // For system VM we do not need volume entry in Destroy state 
        
           if (vm.getHypervisorType() == HypervisorType.VMware || vm.getType().isUsedBySystem()) { 
        
               logger.info(String.format("Trying to expunge volume [%s] from primary data storage.", volumeToString)); 
        
               AsyncCallFuture<VolumeApiResult> future = volService.expungeVolumeAsync(volFactory.getVolume(existingVolume.getId()));

SSVM / CPVM test issue
Agent will be in Up state for a while after reboot, so need to wait for sometime for the agent to Disconnect and back to Up. Added wait time after reboot.

Other PR references:

[VMware] Add support for VMware 8.0u2 (8.0.2.x) and 8.0u3 (8.0.3.x) #9591 (4.20.0.0) - Updates hypervisor capabilities and guest os mappings for VMware 80u2 and 80u3.
SystemVM template changes - updated debian version & other changes #9625 (4.20.0.0) - Addresses SystemVM template issues for VMware 80u2 and 80u3.

VMware version references:

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested VM/Volume operations manually, and smoke tests.

How did you try to break this feature and the system with this change?

@blueorangutan package

Codecov Report

Attention: Patch coverage is 0% with 129 lines in your changes missing coverage. Please review.

Project coverage is 16.99%. Comparing base (dd84c74) to head (69d2c10).
Report is 18 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...oud/hypervisor/vmware/resource/VmwareResource.java	0.00%	63 Missing ⚠️
...ud/storage/resource/VmwareStorageLayoutHelper.java	0.00%	27 Missing ⚠️
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.00%	22 Missing ⚠️
...cloud/storage/resource/VmwareStorageProcessor.java	0.00%	16 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.20   #10586       +/-   ##
============================================
+ Coverage     4.00%   16.99%   +12.98%     
- Complexity       0    13238    +13238     
============================================
  Files          400     5255     +4855     
  Lines        32555   464813   +432258     
  Branches      5770    54518    +48748     
============================================
+ Hits          1305    78974    +77669     
- Misses       31101   377045   +345944     
- Partials       149     8794     +8645

Flag	Coverage Δ
uitests	?
unittests	16.99% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan package

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

lgtm

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12828

@blueorangutan package

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12829

@blueorangutan test ol8 vmware-80u3

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan test ol8 vmware-80u2

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u2) has been kicked to run smoke tests

@blueorangutan test ol8 vmware-80u1

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u1) has been kicked to run smoke tests

[SF] Trillian test result (tid-12768)
Environment: vmware-80u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 78805 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12768-vmware-80u3.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_more_vms_than_limit_allows	Error	223.40	test_deploy_vms_in_parallel.py
test_03_deploy_and_scale_kubernetes_cluster	Failure	102.81	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	Error	88.31	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	Failure	255.87	test_kubernetes_clusters.py
test_reboot_router	Error	459.86	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	1521.98	test_network.py
test_02_restore_vm_with_disk_offering	Error	104.27	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	Error	60.25	test_restore_vm.py
test_01_snapshot_to_volume	Error	9.37	test_snapshots.py
test_08_reboot_cpvm	Failure	39.37	test_ssvm.py
test_01_volume_usage	Error	95.45	test_usage.py

[SF] Trillian test result (tid-12771)
Environment: vmware-80u1 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 76196 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12771-vmware-80u1.zip
Smoke tests completed. 136 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_09_connectivity_between_network_and_vpc_tier	Failure	34.56	test_ipv4_routing.py
test_11_isolated_network_with_dynamic_routed_mode	Failure	163.88	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	Failure	411.53	test_ipv4_routing.py
test_01_ping_in_vr_success	Failure	60.52	test_diagnostics.py
test_04_autoscale_kubernetes_cluster	Failure	158.03	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	Failure	800.65	test_kubernetes_clusters.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	1524.38	test_network.py
test_02_restore_vm_with_disk_offering	Error	58.20	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	Error	114.73	test_restore_vm.py

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 12844

[SF] Trillian test result (tid-12770)
Environment: vmware-80u2 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 80994 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t12770-vmware-80u2.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	Error	0.00	test_cluster_drs.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Failure	488.47	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Error	488.49	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Error	123.15	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Error	123.16	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Error	331.55	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Error	331.58	test_internal_lb.py
test_09_connectivity_between_network_and_vpc_tier	Error	149.42	test_ipv4_routing.py
test_08_upgrade_kubernetes_ha_cluster	Failure	3692.24	test_kubernetes_clusters.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	1524.99	test_network.py
test_02_restore_vm_with_disk_offering	Error	60.36	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	Error	102.21	test_restore_vm.py
test_01_volume_usage	Error	88.37	test_usage.py

@blueorangutan test ol8 vmware-80u1

…ume from snapshot

…h more size than the template size

@blueorangutan package

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13381

@blueorangutan test ol8 vmware-80u3

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u3) has been kicked to run smoke tests

[SF] Trillian test result (tid-13307)
Environment: vmware-80u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 122502 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t13307-vmware-80u3.zip
Smoke tests completed. 137 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	Failure	3711.40	test_kubernetes_clusters.py
test_01_deployVMInSharedNetwork	Error	414.13	test_network.py
test_03_restore_vm_with_disk_offering_custom_size	Error	67.66	test_restore_vm.py
test_01_migrate_vm_strict_tags_success	Error	3607.88	test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure	Error	4.12	test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success	Error	23.90	test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure	Error	3604.35	test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success	Error	24.86	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	Error	3607.30	test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags	Error	23.42	test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags	Error	3605.85	test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success	Error	6.14	test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success	Error	6.03	test_vm_strict_host_tags.py

@blueorangutan test ol8 vmware-80u2

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u2) has been kicked to run smoke tests

@blueorangutan test ol8 vmware-80u1

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u1) has been kicked to run smoke tests

LGTM based on manual testing.

[SF] Trillian test result (tid-13328)
Environment: vmware-80u2 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 75117 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10586-t13328-vmware-80u2.zip
Smoke tests completed. 137 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_more_vms_than_limit_allows	Error	196.18	test_deploy_vms_in_parallel.py
test_01_deployVMInSharedNetwork	Failure	3604.36	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:teardown	Error	3605.64	test_network.py
test_03_restore_vm_with_disk_offering_custom_size	Error	57.38	test_restore_vm.py
test_01_scale_vm	Error	1.53	test_scale_vm.py
test_02_scale_vm_negative_offering_disable_scaling	Error	1.49	test_scale_vm.py
test_03_scale_vm_negative_vm_disable_scaling	Error	1.46	test_scale_vm.py

* VMware - Ignore disk not found error on cleanup when the VM disk doesn't exists * VMware - Retry powerOn on lock issues * addressed comments * Update CPVM reboot tests - wait for the agent to Disconnect and back Up * Retry moveDatastoreFile when any file access issue while creating volume from snapshot * Update full clone flag when restoring vm using root disk offering with more size than the template size * refactored (mainly,for diskInfo - causing NPE in some cases) * Retry moveDatastoreFile when there is any file access issue

boring-cyborg Bot added the component:vmware label Mar 19, 2025

sureshanaparti changed the base branch from main to 4.20 March 19, 2025 08:10

sureshanaparti requested a review from weizhouapache March 19, 2025 08:13

weizhouapache reviewed Mar 19, 2025

View reviewed changes

Comment thread ...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Resolved

weizhouapache approved these changes Mar 19, 2025

View reviewed changes

yadvr added this to the 4.20.1 milestone Mar 19, 2025

yadvr approved these changes Mar 19, 2025

View reviewed changes

Pearl1594 added this to ACS 4.20.1 Mar 19, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Mar 19, 2025

sureshanaparti added 5 commits May 13, 2025 15:58

VMware - Retry powerOn on lock issues

7846c18

addressed comments

470d3ad

Update CPVM reboot tests - wait for the agent to Disconnect and back Up

06248c5

Retry moveDatastoreFile when any file access issue while creating vol…

22fbe4e

…ume from snapshot

Update full clone flag when restoring vm using root disk offering wit…

e8ada29

…h more size than the template size

DaanHoogland assigned sureshanaparti May 13, 2025

sureshanaparti added 2 commits May 14, 2025 15:48

refactored (mainly,for diskInfo - causing NPE in some cases)

c9073c4

Retry moveDatastoreFile when there is any file access issue

69d2c10

sureshanaparti force-pushed the vmware-80u2-and-80u3-updates branch from f0d113e to 69d2c10 Compare May 14, 2025 10:59

apache deleted a comment from blueorangutan May 16, 2025

vladimirpetrov approved these changes May 16, 2025

View reviewed changes

yadvr merged commit 90316b2 into apache:4.20 May 16, 2025

github-project-automation Bot moved this from In Progress to Done in ACS 4.20.1 May 16, 2025

yadvr deleted the vmware-80u2-and-80u3-updates branch May 16, 2025 19:09

sureshanaparti mentioned this pull request May 19, 2025

test: fix test_restore_vm failure on vmware #10885

Merged

2 of 14 tasks

sureshanaparti mentioned this pull request Nov 12, 2025

Unable to access vmdk sice it is locked #12035

Closed

kiranchavala mentioned this pull request Dec 23, 2025

Unable to launch a vm on vmware cluster #12328

Closed

sureshanaparti mentioned this pull request Dec 24, 2025

Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 #12335

Merged

2 of 14 tasks

	// In case of VMware VM will continue to use the old root disk until expunged, so force expunge old root disk
	// For system VM we do not need volume entry in Destroy state
	if (vm.getHypervisorType() == HypervisorType.VMware \|\| vm.getType().isUsedBySystem()) {
	logger.info(String.format("Trying to expunge volume [%s] from primary data storage.", volumeToString));
	AsyncCallFuture<VolumeApiResult> future = volService.expungeVolumeAsync(volFactory.getVolume(existingVolume.getId()));

VMware 80u2 and 80u3 updates/fixes by sureshanaparti · Pull Request #10586 · apache/cloudstack · GitHub

Repository navigation

VMware 80u2 and 80u3 updates/fixes - #10586

VMware 80u2 and 80u3 updates/fixes#10586yadvr merged 8 commits intoapache:4.20apache/cloudstack:4.20from shapeblue:vmware-80u2-and-80u3-updatesshapeblue/cloudstack:vmware-80u2-and-80u3-updatesCopy head branch name to clipboard

Conversation

sureshanaparti commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

codecov Bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

sureshanaparti commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 19, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

weizhouapache commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

blueorangutan commented Mar 20, 2025

Uh oh!

Pearl1594 commented Mar 21, 2025

Uh oh!

sureshanaparti commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

DaanHoogland commented May 14, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

blueorangutan commented May 16, 2025

Uh oh!

sureshanaparti commented May 16, 2025

Uh oh!

blueorangutan commented May 16, 2025

Uh oh!

VMware 80u2 and 80u3 updates/fixes#10586
yadvr merged 8 commits into
apache:4.20from
shapeblue:vmware-80u2-and-80u3-updates

sureshanaparti commented Mar 19, 2025 •
edited
Loading

codecov Bot commented Mar 19, 2025 •
edited
Loading