Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 by sureshanaparti · Pull Request #12335 · apache/cloudstack · GitHub

Description

This PR addresses Clone VM issue with VMware 80u3, retries cloneVM task when any file access issue while cloning from volume or template and should succeed after few attempts.

This is another issue, recently identified with VMware 80u3 (other related issues addressed here: #10586)

Fixes #12035 #12328

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@blueorangutan package

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Codecov Report

❌ Patch coverage is 0% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.23%. Comparing base (5cb279b) to head (cfb35d2).
⚠️ Report is 6 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.00%	28 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #12335      +/-   ##
============================================
- Coverage     16.23%   16.23%   -0.01%     
+ Complexity    13379    13378       -1     
============================================
  Files          5657     5657              
  Lines        498866   498880      +14     
  Branches      60545    60546       +1     
============================================
+ Hits          81011    81012       +1     
- Misses       408821   408835      +14     
+ Partials       9034     9033       -1

Flag	Coverage Δ
uitests	4.00% <ø> (ø)
unittests	17.09% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16151

@blueorangutan package

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16156

LGTM, Tested with multicluster env

Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

@blueorangutan test matrix

@kiranchavala a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

LGTM, Tested with multicluster env

Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

as I remember, the issue appears very randomly.
it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue

[SF] Trillian test result (tid-15064)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50783 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15064-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[SF] Trillian test result (tid-15067)
Environment: xcpng82 (x2), zone: Advanced Networking with Mgmt server ol9
Total time taken: 53169 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15067-xcpng82.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	Error	0.00	test_cluster_drs.py
test_01_add_primary_storage_disabled_host	Error	15.34	test_primary_storage.py
test_01_primary_storage_iscsi	Error	0.15	test_primary_storage.py
test_01_primary_storage_nfs	Error	0.14	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	Error	0.23	test_primary_storage.py
test_02_list_snapshots_with_removed_data_store	Error	16.59	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	16.59	test_snapshots.py
test_01_deploy_vm_on_specific_host	Error	0.10	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	Error	0.09	test_vm_deployment_planner.py
test_08_migrate_vm	Error	0.05	test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_success	Error	0.19	test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure	Error	0.19	test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success	Error	0.17	test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure	Error	0.17	test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success	Error	0.19	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	Error	0.20	test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags	Error	0.17	test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags	Error	5.47	test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success	Error	0.16	test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success	Error	5.51	test_vm_strict_host_tags.py
test_05_deploy_vm_on_specific_host_with_strict_tags_failure	Failure	0.19	test_vm_strict_host_tags.py

[SF] Trillian test result (tid-15065)
Environment: kvm-ubuntu22 (x2), zone: Advanced Networking with Mgmt server u22
Total time taken: 62159 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15065-kvm-ubuntu22.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	Failure	241.82	test_routers_network_ops.py

[SF] Trillian test result (tid-15066)
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 69048 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15066-vmware-70u3.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_redundant_vpc_site2site_vpn	Error	872.37	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	Error	741.51	test_vpc_vpn.py

Do we really need to ignore interrupts for vm cloning?

that's for only sleep call @abh1sar

LGTM, Tested with multicluster env
Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

as I remember, the issue appears very randomly. it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue

Tested with multiple templates and the issue is not reproducible

Code LGTM

…me or template

LGTM, Tested with multicluster env
Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

as I remember, the issue appears very randomly. it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue

Tested with multiple templates and the issue is not reproducible

does this remove your doubts @weizhouapache ?

…me or template (apache#12335)

boring-cyborg Bot added the component:vmware label Dec 24, 2025

sureshanaparti linked an issue Dec 24, 2025 that may be closed by this pull request

Unable to launch a vm on vmware cluster #12328

Closed

sureshanaparti requested review from kiranchavala and weizhouapache December 24, 2025 06:46

sureshanaparti assigned kiranchavala Dec 24, 2025

sureshanaparti added this to the 4.20.3 milestone Dec 24, 2025

kiranchavala approved these changes Dec 26, 2025

View reviewed changes

sureshanaparti marked this pull request as ready for review December 29, 2025 03:57

abh1sar reviewed Dec 29, 2025

View reviewed changes

abh1sar approved these changes Dec 30, 2025

View reviewed changes

abh1sar added the status:ready-for-merge label Dec 30, 2025

Retry cloneVM task when any file access issue while cloning from volu…

cfb35d2

…me or template

sureshanaparti force-pushed the vmware80u3-clone-vm-lock-issue branch from c72cce8 to cfb35d2 Compare December 30, 2025 17:38

sureshanaparti linked an issue Dec 30, 2025 that may be closed by this pull request

Unable to access vmdk sice it is locked #12035

Closed

DaanHoogland merged commit a29de0e into apache:4.20 Jan 5, 2026
25 of 26 checks passed

DaanHoogland deleted the vmware80u3-clone-vm-lock-issue branch January 5, 2026 15:30

DaanHoogland mentioned this pull request Jan 5, 2026

Unable to access vmdk sice it is locked #12035

Closed

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jan 21, 2026

Retry cloneVM task when any file access issue while cloning from volu…

1cb96aa

…me or template (apache#12335)

sandeeplocharla pushed a commit to NetApp/cloudstack that referenced this pull request Feb 6, 2026

Retry cloneVM task when any file access issue while cloning from volu…

444278b

…me or template (apache#12335)

sureshanaparti mentioned this pull request Apr 30, 2026

First VM deployment from VMware template fails with VMDK locked; second deployment succeeds #11997

Open

DaanHoogland removed the status:ready-for-merge label Sep 2, 2026

+                                  try {
+                                      Thread.sleep(1000);
+                                  } catch (InterruptedException ie) {
+                                      logger.debug("Waiting to clone VM {} been interrupted: ", cloneName);

Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 by sureshanaparti · Pull Request #12335 · apache/cloudstack · GitHub

Repository navigation

Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 - #12335

Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3#12335DaanHoogland merged 1 commit intoapache:4.20apache/cloudstack:4.20from shapeblue:vmware80u3-clone-vm-lock-issueshapeblue/cloudstack:vmware80u3-clone-vm-lock-issueCopy head branch name to clipboard

Conversation

sureshanaparti commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Dec 24, 2025

Uh oh!

blueorangutan commented Dec 24, 2025

Uh oh!

codecov Bot commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Dec 24, 2025

Uh oh!

kiranchavala commented Dec 24, 2025

Uh oh!

blueorangutan commented Dec 24, 2025

Uh oh!

blueorangutan commented Dec 24, 2025

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

kiranchavala commented Dec 26, 2025

Uh oh!

blueorangutan commented Dec 26, 2025

Uh oh!

weizhouapache commented Dec 26, 2025

Uh oh!

blueorangutan commented Dec 26, 2025

Uh oh!

blueorangutan commented Dec 26, 2025

Uh oh!

blueorangutan commented Dec 27, 2025

Uh oh!

blueorangutan commented Dec 27, 2025

Uh oh!

abh1sar Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kiranchavala commented Dec 30, 2025

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Dec 30, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3#12335
DaanHoogland merged 1 commit into
apache:4.20from
shapeblue:vmware80u3-clone-vm-lock-issue

sureshanaparti commented Dec 24, 2025 •
edited
Loading

codecov Bot commented Dec 24, 2025 •
edited
Loading

sureshanaparti Dec 30, 2025 •
edited
Loading