Repository navigation
Create template from snapshot parallel allocating same NFS secondary storage on agent #8899
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Storage
CLOUDSTACK VERSION
4.19.0
But I assume also 4.18 is affected
CONFIGURATION
I could recreate this with Linstor primary storage and the config option: lin.backup.snapshots disabled
I'm not sure yet which other storage combination could be used to trigger this.
OS / ENVIRONMENT
Ubuntu 22.04 non hyperconverged (2 storage nodes, 3 compute nodes (with agents)) setup.
SUMMARY
If I create 2 templates from snapshot (from different snapshots) at nearly the same time and the copy command is sent to the same agent,
both copy commands use the same storage pool object and the 1. finished will try to unmount
the still used storage pool (from the second copy command).
In the worst case the first copy command is finished while the second didn't even start yet to copy, resulting in the second copy command to write into the local mount directory.
STEPS TO REPRODUCE
Do both at the same time, with a primary storage that copies to secondary storage: * Create a snapshot from volume1. * Create a snapshot from volume2. If the copy commands run on the same agent you would see the secondary storage created twice. e.g.: 2024-04-10 12:03:13,081 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 (NetworkFilesystem) in libvirt 2024-04-10 12:03:13,089 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 2024-04-10 12:03:23,319 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1fab4dd4) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 (NetworkFilesystem) in libvirt 2024-04-10 12:03:43,257 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to remove storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 from libvirt 2024-04-10 12:03:48,862 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1fab4dd4) Attempting to remove storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 from libvirt
Attached are management and agent logs.
management.log
agent.log
EXPECTED RESULTS
Either running the copy commands in sequence or having a ref counted storage pool, that doesn't remove itself while still used.
ACTUAL RESULTS
2 storage pool objects created, pointing to the same mount point. The first finished would unmount the mount point for the other.
Activity
DaanHoogland commented on May 1, 2024
@rp- , I used the scenario and a script
but didn't reproduce the issue. I think it is credible though. By the looks of your description it shouldn't even be storage specific, but maybe there is a timing constrained that makes me not reproduce it.
Do you have a fix/idea/design to deal with this?
rp- commented on May 2, 2024
@DaanHoogland
I think it is kinda storage related, by the storage motion strategy CloudStack takes.
In this specific Linstor case CloudStack has to first copy the Snapshot into the secondary storage as template (as the snapshot in this case is only Linstor storage)
First I thought this is the only storage motion strategy to trigger that, but apparently a Customer reported that they still have problems with NFS mounts not correctly mounted/unmounted.
So it might either be that other storage motions are also affected, or the Customer setup has some extra flaw which I can't figure out.
Well my idea would be the 2 points from the "Expected results", either it is done in sequence OR the storage pool are refcounted.
DaanHoogland commented on May 2, 2024
@rp- ,
€0,02
btzq commented on May 17, 2024
Hi @DaanHoogland @rp- ,
Just curious, does this issue happen with Ceph users? I havent heard of any so far,
So if its working with Ceph, means that Linstor is using a different method?
DaanHoogland commented on Jun 18, 2024
@rp- I am moving this to 4.19.2 (or later) as we do not have a way forward yet.
rp- commented on Jun 20, 2024
@DaanHoogland
Here would be some way to fix this, but no feedback yet if it helps:
LINBIT@e53ea85
DaanHoogland commented on Jun 20, 2024
@rp- , I imagine you propose to move that logic into a more generic place like LibvirtStorageAdaptor or LibvirtStoragePool ?
rp- commented on Jul 1, 2024
Yes, wherever if would fit best.
DaanHoogland commented on Jul 1, 2024
@rp- , I would not know off the top of my head. The more I think about it the more locations seem apropriat 🤯 CloudStackImageStoreDriverImpl could also be a location to put this.
@harikrishna-patnala @slavkap any ideas?
slavkap commented on Jul 1, 2024
@DaanHoogland, I think the current suggestion (the LibvirtStorageAdaptor) from @rp- is the right place. I haven't faced the issue but it seems valid for all storage pools that need to make copies on secondary storage at some point. I didn't do any research but the fix from Rene will probably handle some corner cases with unmounting NFS as a primary storage as well.
rp- commented on Nov 8, 2024
I updated the reproduce steps, because it is even simpler.
Just have 2 snapshots created at the same time from different volumes and only have 1 agent running and you will see a error like this in the logs:
DaanHoogland commented on Nov 11, 2024
@rp- I think you have a PR that would close this issue, right? Can we link it?
DaanHoogland commented on Dec 6, 2024
fixed in #9498 (cc @rp- ?)