Looks like there is an additional Issue open #2657 and related PR #2658

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2154

@blueorangutan test

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-2827)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30394 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2722-t2827-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

@Slair1 why not master instead of 4.11?
4.11.1 just went out.

@borisstoyanov I didn't know if there was going to be another 4.11.x release or not. If we prefer, i can rebase to master?

@Slair1 this is a bug fix. It can go into 4.11 and if we merge we also merge forward to master

This PR doesn't seem to completely fix the problem (or maybe this is a completely new problem). We installed the RC release with this PR on a test system and are able to get the KVM host to be marked as Down by using iptables to drop outgoing requests to NFS. My investigation shows that the line storage = conn.storagePoolLookupByUUIDString(uuid); blocks indefinitely. So, kvmheartbeat.sh is never executed, a host investigation is started, the host with blocked NFS is marked as Down and finally all VMs on that host are rescheduled and result in duplicate VMs.

I pulled a thread dump and found the KVMHAMonitor thread will hang here until NFS is unblocked, didn't dig any deeper yet though.

   java.lang.Thread.State: RUNNABLE
        at com.sun.jna.Native.invokePointer(Native Method)
        at com.sun.jna.Function.invokePointer(Function.java:470)
        at com.sun.jna.Function.invoke(Function.java:404)
        at com.sun.jna.Function.invoke(Function.java:315)
        at com.sun.jna.Library$Handler.invoke(Library.java:212)
        at com.sun.proxy.$Proxy3.virStoragePoolLookupByUUIDString(Unknown Source)
        at org.libvirt.Connect.storagePoolLookupByUUIDString(Unknown Source)
        at com.cloud.hypervisor.kvm.resource.KVMHAMonitor$Monitor.runInContext(KVMHAMonitor.java:95)
        - locked <1afb3370> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None```

Good find @csquire, @Slair1 any comments? /cc @PaulAngus @borisstoyanov @DaanHoogland

Sorry, I misspoke in my last comment (edited to make it correct). The blocked host doesn't reboot, it just gets marked as Down and the VMs are actually still running on it when duplicate VMs get provisioned. Maybe it's a completely separate issue, but will still prevent us from using 4.11 in production. EDIT: Actually, looks like it may have been present before 4.11.

@csquire The VMs should eventually crash themselves since they won't have access to their disks. In our environment when a host gets marked as down we get alerted and an engineer can take a look at what is going on. Yes, we rely on VM HA to decide what to do with the problem host also. We make sure lockd is enabled so we don't get a split-brain situation.

https://libvirt.org/locking-lockd.html

@Slair1 thanks for the feedback, I will run more tests. The behavior is still incorrect, so I went ahead and submitted issue #2890

@Slair1 Testing with virtlockd, if the host that crashed can't come back I had to manually clear the locks for VM HA to fire and restart the VM. At that point it's the same as disabling VM HA and manually restart VM's after a KVM host crash. How is your setup different ?

@somejfn maybe because I have PR #2474 also installed?

I should n't have taken word on this, I tested and could not reproduce the behaviour @csquire reported. I think we should revert the change and let the KVM host reset. VM disk corruption is worse than VM downtime.

I'll try to summarise the scenario so that we're all trying to fix the same thing...

1. A host cannot write to one of it's primary storage pools.
2. The some of the VMs on that host are on that pool, so their disks have gone read-only, but the VM is still running.
3. BUT there may be VMs on other primary storage pools that are absolutely fine

IMHO i'd say if a VM on that storage is marked as ha-enabled, it should be powered-off and restarted somewhere else, and if it isn't HA enabled, we shouldn't do anything with the running VM (as it's for the user of the VM to deal with it),
in either case we should probably set the host to 'alert' so then an admin can see it and do something about it.