KVM: Enable HA heartbeat on ShareMountPoint by weizhouapache · Pull Request #12773 · apache/cloudstack · GitHub

Description

This PR improves KVM HA heartbeat to support SharedMountPoint

Tested

add shared mount point pool
kvm hosts write heartbeat to the mount point
deploy vms
set global settingforce.ha to true
force shutdown one of the kvm hosts
after 3 mins, the VMs are started on other kvm hosts.

Example of heartbeat files

[root@kvm1 ~]# ls -l /mnt/cloudstack*/KVMHA
/mnt/cloudstack1/KVMHA:
total 1
-rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
-rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76

/mnt/cloudstack2/KVMHA:
total 1
-rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
-rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Codecov Report

❌ Patch coverage is 3.84615% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.60%. Comparing base (c1af36f) to head (9cebe72).
⚠️ Report is 22 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...oud/hypervisor/kvm/storage/LibvirtStoragePool.java	0.00%	15 Missing ⚠️
...om/cloud/hypervisor/kvm/resource/KVMHAMonitor.java	0.00%	4 Missing ⚠️
.../hypervisor/kvm/storage/KVMStoragePoolManager.java	0.00%	3 Missing ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	0.00%	1 Missing ⚠️
...irtCheckVMActivityOnStoragePoolCommandWrapper.java	0.00%	1 Missing ⚠️
...e/driver/CloudStackPrimaryDataStoreDriverImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #12773      +/-   ##
============================================
- Coverage     17.61%   17.60%   -0.01%     
+ Complexity    15676    15674       -2     
============================================
  Files          5917     5917              
  Lines        531537   531546       +9     
  Branches      64985    64988       +3     
============================================
- Hits          93610    93598      -12     
- Misses       427369   427391      +22     
+ Partials      10558    10557       -1

Flag	Coverage Δ
uitests	3.70% <ø> (ø)
unittests	18.68% <3.84%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueoranguran package

Pull request overview

Enhances KVM HA heartbeat to support SharedMountPoint primary storage by introducing a new heartbeat script and extending HA support checks across KVM storage and monitor components.

Changes:

Added a new KVM “SharedMountPoint” heartbeat script (kvmsmpheartbeat.sh) that writes heartbeat files to a mountpoint.
Expanded HA-supported storage pool types to include SharedMountPoint in driver, monitor, and pool logic.
Added a setType hook to KVMStoragePool and set the pool type after creation / retrieval to ensure HA detection works.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh	New local mountpoint-based heartbeat writer/checker script for SharedMountPoint HA.
plugins/storage/volume/default/.../CloudStackPrimaryDataStoreDriverImpl.java	Allows HA support for SharedMountPoint in primary storage capability logic.
plugins/hypervisors/kvm/.../LibvirtStoragePool.java	Enables HA support for SharedMountPoint and selects the correct heartbeat script.
plugins/hypervisors/kvm/.../KVMStoragePoolManager.java	Sets pool type after creation; exposes getStorageAdaptor.
plugins/hypervisors/kvm/.../KVMStoragePool.java	Introduces a default setType method for backward compatibility.
plugins/hypervisors/kvm/.../LibvirtCheckVMActivityOnStoragePoolCommandWrapper.java	Ensures retrieved pool has its type set before HA checks.
plugins/hypervisors/kvm/.../KVMHAMonitor.java	Expands heartbeat monitoring to include SharedMountPoint pools.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17048

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17054

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@sureshanaparti @rajujith
added this to 4.22.1 milestone, a small improvement but may benefit several users

cc @DaanHoogland @NuxRo

[SF] Trillian test result (tid-15586)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55643 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15586-kvm-ol8.zip
Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	Error	1.12	test_list_ids_parameter.py
test_01_snapshot_root_disk	Error	7.74	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	47.53	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	47.54	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	Error	31.34	test_snapshots.py
test_01_snapshot_usage	Error	25.61	test_usage.py
test_01_vpn_usage	Error	1.08	test_usage.py

…/storage/KVMStoragePoolManager.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

clgtm

…d rename it

thanks @DaanHoogland @sureshanaparti , and Copilot for review

@NuxRo
I think this is ready for testing now

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17117

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-15631)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 58208 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15631-kvm-ol8.zip
Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	Error	1.14	test_list_ids_parameter.py
test_01_snapshot_root_disk	Error	4.78	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	47.49	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	47.50	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	Error	30.40	test_snapshots.py
test_01_snapshot_usage	Error	34.85	test_usage.py
test_01_vpn_usage	Error	1.08	test_usage.py

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17254

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-15749)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 57842 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15749-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_deployVMInSharedNetwork	Failure	331.05	test_network.py

LGTM
Tested

boring-cyborg Bot added component:kvm component:primary-storage component:storage labels Mar 9, 2026

KVM: Enable HA heartbeat on ShareMountPoint

93b95aa

weizhouapache force-pushed the 4.22-kvm-ha-on-shared-mount-point branch from 296835a to 93b95aa Compare March 9, 2026 12:47

weizhouapache requested a review from Copilot March 9, 2026 16:02

Copilot AI reviewed Mar 9, 2026

View reviewed changes

Copilot started reviewing on behalf of weizhouapache March 9, 2026 16:35

View session

weizhouapache added this to the 4.22.1 milestone Mar 10, 2026

weizhouapache added status:needs-testing status:needs-review labels Mar 10, 2026

weizhouapache added this to Apache CloudStack 4.22.1 Mar 10, 2026

weizhouapache linked an issue Mar 10, 2026 that may be closed by this pull request

KV Host HA/instance HA with shared mountpoint #9750

Closed

weizhouapache and others added 3 commits March 10, 2026 17:01

Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm…

f6c6a17

…/storage/KVMStoragePoolManager.java

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

a803c2f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

eacece1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rajujith moved this to In Progress in Apache CloudStack 4.22.1 Mar 11, 2026

DaanHoogland approved these changes Mar 11, 2026

View reviewed changes

sureshanaparti reviewed Mar 12, 2026

View reviewed changes

Comment thread plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java Resolved

move STORAGE_POOL_TYPES_WITH_HA_SUPPORT to HighAvailabilityManager an…

4e4da44

…d rename it

Merge branch '4.22' into 4.22-kvm-ha-on-shared-mount-point

9cebe72

sureshanaparti removed the status:needs-review label Mar 27, 2026

weizhouapache added the status:ready-for-testing label Mar 27, 2026

rajujith approved these changes Apr 10, 2026

View reviewed changes

rajujith removed status:needs-testing status:ready-for-testing labels Apr 10, 2026

sureshanaparti added the enhancement label Apr 10, 2026

sureshanaparti merged commit e297644 into apache:4.22 Apr 10, 2026
25 of 26 checks passed

github-project-automation Bot moved this from In Progress to Done in Apache CloudStack 4.22.1 Apr 10, 2026

sureshanaparti mentioned this pull request Apr 10, 2026

KV Host HA/instance HA with shared mountpoint #9750

Closed

DaanHoogland deleted the 4.22-kvm-ha-on-shared-mount-point branch April 10, 2026 08:46

dhslove pushed a commit to dhslove/ablestack-cloud that referenced this pull request Apr 21, 2026

KVM: Enable HA heartbeat on ShareMountPoint (apache#12773 )

579a228

weizhouapache mentioned this pull request May 11, 2026

HA : Hosts persist in the Suspect state in HA cluster with ShareMountPoint #10166

Closed

abh1sar mentioned this pull request Jul 3, 2026

Fix for encryption and migration not working on Shared Mount Point #13537

Merged

2 of 14 tasks

KVM: Enable HA heartbeat on ShareMountPoint by weizhouapache · Pull Request #12773 · apache/cloudstack · GitHub

Repository navigation

KVM: Enable HA heartbeat on ShareMountPoint - #12773

KVM: Enable HA heartbeat on ShareMountPoint#12773sureshanaparti merged 17 commits intoapache:4.22apache/cloudstack:4.22from weizhouapache:4.22-kvm-ha-on-shared-mount-pointweizhouapache/cloudstack:4.22-kvm-ha-on-shared-mount-pointCopy head branch name to clipboard

Conversation

weizhouapache commented Mar 9, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 10, 2026

Uh oh!

blueorangutan commented Mar 10, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

weizhouapache commented Mar 12, 2026

Uh oh!

blueorangutan commented Mar 12, 2026

Uh oh!

weizhouapache commented Mar 12, 2026

Uh oh!

blueorangutan commented Mar 12, 2026

Uh oh!

blueorangutan commented Mar 13, 2026

Uh oh!

weizhouapache commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

weizhouapache commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 27, 2026

Uh oh!

rajujith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KVM: Enable HA heartbeat on ShareMountPoint#12773
sureshanaparti merged 17 commits into
apache:4.22from
weizhouapache:4.22-kvm-ha-on-shared-mount-point

codecov Bot commented Mar 9, 2026 •
edited
Loading