kvm: look RBD volumes up through librbd instead of refreshing the whole pool by bhouse-nexthop · Pull Request #14188 · apache/cloudstack · GitHub

Description

Starting a VM whose root disk is on RBD primary storage refreshes the entire libvirt storage pool. The cost grows with the number of volumes in the Ceph pool, and it is paid on every single VM start.

Why it happens

LibvirtStorageAdaptor.getPhysicalDisk() asks libvirt for the volume. The volume was just created by the management server, so this host's libvirt has never seen it and the lookup misses. getVolume() then refreshes the whole pool:

vol = pool.storageVolLookupByName(volName);
if (vol == null) {
    refreshPool(pool);                        // opens and stats EVERY image in the pool
    vol = pool.storageVolLookupByName(volName);
}

The existing comment in getVolume() already describes the case: "This can happen when a volume has just been created on a different host and since then the libvirt storage pool has not been refreshed." For a cluster that creates VMs continuously, that is every start.

Measured cost

Ceph pool holding 950 RBD images, measured on the hypervisor:

operation	time
list image names only	0.005 s
open + stat + close every image, serially (what a pool refresh does)	11.67 s

Running four of these at once does not slow them down (9.9 - 11.1 s each), so this is not Ceph contention. It is 950 serial round trips.

Agent-side StartCommand on a host in a 12-node cluster running short-lived VMs:

	p50	p90	max
whole StartCommand	35.2 s	68.4 s	141 s
the stall inside getPhysicalDisk	20.3 s	46.6 s	304 s

The stall is 0 when the volume happens to be cached, which is what points at the refresh rather than at fixed work.

Fix

Look RBD volumes up directly through librbd and skip libvirt entirely.

This makes getPhysicalDisk() consistent with the rest of the class. Creating, cloning, resizing, copying and deleting RBD volumes in LibvirtStorageAdaptor already use Rados/Rbd directly. Only the lookup went through libvirt.

Applies to StoragePoolType.RBD only. Every other pool type is untouched.
The disk path <ceph pool>/<volume uuid> is the same format the class already builds in createPhysicalDisk, createDiskFromTemplate and createDiskFromTemplateOnRBD.
Size and virtual size come from rbd stat, matching what the class already does after converting an image into an RBD volume.

What this does not fix

getStorageStats still calls pool.refresh() explicitly, so the full walk of the pool still happens on the storage stats poll. That is a periodic background poll rather than something on the VM start path, so it is left alone here.

Connection handling

Because this runs on every VM start rather than occasionally, the rados connection is handled more carefully than at the existing call sites in this class:

	why
shutDown() in a finally	KVMStoragePoolManager.getPhysicalDisk retries a missing volume 100 times with a 3 s sleep, so leaking the client would mean 101 live librados clients for one absent volume
openReadOnly() rather than open()	the image is only stat'ed, so it should not be able to take the exclusive lock
a close helper that does not throw	Rbd.close() throws, and in a finally that would discard a successful lookup or hide the exception already on its way out
cephx key only set when the pool has a user	librados aborts the process, rather than returning an error, if rados_conf_set("key", NULL) is called. This is pre-existing at the other call sites in this class and is guarded only on the new one

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

How Has This Been Tested?

Measured the cost of the pool refresh against the observed agent stall, as in the tables above. rbd_list on its own is 0.005 s; opening and stating all 950 images serially, which is what the refresh does, is 11.67 s.
Ran the same enumeration four times in parallel to rule out Ceph contention: 9.9 - 11.1 s each, no slowdown.
Checked that the new path produces the same disk path format, format and size as the libvirt path for RBD volumes.
Full KVM plugin test suite on this branch: 676 tests, 0 failures, 1 skipped.

On the one non-obvious point, the librbd lookup reports rbd stat size as both size and virtual size, where the libvirt path reported allocation and capacity separately. These are the same number for RBD: libvirt derives allocation as obj_size * num_objs. Checked against libvirt 10.6.0 on three RBD volumes of different sizes:

Capacity:  32212254720 bytes   Allocation:  32212254720 bytes
Capacity:  32212254720 bytes   Allocation:  32212254720 bytes
Capacity: 644245094400 bytes   Allocation: 644245094400 bytes

Consumers of those two values (LibvirtGetVolumeStatsCommandWrapper, the template and volume sizes reported back by KVMStorageProcessor, and the snapshot pre-check in validateAvailableSizeOnPoolToTakeVolumeSnapshot) therefore see no change.

Codecov Report

❌ Patch coverage is 0% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.94%. Comparing base (10037c8) to head (d74bf10).
⚠️ Report is 2 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	42 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14188   +/-   ##
=========================================
  Coverage     17.93%   17.94%           
- Complexity    16142    16148    +6     
=========================================
  Files          5928     5928           
  Lines        535205   535247   +42     
  Branches      65501    65503    +2     
=========================================
+ Hits          95989    96028   +39     
- Misses       428286   428290    +4     
+ Partials      10930    10929    -1

Flag	Coverage Δ
uitests	4.02% <ø> (ø)
unittests	19.01% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

getPhysicalDisk() asks libvirt for the volume. A volume that was just created by the management server is not in this host's libvirt pool cache, so the lookup misses and getVolume() falls back to refreshing the whole pool. Refreshing an RBD pool opens and stats every image in it, so the cost grows with the number of volumes in the pool and is paid on every VM start. On a pool holding 950 images that is 11.7 seconds added to each start, against 0.005 seconds to list the image names. Look RBD volumes up directly through librbd instead. Creating, cloning, resizing, copying and deleting RBD volumes in this class already use librbd directly; only the lookup went through libvirt. Because this runs on every VM start, the connection is handled more carefully than at the existing call sites: - the rados connection is shut down in a finally, so a client is not left for the finalizer. KVMStoragePoolManager.getPhysicalDisk retries a missing volume 100 times, so leaking here would mean 101 live librados clients for one absent volume. - the image is opened read only, since it is only stat'ed, and so cannot take the exclusive lock. - the image is closed through a helper that does not throw, so a failed close cannot discard a successful lookup or hide the original error. - the cephx key is only set when the pool has a user. librados aborts the process if it is handed a null value. Signed-off-by: Brad House <bhouse@nexthop.ai>

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19299

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-17030)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51014 seconds
Marvin logs: [archive removed]
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

@vladimirpetrov @sureshanaparti could you take a look at this one? We'd like this fix to make it into the upcoming 4.22.2 release.

code lgtm

run several tests on ceph storage pools, no issues found

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

Rados is constructed with pool.getAuthUserName() even when that value can be null (the later… · New
"30" is a newly introduced magic number for the Ceph client mount timeout, and it’s not obvious… · New
The error logging eagerly builds strings via concatenation and does not include the exception as a… · New

What changed in this PR

Updates KVM RBD volume discovery to avoid libvirt storage pool refreshes during VM start by querying Ceph directly via librbd.

Changes:

Route getPhysicalDisk() lookups for StoragePoolType.RBD through a new librbd-based lookup path.
Add getRbdPhysicalDisk() to stat RBD images via Rados/Rbd and build a KVMPhysicalDisk without libvirt.
Add a best-effort helper to close RbdImage without masking prior exceptions.

File	Description
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java	Adds a librbd-backed RBD volume lookup to avoid expensive full pool refreshes on VM start.

🧠 Review effort: Lite

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

apache#14188) getPhysicalDisk() asks libvirt for the volume. A volume that was just created by the management server is not in this host's libvirt pool cache, so the lookup misses and getVolume() falls back to refreshing the whole pool. Refreshing an RBD pool opens and stats every image in it, so the cost grows with the number of volumes in the pool and is paid on every VM start. On a pool holding 950 images that is 11.7 seconds added to each start, against 0.005 seconds to list the image names. Look RBD volumes up directly through librbd instead. Creating, cloning, resizing, copying and deleting RBD volumes in this class already use librbd directly; only the lookup went through libvirt. Because this runs on every VM start, the connection is handled more carefully than at the existing call sites: - the rados connection is shut down in a finally, so a client is not left for the finalizer. KVMStoragePoolManager.getPhysicalDisk retries a missing volume 100 times, so leaking here would mean 101 live librados clients for one absent volume. - the image is opened read only, since it is only stat'ed, and so cannot take the exclusive lock. - the image is closed through a helper that does not throw, so a failed close cannot discard a successful lookup or hide the original error. - the cephx key is only set when the pool has a user. librados aborts the process if it is handed a null value.

Commit 26e7c3f (apache#13763, let libvirt remove RBD snapshots on volume delete) removed the com.ceph.rados.exceptions.ErrorCode import together with its last uses. Commit 08a706d (apache#14188, look RBD volumes up through librbd instead of refreshing the pool) brought the uses back without the import, so plugins/hypervisors/kvm failed to compile.

The cherry-pick of PR apache#14188 (look RBD volumes up through librbd instead of refreshing the pool) referenced com.ceph.rados.exceptions.ErrorCode without importing it, so plugins/hypervisors/kvm did not compile.

getRbdPhysicalDisk came in with apache#14188 after this change branched, so it still built its own Rados handle with only client_mount_timeout set. The lookup that runs on every deploy and VM start would then block forever during an OSD outage. Open through CephUtil.connect(), so it gets the same operation timeouts as every other Ceph handle in the plugin, and release the IO context and the cluster handle from the finally with the non-throwing helpers, so a failure to release cannot mask the exception on its way out. The cephx guard that this method carried is now inside CephUtil.connect() and covers every caller. Signed-off-by: Brad House <bhouse@nexthop.ai>

LibvirtResizeVolumeCommandWrapper gets the disk with pool.getPhysicalDisk() and then looks the volume up through libvirt with storageVolLookupByPath() to resize it. Before apache#14188, getPhysicalDisk() went through LibvirtStorageAdaptor.getVolume(), which refreshes the libvirt pool when the volume is not found, so the volume was in libvirt's cache by the time the wrapper looked it up. apache#14188 made getPhysicalDisk() stat RBD images through librbd, so nothing refreshes the pool any more. RBD volumes are created through librbd, not libvirt, so a new volume is not in libvirt's cache on a host until something refreshes the pool there, and resizing it fails with: Storage volume not found: no storage vol with matching path '<pool>/<uuid>' When the lookup by path fails with VIR_ERR_NO_STORAGE_VOL on an RBD pool, refresh the pool and look the volume up once more. That is what getVolume() did for this path before apache#14188, and the refresh only happens when the volume is not cached. Other errors and other pool types are handled as before. Signed-off-by: Brad House <bhouse@nexthop.ai> (cherry picked from commit cf319a0)

boring-cyborg Bot added the component:kvm label Sep 17, 2026

bhouse-nexthop force-pushed the kvm-rbd-avoid-pool-refresh branch from 89d9f50 to a3a511b Compare September 17, 2026 02:27

bhouse-nexthop force-pushed the kvm-rbd-avoid-pool-refresh branch from a3a511b to d74bf10 Compare September 17, 2026 03:16

bhouse-nexthop requested review from DaanHoogland and weizhouapache September 18, 2026 00:51

weizhouapache added the component:ceph label Sep 18, 2026

bhouse-nexthop requested review from sureshanaparti and vladimirpetrov October 1, 2026 13:03

weizhouapache approved these changes Oct 1, 2026

View reviewed changes

weizhouapache requested a lite review from Copilot October 8, 2026 09:05

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Copilot stopped reviewing on behalf of weizhouapache due to an error October 8, 2026 10:14
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Damans227 approved these changes Oct 9, 2026

View reviewed changes

weizhouapache merged commit 1805dde into apache:4.22 Oct 9, 2026
41 of 43 checks passed

weizhouapache mentioned this pull request Oct 16, 2026

KVM: volume migration on Ceph RBD, and between RBD, NFS, SharedMountPoint, local and CLVM pools #14426

Draft

2 of 14 tasks

This was referenced Nov 9, 2026

KVM: resizing an RBD volume fails with "Storage volume not found" until libvirt refreshes the pool #14622

Open

kvm: refresh the RBD pool when resizing a volume libvirt has not cached #14623

Open

This was referenced Nov 9, 2026

Allow growing a running KVM VM's ROOT volume through a new disk offering #14602

Open

kvm: stop a stuck rbd_open from wedging the agent #14199

Open

kvm: look RBD volumes up through librbd instead of refreshing the whole pool by bhouse-nexthop · Pull Request #14188 · apache/cloudstack · GitHub

Repository navigation

kvm: look RBD volumes up through librbd instead of refreshing the whole pool - #14188

kvm: look RBD volumes up through librbd instead of refreshing the whole pool#14188weizhouapache merged 1 commit intoapache:4.22apache/cloudstack:4.22from bhouse-nexthop:kvm-rbd-avoid-pool-refreshbhouse-nexthop/cloudstack:kvm-rbd-avoid-pool-refreshCopy head branch name to clipboard

Conversation

bhouse-nexthop commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fix

What this does not fix

Connection handling

Types of changes

How Has This Been Tested?

Uh oh!

codecov Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

weizhouapache commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhouse-nexthop commented Oct 1, 2026

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kvm: look RBD volumes up through librbd instead of refreshing the whole pool#14188
weizhouapache merged 1 commit into
apache:4.22from
bhouse-nexthop:kvm-rbd-avoid-pool-refresh

bhouse-nexthop commented Sep 17, 2026 •
edited
Loading

codecov Bot commented Sep 17, 2026 •
edited
Loading

blueorangutan commented Sep 22, 2026 •
edited
Loading