kvm: stop a stuck rbd_open from wedging the agent by bhouse-nexthop · Pull Request #14199 · apache/cloudstack · GitHub

Description

A VM deploy wedged a KVM agent thread for 7.8 hours and took the cluster's VM creation with it. The thread was cloning a template on RBD and never returned.

Stack from the agent, with the native frames underneath:

"AgentRequest-Handler-28"   last log line: Creating volume <uuid> from template <uuid>
  KVMStorageProcessor.cloneVolumeFromBaseTemplate
  LibvirtStorageAdaptor.createDiskFromTemplateOnRBD
  com.ceph.rbd.Rbd.open -> JNA -> rbd_open

  rbd_open + 366                                  librbd.so.1.19.0
  std::condition_variable::wait(unique_lock&)     the overload with no timeout
  __pthread_cond_wait_2_0                         timeout = NULL

Ceph was healthy the whole time: HEALTH_OK, every OSD up, no watchers on the template, no blocklisted clients, and the same template opened instantly with rbd on that same host.

The user visible effect was that new VM deploys kept failing with InsufficientServerCapacityException, on a cluster where every host had terabytes of free memory. That error is what a deploy reports when the planner ends up with no usable destination, whatever the reason, so it sent us looking at capacity for a while before the real cause turned up in an agent thread dump.

Three things in the KVM plugin turned one stuck call into an outage:

Problem	Detail
Nothing bounds the call	librados leaves rados_osd_op_timeout and rados_mon_op_timeout at 0, which means wait forever
Handles are never released	Rados only calls rados_shutdown() from finalize(), and nothing in the plugin called shutDown(), so every handle waited for the garbage collector
Cleanup only on the happy path	The clone path had no finally at all, so a failure part way through leaked the open image and the IO context

The agent runs a host's storage commands in sequence, so the stuck thread also blocked every command queued behind it for that host.

Fix

1. Bound the operations. A new CephUtil.connect() applies per-operation timeouts to every handle:

property	default	effect
rados.client.mount.timeout	30	unchanged — the value the code already passed
rados.osd.op.timeout	60	an OSD operation fails after 60s instead of waiting forever
rados.mon.op.timeout	30	a monitor operation fails after 30s instead of waiting forever

The operation timeouts are on by default. Left at 0, the hang still happens for anyone who doesn't know to set them, and a failed operation can be retried while a wedged host can't. Setting either to 0 leaves the librados option unset and keeps the old wait-forever behaviour, and a cluster whose single operations legitimately run longer, for example during heavy backfill or recovery, can raise them. agent/conf/agent.properties documents both.

They limit a single operation and not a whole request, so a long copy or flatten is made up of many operations that are each well inside the limit.

2. Release what we open. CephUtil adds non-throwing closeQuietly, ioCtxDestroyQuietly and shutDownQuietly, and every path cleans up from a finally.

	before	after
Rados handles created	11	11, all through CephUtil.connect()
Rados handles released	0	11
Clone path cleanup	none	image, then IO context, then handle
Snapshot backup closed its image	no	yes

3. Open read only where we only read. A read write open registers a watcher on the image header and lets the image take its exclusive lock. A clone parent needs neither:

// before
RbdImage srcImage = rbd.open(template.getName());

// after
RbdImage srcImage = rbd.openReadOnly(template.getName());

Creating and protecting the base snapshot are writes, so the first clone of a template reopens it read write for that one step. Every later clone of that template keeps the read only handle. The same change applies to the cross cluster copy and to reading a snapshot during backup. Paths that really do write, such as snapCreate, snapUnprotect and snapRemove, are unchanged.

4. Report failures that were silent. Wrong regardless, and reachable the moment a timeout is set:

rbd_read returns a negative errno rather than throwing, and the snapshot backup loop treated that as end of image — a short file on secondary storage, recorded as a good backup. A read or write error now fails the answer.
A failed RBD snapshot create or remove logged and then returned a success answer, recording a snapshot in CloudStack with nothing behind it on the cluster.
The cephx secret is null for a pool with no cephx user, and librados aborts the process rather than returning an error if it is handed one. Now guarded, as getPhysicalDisk already did.

What this does not fix

The thread was waiting on an internal librbd completion that never arrived, so a per operation timeout may not reach that particular wait. Where the timeouts do reach it, this change lets the agent recover and keeps one stuck call from blocking a host's whole command queue. It does not fix librbd.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

How Has This Been Tested?

mvn -pl core,agent,plugins/hypervisors/kvm test builds clean and the existing unit tests pass.
The behaviour above was traced on a production KVM and Ceph cluster: agent thread dump, kernel wchan and syscall state showing an untimed futex wait, and the librbd frames recovered by matching return addresses on the stuck thread's stack against a disassembly of the same build.
Both operation timeouts now default to on (60s OSD, 30s mon). Setting them to 0 leaves the librados options unset exactly as before.
Unit tests were added asserting that the cluster handle is shut down on the success path, on a snapshot failure, and when opening the image fails.
Not yet exercised against a live clone on a patched agent.

A clone from a template opens the template with rbd_open(). librados leaves rados_osd_op_timeout and rados_mon_op_timeout at 0, which means an operation waits forever, so a completion that never arrives parks the agent thread that issued it for the life of the process. The agent runs a host's storage commands in sequence, so every command queued behind that thread stops with it. Nothing released the cluster handles either. Rados only calls rados_shutdown() from finalize(), so a handle the KVM plugin opened stayed alive until the garbage collector happened to reach it, and then shut down on the finalizer thread rather than on the thread that did the work. Several paths also leaked the IO context and the open image when a call failed part way through, because the clean up ran only on the success path, and the snapshot backup never closed its image at all. Add CephUtil, which opens a connected handle with the timeouts applied and offers non-throwing helpers to close an image, destroy an IO context and shut a handle down. Route every handle in the plugin through it and clean up from finally blocks, so that each path releases what it opened. The timeouts bound a single operation rather than a whole request, so a long copy or flatten is made up of many operations that are each well inside the limit. They are exposed as rados.osd.op.timeout and rados.mon.op.timeout in agent.properties; setting either to 0 restores the previous behaviour of waiting forever. Signed-off-by: Brad House <bhouse@nexthop.ai>

A read write open registers a watcher on the image header and lets the image take its exclusive lock. A clone parent, and an image that is only read or stat'ed, need neither, and the extra work is done inside rbd_open() while the caller waits. Open the template read only when cloning from it, on both the same cluster and the cross cluster path, and open the snapshot read only when backing it up. The base snapshot still has to be created and protected on a writable handle, so the first clone of a template reopens it read write for that step alone; every later clone of the same template finds the snapshot and keeps the read only handle. Signed-off-by: Brad House <bhouse@nexthop.ai>

Codecov Report

❌ Patch coverage is 8.50340% with 269 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (637df4a).

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	121 Missing ⚠️
...ava/com/cloud/hypervisor/kvm/storage/CephUtil.java	9.09%	58 Missing and 2 partials ⚠️
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java	9.37%	29 Missing ⚠️
...e/wrapper/LibvirtManageSnapshotCommandWrapper.java	0.00%	18 Missing ⚠️
...e/wrapper/LibvirtBackupSnapshotCommandWrapper.java	0.00%	17 Missing ⚠️
...e/wrapper/LibvirtRevertSnapshotCommandWrapper.java	0.00%	17 Missing ⚠️
.../hypervisor/kvm/storage/KVMStoragePoolManager.java	65.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               4.22   #14199    +/-   ##
==========================================
  Coverage     18.32%   18.32%            
- Complexity    16644    16656    +12     
==========================================
  Files          5939     5940     +1     
  Lines        537768   537869   +101     
  Branches      65929    65939    +10     
==========================================
+ Hits          98551    98586    +35     
- Misses       428008   428070    +62     
- Partials      11209    11213     +4

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.42% <8.50%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Correctness fixes to the two preceding commits: - deleteSnapshot reported success when the image could not be opened. Moving the connect and open inside the inner try put rbd_open() failures into a catch that only logs, so the method fell through to its success return. The management server would drop the snapshot record while the snapshot stayed on the cluster, and a protected one then blocks removal of its parent volume. The open is back outside that catch. - The template reopen closed the read only handle and left the field pointing at it, so a failure of the reopen sent the already freed pointer to rbd_close() a second time in the finally. That is a use after free in librbd, which takes the agent down rather than raising an exception. The field is cleared first. - CephUtil.connect() abandoned the cluster handle if anything between rados_create() and the return threw, which is the leak the class exists to prevent, and the new connect timeout makes that path ordinary rather than rare. It now releases the handle before rethrowing. - The cephx secret is null for a pool with no cephx user, and librados aborts the process rather than returning an error if it is handed one. Guard as getRbdPhysicalDisk already does. Failures that were silent are now reported, which matters because a timeout makes them reachable: - rbd_read returns a negative errno rather than throwing, and the snapshot backup loop treated that as end of image: a short file on secondary storage, recorded as a good backup. It now fails, as does a write error, which was swallowed too. - A failed RBD snapshot create or remove logged and then returned a success answer, recording a snapshot in CloudStack with nothing behind it on the cluster. Also: client_mount_timeout now honours 0 like the other two, ioCtxDestroyQuietly guards the handle it dereferences, the constants left unused by the previous commits are removed, the three properties are documented in agent.properties, and the RBD cleanup tests assert the handle is shut down. Signed-off-by: Brad House <bhouse@nexthop.ai>

Default rados.osd.op.timeout and rados.mon.op.timeout to 0, which leaves the options unset and keeps the librados behaviour of waiting forever. Nothing about how an existing deployment talks to Ceph changes unless an operator sets them. rados.client.mount.timeout keeps its default of 30, which is the value the code passed before it became a property. agent.properties carries the reasoning and a suggested starting point, since an operator has no way to pick a value from the property name alone. Signed-off-by: Brad House <bhouse@nexthop.ai>

…ccess Closing an RBD image is where librbd flushes, so a close that fails on an image just written is the last chance a lost write has to surface. Routing those two closes through the non-throwing helper turned that into a warning and returned the disk anyway, which would register a volume whose tail writes never landed. Close the written destination explicitly and let it fail the copy; the finally still covers the paths that did not get that far. deleteSnapshot had the same shape: a failed snapUnprotect or snapRemove was logged and the method still answered "removed successfully", dropping the record while the snapshot kept pinning space on the cluster, and one left protected also blocks removal of its parent volume. It now fails, except for ENOENT, where the snapshot is already gone and the delete has nothing left to do. The same ENOENT tolerance keeps the RBD branch of ManageSnapshotCommand idempotent for deletes. Also corrects the applyTimeouts javadoc, which said 0 means waiting forever for all three options; that is true of the two operation timeouts but the connect timeout falls back to the librados default of 300s. The snapshot test now pins the unwind order, since destroying an IO context after its cluster handle has been shut down would be a use after free. Signed-off-by: Brad House <bhouse@nexthop.ai>

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19300

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-17029)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50917 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14199-t17029-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

@vladimirpetrov @sureshanaparti could you take a look at this one? We'd like this fix to make it into the upcoming 4.22.2 release.

code lgtm

run 156 smoke tests with this PR on an environment with 2 ceph pools, no issues found

with both timeouts at 0 by default, does anything here stop the original hang out of the box? looks like the agent would still wait forever on that rbd_open

Yes, I measured it. I fed the rules AclDevice generates through the unmodified CsNetfilters.compare(), against real iptables (1.8.10, nf_tables backend) in a throwaway network namespace. The test ACL has one TCP rule with N CIDRs, followed by a single-CIDR allow and a deny. Times are wall clock, with the number of iptables processes run:

N before, first apply before, re-apply after, first apply after, re-apply

100 0.02s / 8 1.5s / 106 0.3s / 107 2.0s / 205

300 0.03s / 8 5.1s / 306 1.0s / 307 6.6s / 605

1000 rule not applied (iptables: Message too long) - 9.8s / 1007 43.9s / 2005

Generating the rules in Python takes about 1ms either way.

Most of this cost was already there before the PR. has_rule() never matches a rule that has a position, so on every apply each ACL rule is inserted again and its old copy is deleted as unseen. Those old copies were already one rule per CIDR, because iptables had expanded -s a,b,c. So a re-apply was already 1 insert + N deletes. With this PR it is N inserts + N deletes. The deletes are the slow part: about 17ms each at N=300, against about 3ms per insert, because each call reads the whole ruleset. At N=300 this PR adds roughly 1 to 1.5s to an apply that already took 5s. That is also exactly what the current workaround costs, since N single-CIDR rules produce the same iptables calls.

The real fix is to batch the changes through iptables-restore --noflush. That way, 300 inserts plus 300 deletes take about 0.02s. It would change how every iptables rule on the VR is applied, though:

Failure handling changes: one bad line fails the whole batch.

It needs chunking: a 1000-rule batch also fails with Message too long.

So I'd rather keep it out of this bug fix and do it as a separate PR. I'm happy to open one.

it looks @bhouse-nexthop replied to a wrong PR

@Damans227 you're right, out of the box this doesn't bound that wait. With both timeouts at 0, CephUtil leaves the librados options unset, and the command still runs on the agent request thread, so an rbd_open that never completes still parks it and the host's queue behind it.

Leaving them at 0 was deliberate. A single value that fits every cluster is hard to pick, and a timeout that's too tight turns a slow but healthy operation during backfill or recovery into a failed deploy or snapshot. So I made it opt in and put a suggested starting point in agent.properties (60s osd / 30s mon).

What does apply by default: the clone parent is now opened read only, so rbd_open no longer registers a header watch or sets up the exclusive lock. That is the kind of internal completion the stuck thread was waiting on. Cluster handles are released deterministically, and the silent failures (short reads, failed snapshot create/remove) are now reported. None of that guarantees the hang can't happen again, though. Only the timeouts bound it, and as the description says, even they may not reach every internal librbd wait.

If you and the other reviewers would prefer non-zero defaults, I'm happy to switch to 60/30. I've also fixed the description, which overstated what happens without the timeouts set.

@weizhouapache @Damans227 this is waiting on one decision: whether the two Ceph timeouts should stay at 0 (opt in, as now) or default to 60s osd / 30s mon (see my reply above on 10-14). I'm happy to switch the defaults today if you'd prefer that. Once that's settled, could it go into 4.22.2? I understand the release is due late this week or early next.

whether the two Ceph timeouts should stay at 0 (opt in, as now) or default to 60s osd / 30s mon

@bhouse-nexthop lets default to 60s osd / 30s mon. at 0 the hang still happens for anyone who doesnt know to set them, and a failed deploy can be retried while a stuck host cant. anyone with a slow cluster can still raise them or set 0.

Default rados.osd.op.timeout to 60 and rados.mon.op.timeout to 30. With the options left unset, an operation that never completes still parks the agent thread, and with it the host's storage command queue, for anyone who does not know to set them. A failed operation can be retried; a wedged host cannot. Setting either property to 0 still leaves the option unset and keeps the librados behaviour of waiting forever, and a cluster whose single operations legitimately run longer can raise them. Signed-off-by: Brad House <bhouse@nexthop.ai>

@Damans227 agreed, done in 85de0b5. rados.osd.op.timeout now defaults to 60 and rados.mon.op.timeout to 30. Setting either to 0 still leaves the librados option unset (wait forever), and a slow cluster can raise them. agent.properties, the AgentProperties javadoc and the PR description are updated to match.

agreed, done in 85de0b5

@bhouse-nexthop that covers it

thanks @bhouse-nexthop, the new defaults look right. CLGTM

code lgtm

run 156 smoke tests with this PR on an environment with 2 ceph pools, no issues found

@weizhouapache , do we considder this as tested with this?

code lgtm
run 156 smoke tests with this PR on an environment with 2 ceph pools, no issues found

@weizhouapache , do we considder this as tested with this?

I have not tested the latest changes

code lgtm
run 156 smoke tests with this PR on an environment with 2 ceph pools, no issues found

@weizhouapache , do we considder this as tested with this?

I have not tested the latest changes

ok, can you re-run against your ceph env?

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 19772

Force-pushed to fix the author and committer identity on the last commit ("kvm: bound Ceph operations by default"). It had been committed with a placeholder git identity. There is no code change: every commit's tree is identical to before, and the head is now 6ae707f. @weizhouapache, your pending re-test applies to this head the same way.

KVMStoragePoolManager.getPhysicalDisk() retries any failure 100 times, 3 s apart, to wait for a volume created on another host to show up. With RBD lookups going through librbd and Ceph operations now bounded, a Ceph outage makes each try fail with -110 after rados.osd.op.timeout, and the loop turns that into about 100 x 63 s, roughly 105 minutes, during which the command holds an agent handler thread. A lab test with the OSD ports blocked saw this on deploy (cloneVolumeFromBaseTemplate) and on createSnapshot. Only librbd's -ENOENT means the image is not there yet. Fail at once when the cause chain carries any other RbdException, or any RadosException (a connect failure, or -ENOENT for a pool that is gone). Errors that are not from Ceph are retried as before. Signed-off-by: Brad House <bhouse@nexthop.ai>

Signed-off-by: Brad House <bhouse@nexthop.ai>

getRbdPhysicalDisk came in with apache#14188 after this change branched, so it still built its own Rados handle with only client_mount_timeout set. The lookup that runs on every deploy and VM start would then block forever during an OSD outage. Open through CephUtil.connect(), so it gets the same operation timeouts as every other Ceph handle in the plugin, and release the IO context and the cluster handle from the finally with the non-throwing helpers, so a failure to release cannot mask the exception on its way out. The cephx guard that this method carried is now inside CephUtil.connect() and covers every caller. Signed-off-by: Brad House <bhouse@nexthop.ai>

Test report: Ceph outages on a KVM host, with the agent's Ceph traffic blocked

Environment

A CloudStack 24.0 (main) build with this PR applied, alongside some of my other open PRs. That includes kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 (merged), which looks RBD volumes up through librbd.
4.22 caveat: I ran this on 24.0, not 4.22. The KVM storage code this PR touches (LibvirtStorageAdaptor, KVMStorageProcessor, KVMStoragePoolManager and the snapshot wrappers) is the same on 4.22 and main, so I expect the same behaviour there. I have not run it on a 4.22 build.
Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested virtualisation. Ceph squid 19.2.3, RBD primary storage, NFS secondary storage.
The agent's defaults: rados.osd.op.timeout=60, rados.mon.op.timeout=30.

How the fault was injected

An nftables rule on the KVM host drops only the traffic from the cloudstack-agent.service cgroup to Ceph:
```
nft add rule inet cephblk out socket cgroupv2 level 2 '"system.slice/cloudstack-agent.service"' ip daddr <ceph> tcp dport '{ 6800-7568 }' counter drop
```
Steps 1, 2 and 5 drop the OSD ports only, so rados_connect succeeds and the image open waits on the OSDs. Step 4 also drops the mon ports.
qemu runs in machine.slice and is not matched, so running guests keep their disk I/O. I checked this before the run, and again after every block by watching the guests' I/O request counters grow.
Processes the agent starts, such as qemu-img, inherit its cgroup and are blocked too.
Ceph was HEALTH_OK after every step, and all hosts stayed Up throughout.

History

The first lab run of this PR found that each Ceph operation was bounded, but KVMStoragePoolManager.getPhysicalDisk() retried any failure 100 times, 3 s apart. A deploy with the OSDs blocked failed one open every ~63 s and would have held the command for about 105 minutes. A snapshot behaved the same way.
The PR now treats a Ceph error other than librbd's -ENOENT as final ("not retrying a Ceph error"). Everything below is on that version.

1. OSD ports blocked, deploy a VM to the host

The deploy failed after 67 s, with Unable to orchestrate the start of VM instance ....

The agent log shows one bounded open, and no retry:

07:12:24 Trying to fetch storage pool <pool> from libvirt
07:13:24 ERROR A Ceph RBD operation failed (-110). The error was: Failed to open image <template> - Connection timed out
07:13:24 WARN  [KVMStoragePoolManager] Failed to find volume:<template> due to ...RbdException: Failed to open image <template>, not retrying a Ceph error

At 45 s, jstack showed the deploy's handler thread in rbd_open_read_only, as expected while the open waits. A second handler was there too, running a GetVolumeStatsCommand from the stats collector. That one also came back after its single bounded open (answered at 56 s). Neither command held the other up, and the host stayed Up.
After I removed the block, a fresh deploy to the same host succeeded.

2. "Without": the timeouts set to 0, which is the stock behaviour

I set rados.osd.op.timeout=0 and rados.mon.op.timeout=0 in agent.properties, restarted the agent, and repeated step 1.
The deploy's handler thread was parked in rbd_open_read_only at 70 s and again at 370 s: the same thread, with its CPU time unchanged (AgentRequest-Handler-5 #56 ... cpu=58.26ms elapsed=96s, then elapsed=397s). At 430 s, a second handler (a later command) was parked the same way.
The agent process was the same throughout; nothing restarted it.
When I removed the block after ~450 s, the parked open completed and the VM came up Running. That is the stock "wait for as long as Ceph is away".
I put the defaults back and restarted the agent afterwards.

3. Snapshot of a running VM's ROOT with the OSD ports blocked

createSnapshot failed after 62 s. The agent logged the same single -110 open and the "not retrying a Ceph error" line, then Failed take snapshot for volume ....
listSnapshots shows the snapshot in Error, not BackedUp, and rbd snap ls on the image is empty, so there is no stray RBD snapshot.
Once unblocked, a new snapshot of the same ROOT reached BackedUp in 4 s.

4. Snapshot backup interrupted mid-copy

A 20 GB data disk, filled from /dev/urandom. I used 20 GB rather than 40 GB to stay clear of the pool's nearfull ratio.

snapshot.backup.to.secondary=true. I started createSnapshot, and found the copy by watching for its process on the hosts:

qemu-img convert -O qcow2 -U rbd:cloudstack/<image>@<snap>:mon_host=...:client_mount_timeout=30 /mnt/<secondary>/snapshots/...
cgroup 0::/system.slice/cloudstack-agent.service

With 7.0 GB copied (and growing at ~240 MB/s), I blocked all of the agent cgroup's Ceph ports on that host for 180 s.
- The copy stalled at 7045709824 bytes for the whole window, and qemu-img stayed alive.
- After the block was removed, qemu-img resumed and finished 85 s later.
The snapshot ended BackedUp with a full-size file: 21478375424 bytes, qemu-img info virtual size 20 GiB, matching physical_size in snapshot_store_ref. No short file was recorded.
Note on scope: this backup path is KVMStorageProcessor.backupSnapshot(), which copies with qemu-img convert. This PR doesn't change that code, and qemu-img gets no op timeout, so a longer outage would hold the copy until backup.snapshot.wait.
The PR's read-loop fix (a negative rbd_read is now an error instead of end-of-image) is in LibvirtBackupSnapshotCommandWrapper. I found no current management-server path that sends BackupSnapshotCommand, so that change is covered by the unit tests only.

5. Watchers on the template image (from the first run, same code for this part)

I sampled rbd status on the template image every ~0.2 s while 6 VMs were deployed from it at once, 3 to each of two hosts.
No watcher appeared, and all 6 VMs were Running. The clone parent is opened read-only.

6. Cluster handles released (from the first run, same code for this part)

ceph tell mon.* sessions entries from the host: 22 before and 22 after 10 deploy/destroy cycles on it.

Result: pass.

With the OSDs unreachable, a deploy and a snapshot now fail in about one rados.osd.op.timeout (67 s and 62 s), instead of waiting for as long as Ceph is away.
Other commands on the host keep running, and the host stays Up.
With the timeouts set to 0, the agent goes back to the stock behaviour, with a handler parked in rbd_open for the whole outage.
A failed snapshot leaves no RBD snapshot behind and is not recorded as created. An interrupted backup did not record a short file.
Clone opens hold no watchers, and the cluster handles are released.

Not covered

A mon-only outage during a long copy, the internal librbd completion hang, a cross-cluster copy, and a pool without cephx.
BackupSnapshotCommand (see step 4).

bhouse-nexthop added 2 commits September 18, 2026 10:55

boring-cyborg Bot added component:agent component:kvm labels Sep 18, 2026

weizhouapache added the component:ceph label Sep 18, 2026

bhouse-nexthop added 3 commits September 18, 2026 11:27

bhouse-nexthop requested review from DaanHoogland and weizhouapache September 18, 2026 12:02

bhouse-nexthop requested review from sureshanaparti and vladimirpetrov October 1, 2026 13:03

weizhouapache approved these changes Oct 7, 2026

View reviewed changes

bhouse-nexthop requested a review from Damans227 October 23, 2026 13:53

Damans227 approved these changes Oct 23, 2026

View reviewed changes

bhouse-nexthop force-pushed the kvm-rbd-bound-ceph-ops branch from 85de0b5 to 6ae707f Compare November 8, 2026 19:04

bhouse-nexthop added 3 commits November 9, 2026 06:23

Merge branch '4.22' into kvm-rbd-bound-ceph-ops

58ca859

Signed-off-by: Brad House <bhouse@nexthop.ai>

kvm: stop a stuck rbd_open from wedging the agent by bhouse-nexthop · Pull Request #14199 · apache/cloudstack · GitHub

Repository navigation

kvm: stop a stuck rbd_open from wedging the agent - #14199

kvm: stop a stuck rbd_open from wedging the agent#14199bhouse-nexthop wants to merge 9 commits intoapache:4.22apache/cloudstack:4.22from bhouse-nexthop:kvm-rbd-bound-ceph-opsbhouse-nexthop/cloudstack:kvm-rbd-bound-ceph-opsCopy head branch name to clipboard

Conversation

bhouse-nexthop commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fix

What this does not fix

Types of changes

How Has This Been Tested?

Uh oh!

codecov Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

weizhouapache commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 21, 2026

Uh oh!

blueorangutan commented Sep 22, 2026

Uh oh!

bhouse-nexthop commented Oct 1, 2026

Uh oh!

weizhouapache left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Damans227 commented Oct 14, 2026

Uh oh!

weizhouapache commented Oct 14, 2026

Uh oh!

bhouse-nexthop commented Oct 14, 2026

Uh oh!

bhouse-nexthop commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Damans227 commented Oct 22, 2026

Uh oh!

bhouse-nexthop commented Oct 23, 2026

Uh oh!

Damans227 commented Oct 23, 2026

Uh oh!

Damans227 left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Oct 26, 2026

Uh oh!

weizhouapache commented Oct 26, 2026

Uh oh!

DaanHoogland commented Oct 26, 2026

Uh oh!

weizhouapache commented Oct 27, 2026

Uh oh!

blueorangutan commented Oct 27, 2026

Uh oh!

blueorangutan commented Oct 27, 2026

Uh oh!

bhouse-nexthop commented Nov 8, 2026

Uh oh!

bhouse-nexthop commented Nov 9, 2026

Test report: Ceph outages on a KVM host, with the agent's Ceph traffic blocked

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kvm: stop a stuck rbd_open from wedging the agent#14199
bhouse-nexthop wants to merge 9 commits into
apache:4.22from
bhouse-nexthop:kvm-rbd-bound-ceph-ops

bhouse-nexthop commented Sep 18, 2026 •
edited
Loading

codecov Bot commented Sep 18, 2026 •
edited
Loading

weizhouapache left a comment •
edited
Loading

bhouse-nexthop commented Oct 22, 2026 •
edited
Loading