Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper - #14006
Open
abh1sar wants to merge 1 commit into
Open
Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper#14006abh1sar wants to merge 1 commit into
abh1sar wants to merge 1 commit into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
…up restore Script.executeCommand returns null when the command fails, it does not throw, so the try/catch around the mount and umount of the backup repository could never fire and the return value was discarded. A repository that fails to mount was therefore treated as mounted, and the restore carried on against an empty directory until it failed later with a misleading "backup file not found". A failed umount was ignored the same way, leaking the mount. Both now go through executeCommandForExitValue and check the exit value. The same refactor also dropped the timeouts. mountTimeout was still passed into mountBackupDirectory but never used, and the rsync of the volume lost the command timeout, so both fell back to the one hour default in Script instead of the configured values. An unresponsive repository could hold a restore up for an hour rather than failing after nas.backup.restore.mount.timeout seconds.
Copy link
Copy Markdown
Codecov Report❌ Patch coverage is 75.00000% with 3 lines in your changes missing coverage. Please review.
@@ Coverage Diff @@
## 4.22 #14006 +/- ##
=========================================
Coverage 17.79% 17.79%
- Complexity 15995 15998 +3
=========================================
Files 5928 5928
Lines 534306 534315 +9
Branches 65383 65385 +2
=========================================
+ Hits 95069 95087 +18
+ Misses 428467 428457 -10
- Partials 10770 10771 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness.
|
Sorry, something went wrong.
Copy link
Copy Markdown
|
@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19015 |
Sorry, something went wrong.
abh1sar
requested review from
shwstppr and
weizhouapache
and
a lite review from Copilot
August 29, 2026 16:15
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes restore-from-backup behavior for KVM/libvirt by correctly treating failed mount/umount operations as failures (via exit codes instead of relying on exceptions) and by restoring the intended command timeouts so restores don’t hang for Script’s 1-hour default.
Changes:
- Switch mount to Script.executeCommandForExitValue(timeout, ...) and fail fast on non-zero exit codes.
- Switch umount to executeCommandForExitValue(...) and fail on non-zero exit codes (instead of silently ignoring failure).
- Ensure rsync uses the configured restore command timeout; extend unit tests and add a new timeout-focused test.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java | Validates mount/umount exit codes and restores rsync timeout usage during restore operations. |
| plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java | Updates mocks for timeout-aware executeCommandForExitValue and adds a test asserting the configured mount timeout is used. |
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java:605
- This test attempts to capture the timeout used for the mount command, but the "mount" detection has the same issue as other varargs stubs: invocation.getArguments() contains (timeout, String[] cmd), and String.valueOf(cmd) won't equal "mount". As a result, mountTimeout[0] is never set and the assertion can fail (or the test can pass without actually checking the mount invocation, depending on defaults).
if (Arrays.stream(invocation.getArguments()).map(String::valueOf).anyMatch("mount"::equals)) {
mountTimeout[0] = invocation.getArgument(0);
return 1; // stop the restore right after the mount
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sorry, something went wrong.
Comment thread
.../main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java
Comment on lines
+230
to
+234
| if (exitValue != 0) { | ||
| logger.error("Failed to mount repository {} of type {} to the directory {}, mount exited with {}", backupRepoAddress, | ||
| backupRepoType, mountDirectory, exitValue); | ||
| throw new CloudRuntimeException("Failed to mount the backup repository on the KVM host"); | ||
| } |
Comment thread
.../main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java
Comment on lines
241
to
244
| String umountPath = Script.getExecutableAbsolutePath("umount"); | ||
| String[] umountCmd = new String[] { "sudo", umountPath, backupDirectory }; | ||
| Script.executeCommand(umountCmd); | ||
| exitValue = Script.executeCommandForExitValue(umountCmd); | ||
| } catch (Exception e) { |
Comment thread
...t/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java
Comment on lines
409
to
410
| if (Arrays.stream(invocation.getArguments()).map(String::valueOf).anyMatch("rsync"::equals)) { | ||
| return 1; // Rsync failure |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Script.executeCommand returns null when the command fails, it does not throw, so the try/catch around the mount and umount of the backup repository could never fire and the return value was discarded.
A repository that fails to mount was therefore treated as mounted, and the restore carried on against an empty directory until it failed later with a misleading "backup file not found". A failed umount was ignored the same way, leaking the mount. Both now go through executeCommandForExitValue and check the exit value.
The same refactor also dropped the timeouts. mountTimeout was still passed into mountBackupDirectory but never used, and the rsync of the volume lost the command timeout, so both fell back to the one hour default in Script instead of the configured values. An unresponsive repository could hold a restore up for an hour rather than failing after nas.backup.restore.mount.timeout seconds.
This is a regression from 56ad044
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Verified that restore fails after the specified nas.backup.restore.mount.timeout with a dead address.
How did you try to break this feature and the system with this change?