[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail. by daviftorres · Pull Request #13424 · apache/cloudstack · GitHub

Description

This PR tried to prevent the failure of the job at the statistics section of a backup that has actually succeeded.

Apparently, it also fixes some silent failures I previously reported in #11727

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

This is the equivalent command for applying the fix:

sed -i 's_du -sb $dest | cut -f1_du -sb $dest 2>/dev/null | cut -f1 || true_g' /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh

We haven't confirmed the exact root cause of the du failure yet. As a precaution, we applied this fix to all servers and will monitor backups over the next few days.

So, I am running tests with 2>>/var/log/cloudstack/agent/nasbackup.err so I can see what is the error message.

Proposed Changes Rationale

backup_size=$(du -sb "$dest" 2>/dev/null | cut -f1) || true

NFS issues may cause du command to fail.
A size retrieval failure should not invalidate a successful backup.

timeout 60 umount "$mount_point" 2>/dev/null || true
rmdir "$mount_point" 2>/dev/null || true

Another process may keep the device busy (e.g., parallel backups).
Network issues may cause hangs on NFS.
Cleanup failures should not invalidate a successful backup.

echo -n "$backup_size"

Outputs the size at the end to confirm the script completed past the potentially problematic commands.

Dear @abh1sar , do you think you can help me with this bug? Regards,

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.93%. Comparing base (5ed4894) to head (664b2d5).
⚠️ Report is 42 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13424      +/-   ##
============================================
- Coverage     18.94%   18.93%   -0.01%     
- Complexity    18376    18473      +97     
============================================
  Files          6192     6221      +29     
  Lines        556550   560039    +3489     
  Branches      67954    68285     +331     
============================================
+ Hits         105454   106066     +612     
- Misses       439517   442347    +2830     
- Partials      11579    11626      +47

Flag	Coverage Δ
uitests	3.50% <ø> (-0.02%)	⬇️
unittests	20.14% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Pull request overview

This PR adjusts the KVM NAS backup script’s “statistics/cleanup” section so that failures while computing backup disk usage (and related cleanup commands) don’t cause an otherwise successful backup job to be marked as failed.

Changes:

Capture du output into backup_size and suppress du stderr to avoid failing the script during size calculation.
Add timeout around umount and suppress errors from umount/rmdir.
Emit the computed backup size at the end of backup_running_vm().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

clgtm

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

hi @daviftorres
would it be possible to test script changes in your env where it is reproducible?
Some log outputs would be nice.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

would it be possible to test script changes in your env where it i

2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Request:Seq 218-281474976710814:  { Cmd , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-84-6493-VM","backupPath":"i-84-6493-VM/2026.06.25.13.22.42","backupRepoType":"nfs","backupRepoAddress":"10.4.2.145:/mnt/VAN3-NAS01-STOR-POOL-A1/VAN3-NAS01-DS-01","wait":"0","bypassHostMaintenance":"false"}}] }
2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Processing command: org.apache.cloudstack.backup.TakeBackupCommand
2026-06-25 13:25:04,120 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Seq 218-281474976710814:  { Ans: , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 110, [{"org.apache.cloudstack.backup.BackupAnswer":{"size":"(47.67 GB) 51190241222","result":"true","details":"Job type:         Completed

Nothing else that I can find in the logs are related to the backup job.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@blueorangutan package

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

LGTM

@blueorangutan package

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18442

@blueorangutan test

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-16468)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13424-t16468-kvm-ol8.zip
Smoke tests completed. 152 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

code lgtm

…ckup to Fail (apache#13424) (cherry picked from commit f9a9451)

boring-cyborg Bot added the component:kvm label Jun 15, 2026

github-actions Bot mentioned this pull request Jun 15, 2026

[repo-status] Daily Status Report — June 15, 2026 #13425

Closed

daviftorres marked this pull request as ready for review June 16, 2026 15:02

github-actions Bot mentioned this pull request Jun 16, 2026

[repo-status] Daily Status Report — June 16, 2026 #13435

Closed

Copilot AI review requested due to automatic review settings June 18, 2026 20:07

Copilot started reviewing on behalf of daviftorres June 18, 2026 20:07

View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

DaanHoogland approved these changes Jun 19, 2026

View reviewed changes

DaanHoogland added the status:needs-testing label Jun 19, 2026

DaanHoogland requested a review from abh1sar June 23, 2026 06:53

Copilot AI review requested due to automatic review settings June 23, 2026 12:20

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Copilot started reviewing on behalf of daviftorres June 23, 2026 12:31

View session

abh1sar reviewed Jun 23, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Copilot AI review requested due to automatic review settings June 24, 2026 20:55

Copilot started reviewing on behalf of daviftorres June 24, 2026 20:55

View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

abh1sar reviewed Jun 25, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Copilot AI review requested due to automatic review settings June 25, 2026 13:19

Copilot started reviewing on behalf of daviftorres June 25, 2026 13:21

View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Copilot AI review requested due to automatic review settings June 26, 2026 01:49

Copilot AI reviewed Jun 26, 2026

remove extra whitespace

5af17c2

Copilot AI review requested due to automatic review settings July 2, 2026 11:53

remove extra whitespace

830fbac

Copilot started reviewing on behalf of abh1sar July 2, 2026 11:53

View session

revert to echo

200c40e

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Copilot AI review requested due to automatic review settings July 2, 2026 11:56

Copilot started reviewing on behalf of daviftorres July 2, 2026 11:56

View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

abh1sar reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Resolved

Update scripts/vm/hypervisor/kvm/nasbackup.sh

cce95d4

Copilot AI review requested due to automatic review settings July 2, 2026 12:12

Update scripts/vm/hypervisor/kvm/nasbackup.sh

664b2d5

Copilot started reviewing on behalf of abh1sar July 2, 2026 12:14

View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

abh1sar approved these changes Jul 2, 2026

View reviewed changes

weizhouapache approved these changes Jul 7, 2026

View reviewed changes

weizhouapache merged commit f9a9451 into apache:main Jul 7, 2026
24 of 27 checks passed

abh1sar mentioned this pull request Jul 10, 2026

CIFS NAS Backups failing "randomly" #12122

Closed

DaanHoogland removed the status:needs-testing label Jul 20, 2026

DaanHoogland linked an issue Jul 20, 2026 that may be closed by this pull request

CIFS NAS Backups failing "randomly" #12122

Closed

This was referenced Jul 20, 2026

Creating a unmount_operation with safety checks for nasbackup.sh #12133

Closed

backport #13424 on 22? #13649

Open

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Sep 11, 2026

[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Ba…

3b61ee4

…ckup to Fail (apache#13424) (cherry picked from commit f9a9451)

[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail. by daviftorres · Pull Request #13424 · apache/cloudstack · GitHub

Repository navigation

[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail. - #13424

[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail.#13424weizhouapache merged 15 commits intoapache:mainapache/cloudstack:mainfrom daviftorres:nas-backup-faileddaviftorres/cloudstack:nas-backup-failedCopy head branch name to clipboard

Conversation

daviftorres commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

daviftorres commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daviftorres commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes Rationale

Uh oh!

daviftorres commented Jun 17, 2026

Uh oh!

codecov Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

daviftorres commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented Jun 26, 2026

Uh oh!

blueorangutan commented Jun 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail.#13424
weizhouapache merged 15 commits into
apache:mainfrom
daviftorres:nas-backup-failed

daviftorres commented Jun 15, 2026 •
edited
Loading

daviftorres commented Jun 15, 2026 •
edited
Loading

daviftorres commented Jun 17, 2026 •
edited
Loading

codecov Bot commented Jun 18, 2026 •
edited
Loading