Mask vncPasswd being logged in agent.log by sudo87 · Pull Request #12404 · apache/cloudstack · GitHub

Description

This PR fixes #10847

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Codecov Report

❌ Patch coverage is 35.71429% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.23%. Comparing base (0e6d2d9) to head (680ca79).
⚠️ Report is 12 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...resource/wrapper/LibvirtMigrateCommandWrapper.java	41.66%	6 Missing and 1 partial ⚠️
...m/resource/wrapper/LibvirtStartCommandWrapper.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.20   #12404       +/-   ##
============================================
+ Coverage     4.00%   16.23%   +12.22%     
- Complexity       0    13379    +13379     
============================================
  Files          402     5657     +5255     
  Lines        32688   499039   +466351     
  Branches      5822    60567    +54745     
============================================
+ Hits          1309    81014    +79705     
- Misses       31224   408990   +377766     
- Partials       155     9035     +8880

Flag	Coverage Δ
uitests	4.03% <ø> (+0.02%)	⬆️
unittests	17.08% <35.71%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

code lgtm

LGTM

@blueorangutan package

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Suggested change

      
                        if (logger.isDebugEnabled()) {
          
                            logger.debug(String.format("VM [%s] with XML configuration [%s] will be migrated to host [%s].", vmName, xmlDesc, target));
          
                            logger.debug("VM {} with XML configuration {} will be migrated to host {}.", vmName, maskSensitiveInfoInXML(xmlDesc), target);
          
                        }
          
                        logger.debug("VM {} with XML configuration {} will be migrated to host {}.", vmName, () -> maskSensitiveInfoInXML(xmlDesc), target);

just a suggestion, (may apply below as well)

clgtm, the isDebugEnabled are not strictly needed anymore.

we can also just remove it, btw

Wont it confuse admin whether vnc section is not part of the xml or not?

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16334

@blueorangutan test

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Thanks @DaanHoogland for your review and suggestions. Let me know if my response makes sense:

Per argument lambda is not supported, it will perform toString instead of evaluation. It wont work the way we expect:

logger.debug("VM {} with XML configuration {} will be migrated to host {}.", vmName, () -> maskSensitiveInfoInXML(xmlDesc), target);

logger.isDebugEnabled check is added to avoid maskSensitiveInfoInXML invocation. Which is not expensive operation but not required if debug is not enabled. We can definitely remove it.

Per argument lambda is not supported, it will perform toString instead of evaluation. It wont work the way we expect:
logger.debug("VM {} with XML configuration {} will be migrated to host {}.", vmName, () -> maskSensitiveInfoInXML(xmlDesc), target);

this is not how I read the documentation, but I haven’t tested it. it should lazily evaluate the lambda ony if debug is enabled.

logger.isDebugEnabled check is added to avoid maskSensitiveInfoInXML invocation. Which is not expensive operation but not required if debug is not enabled. We can definitely remove it.

I do not care if point 1 is out of the way. It is just cleaner code. If you are right about point 1 we have a bigger issue.

I do not think either should stop this PR, but in general my experience is that follow-ups don’t happen,...

[SF] Trillian test result (tid-15170)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52911 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12404-t15170-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

The PR successfully masks VNC passwords in most scenarios, but there is one missed log statement in the LibvirtMigrateCommandWrapper that still exposes the actual VNC password during all types of live migration.

Bug Summary

Affected log message: "Replaced the VNC IP address [...] with [...]"
Location: LibvirtMigrateCommandWrapper.java - replaceIpForVNCInDescFile method (or similar)

Impact: VNC password exposed in DEBUG level logs during any live migration operation

Confirmed in TC2, TC7, and TC8 across both migration directions and storage migration

Test Execution Summary

Test Case	Description	Result
TC1	VM Start - VNC Password Masking	PASSED
TC2	Live Migration - VNC Password Masking	PARTIAL
TC3	Console Access Regression	PASSED
TC4	VM Stop/Start - VNC Password Masking	PASSED
TC5	VM Reboot - VNC Password Masking	PASSED
TC6	System VM - VNC Password Masking	PASSED
TC7	Reverse Migration - VNC Password Masking	PARTIAL
TC8	Storage Migration - VNC Password Masking	PARTIAL

Detailed Test Report

Test Case 1: VM Start - VNC Password Masking

Objective Verify that the VNC password is masked (shown as *****) in the agent log when a VM is started.

Test Steps

Deploy a new VM (or start a stopped VM) in the CloudStack environment
SSH to the KVM host where the VM is running
Search the agent log for VM start entries containing VNC graphics configuration:

grep -i "Starting.*:" /var/log/cloudstack/agent/agent.log | tail -5

Search for VNC password entries in the log:

grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log

Verify no unmasked passwords exist:

grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Expected Result

The VNC password in the logged XML should be masked as passwd='*****'
No unmasked VNC passwords should appear in the agent log

Actual Result PASSED

The VNC password is correctly masked as passwd='*****'
No unmasked VNC passwords found in the log

Evidence

VM Start log entry found:

2026-01-18 16:57:28,843 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-5:[]) (logid:) Starting r-4-VM : <domain type='kvm'>

VNC password masked (grep output):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>

No unmasked passwords (empty result confirms fix):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Test Case 2: Live Migration - VNC Password Masking.

Objective Verify that the VNC password is masked (shown as *****) in the agent log when a VM is live migrated between KVM hosts.

Test Steps

Identify a running user VM and the host it is currently on
Live migrate the VM to another KVM host:
After migration completes, SSH to the source host (where VM was before migration) and search for live migration log entries:

 grep -i "live migration" /var/log/cloudstack/agent/agent.log

Search for VNC password entries:

grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log

Check for any unmasked passwords:

grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Expected Result

All VNC passwords in the logged XML should be masked as passwd='*****'
No unmasked VNC passwords should appear in any migration-related log entries

Actual Result PARTIALLY PASSED / BUG FOUND

The main migration XML configuration is correctly masked
However, the replaceIpForVNCInDescFile log message still exposes the actual VNC password

Evidence

Live migration entry (masked correctly):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -i "live migration" /var/log/cloudstack/agent/agent.log
2026-01-18 17:05:14,161 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Starting live migration of instance i-2-3-VM to destination host qemu+tls://10.1.33.95/system having the final XML configuration: <domain type='kvm'>

VNC password entries (mixed results):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:)

Replaced the VNC IP address

[<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='*****'>

Unmasked password found (bug):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:)

Replaced the VNC IP address

[<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>

Bug Details

The replaceIpForVNCInDescFile method in LibvirtMigrateCommandWrapper.java logs the old and new XML snippets when replacing the VNC IP address, but this log statement is not using the maskSensitiveInfoInXML() function, causing the actual VNC password (UzAp0PQ9) to be exposed in the agent log.

Affected log message: "Replaced the VNC IP address [...] with [...]"

Test Case 3: Console Access Regression Test

Objective Verify that the VNC password masking in logs does not break the actual VM console access functionality.

Test Steps

Open the CloudStack UI
Navigate to the test VM (i-2-3-VM)
Click on "View Console" button
Verify the console window opens and displays the VM screen
Verify keyboard input works (login to the VM)

Expected Result

Console should connect successfully
VM display should be visible
User should be able to interact with the VM (keyboard/mouse)

**Actual Result: ** PASSED

Console connected successfully via Console Proxy
VM display visible (CentOS 5.5 login screen)
Successfully logged in as root - keyboard input working

Evidence

Test Case 4: VM Stop and Start - VNC Password Masking

Objective Verify that the VNC password remains masked in the agent log when a VM is stopped and then started again.

Test Steps

Stop the test VM:
Wait for the VM to stop completely
Start the VM again:
Check the agent log on the host where the VM started:

grep -i "Starting.*:" /var/log/cloudstack/agent/agent.log | tail -5
grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Expected Result

The VNC password in the logged XML should be masked as passwd='*****'
No unmasked VNC passwords should appear in the agent log

Actual Result: PASSED

VM stopped and started successfully on kvm2
VNC password correctly masked in the start log

Evidence:

VM Start log entry found on kvm2:

2026-01-18 17:17:44,269 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-4:[]) (logid:) Starting i-2-3-VM : <domain type='kvm'>

VNC password masked:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>

No unmasked passwords found:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Test Case 5: VM Reboot - VNC Password Masking

Objective Verify that the VNC password remains masked in the agent log when a VM is rebooted.

Test Steps:

Reboot the test VM:
Check the agent log on the host where the VM is running:

Expected Result

No new StartCommand/XML log entries expected (reboot is handled internally by libvirt)
No unmasked VNC passwords should appear

**Actual Result: ** PASSED

VM reboot does not generate new StartCommand log entries
Reboot is handled internally by libvirt/QEMU without CloudStack agent involvement
No new XML configuration is logged during reboot
Existing entries remain properly masked

Evidence:

No new start entries after reboot (only TC4 entry from 17:17:44):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -i "Starting.*i-2-3-VM" /var/log/cloudstack/agent/agent.log | tail -3
2026-01-18 17:17:44,269 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-4:[]) (logid:) Starting i-2-3-VM : <domain type='kvm'>

VNC password remains masked:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>

No unmasked passwords:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log

Notes VM reboot operation is handled by libvirt's internal restart mechanism and does not trigger a new CloudStack StartCommand. This is expected behavior - the PR's masking is only relevant when CloudStack explicitly logs VM XML configurations during start or migration operations.

Test Case 6: System VM - VNC Password Masking

Objective Verify that the VNC password is also masked for System VMs (SSVM, CPVM, Virtual Router).

Test Steps

Check agent logs (including rotated logs) on kvm1 for SSVM s-1-VM:
Check agent logs (including rotated logs) on kvm2 for CPVM v-2-VM:

Expected Result:

VNC passwords for System VMs should be masked as passwd='*****'
No unmasked VNC passwords (except known TC2 bug)

Actual Result: PASSED

SSVM (s-1-VM) start log from 2026-01-14 shows masked password
CPVM (v-2-VM) start log from 2026-01-14 shows masked password
All System VM VNC entries properly masked

Evidence:

kvm1 - SSVM (s-1-VM) start found in rotated log:

/var/log/cloudstack/agent/agent.log.2026-01-14.gz:2026-01-14 14:12:47,490 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Starting s-1-VM : <domain type='kvm'>

kvm1 - VNC password masking (all logs):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# zgrep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log* | head -20
/var/log/cloudstack/agent/agent.log:<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
/var/log/cloudstack/agent/agent.log:<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
/var/log/cloudstack/agent/agent.log:    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>
/var/log/cloudstack/agent/agent.log:2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
/var/log/cloudstack/agent/agent.log:    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
/var/log/cloudstack/agent/agent.log:    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='*****'>
/var/log/cloudstack/agent/agent.log.2026-01-14.gz:<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>

(Note: Unmasked password visible is the known TC2 bug from migration)

kvm2 - CPVM (v-2-VM) start found in rotated log:

/var/log/cloudstack/agent/agent.log.2026-01-14.gz:2026-01-14 14:13:12,611 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-5:[]) (logid:) Starting v-2-VM : <domain type='kvm'>

kvm2 - VNC password masking (all logs):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# zgrep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log* | head -20
/var/log/cloudstack/agent/agent.log:<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>
/var/log/cloudstack/agent/agent.log.2026-01-14.gz:<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>

Test Case 7: Reverse Migration (kvm2 → kvm1)

Objective: Confirm that the bug found in TC2 (unmasked VNC password in "Replaced the VNC IP address" log message) is reproducible when migrating in the reverse direction.

TestSteps:

Migrate VM from kvm2 back to kvm1:
After migration completes, check agent log on kvm2 (source host):

Expected Result: All VNC passwords should be masked as passwd='*****'

Actual Result: PARTIAL - Bug Confirmed

Main migration XML configuration is properly masked
Bug reproduced: The "Replaced the VNC IP address" log message still exposes the actual VNC password

Evidence:

Live migration entry (masked correctly):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -i "live migration" /var/log/cloudstack/agent/agent.log | tail -3
2026-01-18 17:53:20,092 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Starting live migration of instance i-2-3-VM to destination host qemu+tls://10.1.34.189/system having the final XML configuration: <domain type='kvm'>

VNC password entries (mixed results):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='*****'>
2026-01-18 17:53:20,066 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>

Unmasked password confirmed (bug):

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
2026-01-18 17:53:20,066 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>

Notes: This confirms the bug identified in TC2 is consistent and reproducible in both migration directions (kvm1→kvm2 and kvm2→kvm1). The same VNC password is exposed in both cases.

Test Case 8: Storage Migration

Objective: Verify that the VNC password is masked when performing live migration with storage (migrating VM and its volumes to a different storage pool simultaneously).

Test Steps:

Check current volume location
Migrate VM with volume to different storage pool:

migrate virtualmachinewithvolume virtualmachineid=1170e952-de03-410e-8d8a-896260abf583 hostid=550f5e3f-78e6-4229-b9f4-bc0f07eca952 migrateto[0].volume=a3984f50-61fa-4d41-91a7-3d34058c3c16 migrateto[0].pool=f9d5d615-0697-31e6-9e66-978f2974640f

After migration completes, check agent log on source host (kvm1):

Expected Result:

All VNC passwords should be masked as passwd='*****'
Storage migration logs should not expose VNC passwords

Actual Result PARTIAL - Same bug as TC2/TC7

VM migrated successfully from kvm1 to kvm2 with volume moved from pri1 to pri2
Main migration XML configuration is properly masked
Bug reproduced: The "Replaced the VNC IP address" log message exposes the actual VNC password

Evidence:

VNC password entries showing the bug:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log | tail -10
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) 
Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='*****'>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>
2026-01-18 17:59:41,528 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-1:[]) (logid:) 
Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>

Unmasked passwords confirmed:

[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) 
Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
2026-01-18 17:59:41,528 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-1:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>

Notes Storage migration (17:59:41) uses the same LibvirtMigrateCommandWrapper code path as regular live migration
The same bug occurs regardless of whether storage is being migrated along with the VM
Volume successfully moved from pri1 (404b015d-...) to pri2 (f9d5d615-...)

Thank you @RosiKyu for the extremely detailed QA validation. This is very helpful.

I have addressed the issue by masking the XML in the log statement within replaceIpForVNCInDescFileAndNormalizePassword method. Please re-test TC2, TC7, and TC8 with the updated package.

@blueorangutan package

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16411

LGTM

The identified issue was addressed and password is masked in all logs.

Migrate VM from kvm2 back to kvm1:
After migration completes, check agent log on kvm2 (source host):

Expected Result: All VNC passwords should be masked as passwd='*****'

Actual Result PASSED (verified after fix)
Reverse migration also shows properly masked passwords

Evidence (after fix):

Agent update timeline on kvm1:
Agent stopped: 2026-01-19 08:50:46
Agent started: 2026-01-19 08:53:09 (with cloudstack-agent-4.20.3.0-shapeblue18646)
Before agent update (04:59:59) - password EXPOSED:

[root@ref-trl-10611-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -i "Replaced the VNC IP address" /var/log/cloudstack/agent/agent.log
2026-01-19 04:59:59,508 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.0.35.40' passwd='OjdZYxCj'>
2026-01-19 09:02:39,662 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:) Replaced the VNC IP address <graphics type='vnc' port='-1' autoport='yes' listen='10.0.35.40' passwd='*****'>

After agent update (09:02:39) - password MASKED:

[root@ref-trl-10611-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
2026-01-19 04:59:59,508 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Replaced the VNC IP address [<graphics type='vnc' port='-1' autoport='yes' listen='10.0.35.40' passwd='OjdZYxCj'>
    </graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.0.35.42' passwd='OjdZYxCj'>

(The grep for unmasked passwords only returns the old 04:59:59 entry - the new 09:02:39 entry is properly masked and does not appear)

[root@ref-trl-10611-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -i "Replaced the VNC IP address" /var/log/cloudstack/agent/agent.log
2026-01-19 09:02:39,662 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:) Replaced the VNC IP address <graphics type='vnc' port='-1' autoport='yes' listen='10.0.35.40' passwd='*****'>

Test Execution Summary

Test Case	Description	Result
TC1	VM Start - VNC Password Masking	PASSED
TC2	Live Migration - VNC Password Masking	PASSED
TC3	Console Access Regression	PASSED
TC4	VM Stop/Start - VNC Password Masking	PASSED
TC5	VM Reboot - VNC Password Masking	PASSED
TC6	System VM - VNC Password Masking	PASSED
TC7	Reverse Migration - VNC Password Masking	PASSED
TC8	Storage Migration - VNC Password Masking	PASSED

Detailed Test Results #12404 (review)

Mask vncPasswd being logged in agent.log

b5588fa

boring-cyborg Bot added the component:kvm label Jan 12, 2026

sudo87 requested review from DaanHoogland and weizhouapache January 12, 2026 08:59

weizhouapache approved these changes Jan 12, 2026

View reviewed changes

abh1sar added this to the 4.20.3 milestone Jan 12, 2026

abh1sar approved these changes Jan 12, 2026

View reviewed changes

DaanHoogland reviewed Jan 12, 2026

View reviewed changes

DaanHoogland approved these changes Jan 12, 2026

View reviewed changes

DaanHoogland reviewed Jan 12, 2026

View reviewed changes

DaanHoogland added the needs-testing label Jan 12, 2026

address review comments

8831de3

DaanHoogland linked an issue Jan 14, 2026 that may be closed by this pull request

Agent logs libvirt xml while live migrating vms #10847

Closed

RosiKyu requested changes Jan 18, 2026

View reviewed changes

RosiKyu added status:needs-work and removed needs-testing labels Jan 18, 2026

fix unmask vnc password log

680ca79

RosiKyu added the status:testing label Jan 19, 2026

RosiKyu approved these changes Jan 19, 2026

View reviewed changes

RosiKyu added status:ready-for-merge and removed status:testing status:needs-work labels Jan 19, 2026

DaanHoogland merged commit 42f1e19 into apache:4.20 Jan 19, 2026
26 checks passed

DaanHoogland deleted the maskVncPwd branch January 19, 2026 13:20

DaanHoogland mentioned this pull request Jan 19, 2026

Agent logs libvirt xml while live migrating vms #10847

Closed

sandeeplocharla pushed a commit to NetApp/cloudstack that referenced this pull request Feb 6, 2026

Mask vncPasswd being logged in agent.log (apache#12404 )

aa7624a

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Mar 3, 2026

Mask vncPasswd being logged in agent.log (apache#12404 )

53e1dd0

+                  public static String maskSensitiveInfoInXML(String xmlDesc) {
+                      if (xmlDesc == null) return null;
+                      // Mask VNC password in XML for logging
+                      return xmlDesc.replaceAll("(graphics\\s+[^>]*type=['\"]vnc['\"][^>]*passwd=['\"])([^'\"]*)(['\"])",

Mask vncPasswd being logged in agent.log by sudo87 · Pull Request #12404 · apache/cloudstack · GitHub

Uh oh!

Mask vncPasswd being logged in agent.log - #12404

Mask vncPasswd being logged in agent.log#12404DaanHoogland merged 3 commits intoapache:4.20apache/cloudstack:4.20from shapeblue:maskVncPwdshapeblue/cloudstack:maskVncPwdCopy head branch name to clipboard

Conversation

sudo87 commented Jan 12, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented Jan 12, 2026

Uh oh!

blueorangutan commented Jan 12, 2026

Uh oh!

DaanHoogland Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

sudo87 Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jan 12, 2026

Uh oh!

abh1sar commented Jan 12, 2026

Uh oh!

blueorangutan commented Jan 12, 2026

Uh oh!

sudo87 commented Jan 12, 2026

Uh oh!

DaanHoogland commented Jan 12, 2026

Uh oh!

blueorangutan commented Jan 13, 2026

Uh oh!

RosiKyu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Test Execution Summary

Detailed Test Report

Test Case 1: VM Start - VNC Password Masking

Test Case 2: Live Migration - VNC Password Masking.

Test Case 3: Console Access Regression Test

Test Case 4: VM Stop and Start - VNC Password Masking

Test Case 5: VM Reboot - VNC Password Masking

Test Case 6: System VM - VNC Password Masking

Test Case 7: Reverse Migration (kvm2 → kvm1)

Test Case 8: Storage Migration

Uh oh!

sudo87 commented Jan 19, 2026

Uh oh!

sudo87 commented Jan 19, 2026

Uh oh!

blueorangutan commented Jan 19, 2026

Uh oh!

blueorangutan commented Jan 19, 2026

Uh oh!

RosiKyu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Test Execution Summary

Mask vncPasswd being logged in agent.log#12404
DaanHoogland merged 3 commits into
apache:4.20from
shapeblue:maskVncPwd

codecov Bot commented Jan 12, 2026 •
edited
Loading

sudo87 Jan 13, 2026 •
edited
Loading

RosiKyu left a comment •
edited
Loading

RosiKyu left a comment •
edited
Loading