Fixes for KVM unmanaged instances import on advanced network and VNC password by nvazquez · Pull Request #8492 · apache/cloudstack · GitHub

Description

This PR fixes a regression caused by #8465 on advanced zones, import fails with:

2024-01-10 12:13:33,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-3:ctx-991bbe9f job-128 ctx-f49517d4) (logid:d7b8e716) Allocating nic for vm 142272e8-9e2e-407b-9d7e-e9a03b81653c in network Network {"id": 204, "name": "Isolated", "uuid": "9679fac5-e3ac-4694-a57b-beb635340f39", "networkofferingid": 10} during import
2024-01-10 12:13:33,239 ERROR [o.a.c.v.UnmanagedVMsManagerImpl] (API-Job-Executor-3:ctx-991bbe9f job-128 ctx-f49517d4) (logid:d7b8e716) Failed to import NICs while importing vm: i-2-31-VM
com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to acquire Guest IP  address for network Network {"id": 204, "name": "Isolated", "uuid": "9679fac5-e3ac-4694-a57b-beb635340f39", "networkofferingid": 10}Scope=interface com.cloud.dc.DataCenter; id=1
	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.importNic(NetworkOrchestrator.java:4582)
	at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importNic(UnmanagedVMsManagerImpl.java:859)
	at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importVirtualMachineInternal(UnmanagedVMsManagerImpl.java:1198)
	at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importUnmanagedInstanceFromHypervisor(UnmanagedVMsManagerImpl.java:1511)
	at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.baseImportInstance(UnmanagedVMsManagerImpl.java:1342)
	at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importUnmanagedInstance(UnmanagedVMsManagerImpl.java:1282)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Also, addresses the VNC password field set instead of a fixed string

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Codecov Report

Attention: 18 lines in your changes are missing coverage. Please review.

Comparison is base (59e78cb) 30.79% compared to head (f28f9b5) 30.81%.

Files	Patch %	Lines
...tack/engine/orchestration/NetworkOrchestrator.java	53.84%	7 Missing and 5 partials ⚠️
...er/LibvirtGetUnmanagedInstancesCommandWrapper.java	0.00%	5 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8492      +/-   ##
============================================
+ Coverage     30.79%   30.81%   +0.02%     
- Complexity    33990    34013      +23     
============================================
  Files          5341     5341              
  Lines        374961   374971      +10     
  Branches      54543    54546       +3     
============================================
+ Hits         115452   115558     +106     
+ Misses       244241   244148      -93     
+ Partials      15268    15265       -3

Flag	Coverage Δ
simulator-marvin-tests	24.71% <0.00%> (+0.03%)	⬆️
uitests	4.39% <ø> (ø)
unit-tests	16.48% <43.75%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@blueorangutan package

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 8275

@blueorangutan package

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8278

@blueorangutan test

@nvazquez a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Lot many changes to core NIC import logic. This would need testing for both KVM and VMware for different scenarios - advanced/basic zone, with/without user provided IP, different network types.

@nvazquez cc @DaanHoogland would it make sense to revert #8465 and document basic zone support as a limitation at this point? Common code for both KVM and VMware seems to be causing new regressions each day 😄

@shwstppr I think we can have this fix without reverting the basic zone support, I have already verified basic and advanced KVM import in automatic and manual cases, will need to double check Vmware imports

Tested the following cases for VMware VM import in an advanced zone:

Import VM with single L2 network NIC - OK
Import VM with single Isolated network NIC with auto IP assignment - OK
Import VM with single Isolated network NIC with user-provided IP assignment - OK
Import VM with single Isolated network NIC with user-provided already taken IP assignment - OK (Fails)
Import VM with single Isolated network NIC with user-provided outside range IP assignment - OK (Fails)
Import VM with single Shared network NIC with auto IP assignment - OK
Import VM with single Shared network NIC with user-provided IP assignment - OK
Import VM with single Shared network NIC with user-provided already taken IP assignment - OK (Fails)
Import VM with single Shared network NIC with user-provided outside range IP assignment - OK ? (Not sure if this should work) (Fails)
Import VM with a L2 network NIC and a shared network NIC with auto IP assignment - OK
Import VM with an isolated network NIC and a shared network NIC with auto IP assignment - OK

I guess we also allow VM deployment similar to case#9 (IP outside range)

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@nvazquez @DaanHoogland I retested case#9 from #8492 (comment) and it is failing as expected, therefore it all looks good on VMware side

(localcloud) 🐱 > list vlanipranges id=3f69aecb-1aae-4f7e-b559-1e5c9eeac568 
{
  "count": 1,
  "vlaniprange": [
    {
      "account": "system",
      "cidr": "10.8.0.0/24",
      "domain": "ROOT",
      "domainid": "a2138f1e-af98-11ee-847d-1e0058000a5d",
      "endip": "10.8.0.200",
      "forsystemvms": false,
      "forvirtualnetwork": false,
      "gateway": "10.8.0.1",
      "id": "3f69aecb-1aae-4f7e-b559-1e5c9eeac568",
      "netmask": "255.255.255.0",
      "networkid": "587d654e-fee0-49ab-a736-925e24d684c9",
      "physicalnetworkid": "c654f1ef-6047-4cd8-882b-f407dc604c4c",
      "startip": "10.8.0.101",
      "vlan": "2121",
      "zoneid": "b4a732bb-e837-4682-bbb9-b845b04a17dd"
    }
  ]
}
(localcloud) 🐱 > list unmanagedinstances clusterid=e3087b2f-8be7-4832-b8ef-61d7e25b27cf name=i-2-30-VM
{
  "count": 1,
  "unmanagedinstance": [
    {
      "clusterid": "e3087b2f-8be7-4832-b8ef-61d7e25b27cf",
      "clustername": "p1-c1",
      "cpucorepersocket": 1,
      "cpunumber": 1,
      "cpuspeed": 0,
      "disk": [
        {
          "capacity": 52428800,
          "controller": "lsilogic",
          "controllerunit": 0,
          "datastorehost": "10.1.32.4",
          "datastorename": "5b2964142196369d96d6b95b6ad5e170",
          "datastorepath": "/acs/primary/ref-trl-5272-v-Mr8-abhishek-kumar/ref-trl-5272-v-Mr8-abhishek-kumar-esxi-pri1",
          "datastoretype": "NFS",
          "id": "15-2000",
          "imagepath": "[5b2964142196369d96d6b95b6ad5e170] i-2-30-VM/ROOT-30.vmdk",
          "label": "Hard disk 1",
          "position": 0
        }
      ],
      "hostid": "d178e42e-1c2e-452b-a572-eac1fd2fb11c",
      "hostname": "10.1.32.206",
      "memory": 512,
      "name": "i-2-30-VM",
      "nic": [
        {
          "adaptertype": "E1000",
          "id": "Network adapter 1",
          "macaddress": "02:01:00:cc:00:0d",
          "networkname": "cloud.guest.1747.200.1-vSwitch1",
          "vlanid": 1747
        }
      ],
      "osdisplayname": "Other Linux (64-bit)",
      "osid": "otherLinux64Guest",
      "powerstate": "PowerOn"
    }
  ]
}
(localcloud) 🐱 > import unmanagedinstance name=i-2-30-VM clusterid=e3087b2f-8be7-4832-b8ef-61d7e25b27cf displayname=erwewr zoneid=b4a732bb-e837-4682-bbb9-b845b04a17dd importsource=unmanaged hypervisor=vmware serviceofferingid=cec984be-1c57-468b-b1b1-94ea3cef705e nicnetworklist[0].nic=Network adapter 1 nicnetworklist[0].network=51978449-5557-41f7-8642-35739fcc90ae nicipaddresslist[0].nic="Network adapter 1" nicipaddresslist[0].ip4Address=10.8.0.52
{
  "account": "admin",
  "accountid": "c77ca2f3-af98-11ee-847d-1e0058000a5d",
  "cmd": "org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd",
  "completed": "2024-01-11T12:13:36+0000",
  "created": "2024-01-11T12:13:36+0000",
  "domainid": "a2138f1e-af98-11ee-847d-1e0058000a5d",
  "domainpath": "ROOT",
  "jobid": "2c50b7a2-fe40-4705-8fbd-be08eba1bf54",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "IP address 10.8.0.52 for NIC(ID: Network adapter 1) is not available in network(ID: 51978449-5557-41f7-8642-35739fcc90ae)"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "c77dbaa7-af98-11ee-847d-1e0058000a5d"
}
🙈 Error: async API failed for job 2c50b7a2-fe40-4705-8fbd-be08eba1bf54

Tested with advanced network with KVM and works with NFS/StorPool/Ceph.
code LGTM

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8289

@blueorangutan package

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8291

[SF] Trillian test result (tid-8786)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 56656 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8492-t8786-kvm-centos7.zip
Smoke tests completed. 120 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_07_deploy_kubernetes_ha_cluster	Failure	3621.31	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	Failure	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	Failure	0.04	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	Failure	55.98	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	Error	116.55	test_kubernetes_clusters.py

@blueorangutan test matrix

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

clgtm and tested kvm in-/export as well as migration from vmware to kvm. This works but for a snag when root volumes are small. This last bit will be addressed in either #8488 or #8500.

Merging based on manual testing and trillain results from backend,

[SF] Trillian test result (tid-8811)

Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8

Total time taken: 48240 seconds

Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8492-t8811-vmware-67u3.zip

Smoke tests completed. 120 look OK, 1 have errors, 0 did not run

Only failed and skipped tests results shown below:





Test | Result | Time (s) | Test File

--- | --- | --- | ---

test_02_balanced_drs_algorithm | `Failure` | 138.23 | test_cluster_drs.py



[SF] Trillian test result (tid-8810)

Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7

Total time taken: 43357 seconds

Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8492-t8810-xenserver-71.zip

Smoke tests completed. 120 look OK, 1 have errors, 0 did not run

Only failed and skipped tests results shown below:





Test | Result | Time (s) | Test File

--- | --- | --- | ---

test_01_cancel_host_maintenace_with_no_migration_jobs | `Error` | 308.71 | test_host_maintenance.py

…password (apache#8492) This PR fixes a regression caused by apache#8465 on advanced zones, import fails with: 2024-01-10 12:13:33,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-3:ctx-991bbe9f job-128 ctx-f49517d4) (logid:d7b8e716) Allocating nic for vm 142272e8-9e2e-407b-9d7e-e9a03b81653c in network Network {"id": 204, "name": "Isolated", "uuid": "9679fac5-e3ac-4694-a57b-beb635340f39", "networkofferingid": 10} during import 2024-01-10 12:13:33,239 ERROR [o.a.c.v.UnmanagedVMsManagerImpl] (API-Job-Executor-3:ctx-991bbe9f job-128 ctx-f49517d4) (logid:d7b8e716) Failed to import NICs while importing vm: i-2-31-VM com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to acquire Guest IP address for network Network {"id": 204, "name": "Isolated", "uuid": "9679fac5-e3ac-4694-a57b-beb635340f39", "networkofferingid": 10}Scope=interface com.cloud.dc.DataCenter; id=1 at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.importNic(NetworkOrchestrator.java:4582) at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importNic(UnmanagedVMsManagerImpl.java:859) at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importVirtualMachineInternal(UnmanagedVMsManagerImpl.java:1198) at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importUnmanagedInstanceFromHypervisor(UnmanagedVMsManagerImpl.java:1511) at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.baseImportInstance(UnmanagedVMsManagerImpl.java:1342) at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.importUnmanagedInstance(UnmanagedVMsManagerImpl.java:1282) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Also, addresses the VNC password field set instead of a fixed string

nvazquez added 2 commits January 10, 2024 09:30

Refactor VNC password set on KVM unmanaged instance

ca67e32

Fix IP assignment for advanced and basic zones

1548c5e

nvazquez requested review from DaanHoogland, Pearl1594, harikrishna-patnala, shwstppr, slavkap and weizhouapache January 10, 2024 13:03

boring-cyborg Bot added component:kvm component:orchestration labels Jan 10, 2024

Fix random characters for VNC password to use only letters

2f688c2

Fix for manual IP assignment on basic zones

51d96e9

Refactor method

6c264c9

shwstppr added this to the 4.19.0.0 milestone Jan 11, 2024

Merge branch 'main' into kvm-unmanage-fixes-advanced-andpass

1a1ef27

slavkap approved these changes Jan 11, 2024

View reviewed changes

Add validation to prevent NPE on setting host

f28f9b5

shwstppr mentioned this pull request Jan 11, 2024

vm-import: kvm import and fix volume size when lesser than 1GiB #8500

Merged

1 of 13 tasks

DaanHoogland approved these changes Jan 12, 2024

View reviewed changes

shwstppr approved these changes Jan 12, 2024

View reviewed changes

shwstppr merged commit a3a4833 into apache:main Jan 12, 2024

shwstppr deleted the kvm-unmanage-fixes-advanced-andpass branch January 12, 2024 08:44

Fixes for KVM unmanaged instances import on advanced network and VNC password by nvazquez · Pull Request #8492 · apache/cloudstack · GitHub

Repository navigation

Fixes for KVM unmanaged instances import on advanced network and VNC password - #8492

Fixes for KVM unmanaged instances import on advanced network and VNC password#8492shwstppr merged 7 commits intoapache:mainapache/cloudstack:mainfrom shapeblue:kvm-unmanage-fixes-advanced-andpassshapeblue/cloudstack:kvm-unmanage-fixes-advanced-andpassCopy head branch name to clipboard

Conversation

nvazquez commented Jan 10, 2024

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

DaanHoogland commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

codecov Bot commented Jan 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nvazquez commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

nvazquez commented Jan 10, 2024

Uh oh!

nvazquez commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

nvazquez commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

shwstppr commented Jan 11, 2024

Uh oh!

nvazquez commented Jan 11, 2024

Uh oh!

shwstppr commented Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

shwstppr commented Jan 11, 2024

Uh oh!

slavkap left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

nvazquez commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

shwstppr commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Jan 12, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Fixes for KVM unmanaged instances import on advanced network and VNC password#8492
shwstppr merged 7 commits into
apache:mainfrom
shapeblue:kvm-unmanage-fixes-advanced-andpass

codecov Bot commented Jan 10, 2024 •
edited
Loading

shwstppr commented Jan 11, 2024 •
edited
Loading