Ubuntu 20.04 support (as hypervisor) by weizhouapache · Pull Request #4069 · apache/cloudstack · GitHub

Description

This aims to support Ubuntu 20.04 as a hypervisor in cloudstack 4.15

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

In libvirt repository, there are two related commits 2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present 2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation In libvirt.spec.in /bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :

For more information: https://libvirt.org/kbase/backing_chains.html

Hi @weizhouapache - my colleagues @shwstppr @davidjumani are working to support python2 to python3 in #4068, so all scripts that run on management server and kvm host towards 4.15/milestone. So this change is not needed, and may break things (we probably don't want misleading symlinks that could mean python2 or python3). cc @wido @GabrielBrascher

We should really kill Python 2 and not try to have these things in there. I agree.

@rhtyd @wido
I do not know when cloudstack 4.15 will be released (with python3).
this can be regarded as a patch for cloudstack 4.14.
it works with same mysql version, same scripts (some written in python2, some written in python3), etc
if there is 4.14.1.0, hope this can be included.

Could this break on older CentOS/Ubuntu KVM hosts where the backfile is not supported?

@weizhouapache I checked libvirtd release notes, libvirt doesn't fail VM for version 5.x or below; only libvirt starting version 6.x fails VM deployment if backing file format is not provided. Should we add a check, as the -F option is not available for qemu-img on older distros, say CentOS7, or Ubuntu 16.04 (I didn't check 18.04). Unless we want to discuss and deprecate Ubuntu 16.04 support? cc @wido @GabrielBrascher @shwstppr @davidjumani

@rhtyd
I think it is supported by ubuntu 16.04
I have tested the migration from ubuntu 16.04 to ubuntu 20.04.
it works fine.

not sure if centos7 supports it.

@weizhouapache I'm cherry-picking couple of your commits in centos8 support PR branch for testing. I hope it is fine with you.

@shwstppr while you do the tests etc, can you check if qemu-img on CentOS7 supports the -F argument for specifying the backfile file format?

Related PR for same effort, different distro - likely some overlaping bits - #4068

We should really kill Python 2 and not try to have these things in there. I agree.

Same as on the mgmt server

@wido this is a patch for 4.14.0.0
if we change all python2 scripts to python3 in 4.15, we definitely do not need it.

@weizhouapache As the PR is against master it's for 4.15, therefor I assumed that.

Py3 support is finished for the hypervisors already by @GabrielBrascher , that all works.

No more need for Py2 on hypervisors.

@wido cool. do you have more changes than security_group.py ? could you create a PR ?
there are still some scripts in 4.14 which are written with python2 and used on kvm hosts, for example all scripts in agent/bindir/

@weizhouapache PR #3730 updates VR's scripts from python2 to python3. So far it has been tested on Advanced network with Security Group. We should have all python2 code migrated to python3 in 4.15.0.0; as soon as 4.14 is released and the master branch is "unfreeze", we will get back on porting python2 to python3.

Additionally, PR #3730 will be coordinated with other PRs (e.g. #3797). Any help is welcome 🙂

We should depend on Python 3 only here

When using addHost API, it fails to add host with:

2020-05-17 02:11:42,913 INFO  [c.c.r.ResourceManagerImpl] (qtp1916575798-22:ctx-998dc1cb ctx-3489fe88) (logid:59229745) Trying to add a new host at http://192.168.1.11 in data center 1
2020-05-17 02:11:42,951 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1916575798-22:ctx-998dc1cb ctx-3489fe88) (logid:59229745)  can't setup agent, due to java.io.IOException: There was a problem while connecting to 192.168.1.11:22 - There was a problem while connecting to 192.168.1.11:22

@weizhouapache have you seen such an error (I tried to use both the zone deployment wizard and addHost API where this fails)? Could this be due to default ssh algorithms issue or our ssh utility class needing any change? Thanks.

@weizhouapache (cc @shwstppr @davidjumani - perhaps check on CentOS8 the version of openssh-server/sshd) I found that openssh-server v7+ per their release notes has deprecated some options that our SSH utility/class uses (needs investigation) which is why CloudStack fails to ssh into KVM host for newer distros, to fix it I had to add the following to the sshd config:

PubkeyAcceptedKeyTypes=+ssh-dss
HostKeyAlgorithms=+ssh-dss 
KexAlgorithms=+diffie-hellman-group1-sha1

For reference, my test env:

# lsb_release -d && ssh -V
Description:	Ubuntu 20.04 LTS
OpenSSH_8.2p1 Ubuntu-4, OpenSSL 1.1.1f  31 Mar 2020

@weizhouapache another issue, on deployment of systemvms with MySQL 8.0 cc @shwstppr @davidjumani deployment fails with:

Caused by: java.sql.SQLIntegrityConstraintViolationException: Column 'update_time' cannot be null

I applied a quick fix to modify the default time to be null:

ALTER TABLE nics MODIFY COLUMN update_time timestamp NULL;

@weizhouapache cc @shwstppr @davidjumani @wido @GabrielBrascher @PaulAngus @andrijapanicsb @DaanHoogland @kiwiflyer and others
I found Ubuntu 16.04 has an older qemu-img that does not support that -F option with qemu-img create command:

qemu-img version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.43), Copyright (c) 2004-2008 Fabrice Bellard
usage: qemu-img command [command options]
QEMU disk image utility

Command syntax:
  check [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] filename
  create [-q] [-f fmt] [-o options] filename [size]

I checked CentOS7 and that was okay (qemu-img create supports -F ).

Let's discuss:

Either to add code to work around this issue, or
Deprecate Ubuntu 16.04 as an unsupported/older KVM host platform in 4.15 (I agree on this, given the distribution is old, however, it will EOL in April 2021)
In release/upgrade notes, ask admin to install a more recent qemu-img (and any other qemy or libvirt related dependency from bionic/18.04 repos).

@weizhouapache I am +1 for dropping 16.04 support in 4.15. As this will be an LTS which will be maintained beyond April 2021 and Ubuntu 16.04 is then no longer supported.

18.04 and 20.04 should be the supported Ubuntu distro's then.

@rhtyd
I checked ubuntu 16.04 just now. you are right. "qemu-img -F" is not supported as mentioned in help. However the backing file format is set when I create qcow2 with "-F", hence I think help is not updated. see below.

root@node42:~# qemu-img create -b /mnt/ee9ffb52-bd20-3328-8a7f-090723db12c4/bd84bcd9-d040-4e9f-a777-617c712b10dc -f qcow2 -F qcow2 bb
Formatting 'bb', fmt=qcow2 size=2097152000 backing_file=/mnt/ee9ffb52-bd20-3328-8a7f-090723db12c4/bd84bcd9-d040-4e9f-a777-617c712b10dc backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

root@node42:~# qemu-img info bb
image: bb
file format: qcow2
virtual size: 2.0G (2097152000 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/ee9ffb52-bd20-3328-8a7f-090723db12c4/bd84bcd9-d040-4e9f-a777-617c712b10dc
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

anyway, I agee with @wido we can probably drop Ubuntu 16.04 support if we release 4.15 next year

When using addHost API, it fails to add host with:
2020-05-17 02:11:42,913 INFO  [c.c.r.ResourceManagerImpl] (qtp1916575798-22:ctx-998dc1cb ctx-3489fe88) (logid:59229745) Trying to add a new host at http://192.168.1.11 in data center 1
2020-05-17 02:11:42,951 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1916575798-22:ctx-998dc1cb ctx-3489fe88) (logid:59229745)  can't setup agent, due to java.io.IOException: There was a problem while connecting to 192.168.1.11:22 - There was a problem while connecting to 192.168.1.11:22
@weizhouapache have you seen such an error (I tried to use both the zone deployment wizard and addHost API where this fails)? Could this be due to default ssh algorithms issue or our ssh utility class needing any change? Thanks.

@rhtyd is the issue fixed with the new ssh config ?
I was able to add Ubuntu 20.04 host in my testing env.

Two issues need to mention:
(1) there are some issues with ./python/lib/cloudutils/serviceConfig.py, please configure cloudbr0 or cloudbrX before adding the host.
(2) I had problem when I created and configured a Ubuntu 20.04 vm on CentOS 7 host. so I created a Ubuntu 20.04 vm on a Ubuntu 16.04 host which works fine.

Hi @weizhouapache - yes the addHost ssh issue is fixed with the changes to sshd_config

Thanks I've verified the -F option with qemu-img on Ubuntu 16.04 - you're right. That means we don't need to explicitly deprecate 16.04 wrt these changes. But I'm okay to drop 16.04 support in future releases (4.16 maybe)
Yes, in my testing I had added cloudbr0 before adding the host

On further testing my colleagues (@shwstppr) found an issue with newer qemu when trying to take a volume snapshot:

com.cloud.utils.exception.CloudRuntimeException: qemu-img: Could not open '/mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/20dc7b29-934a-409a-babf-f4527f574377': Failed to get shared "write" lockIs another process using the image [/mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/20dc7b29-934a-409a-babf-f4527f574377]?there is no 62f4eb05-168e-4161-a070-f8804df068d0 on disk /mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/20dc7b29-934a-409a-babf-f4527f574377
	at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301)
	at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1213)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1164)

Have to seen this issue?

@rhtyd The snapshot issue also exists under 18.04. Qemu now locks a QCOW2 file and thus can't be snapshotted while the VM is running in order to protect the QCOW2 file.

@shwstppr ping - can you check and confirm Wido's comment?

@rhtyd The snapshot issue also exists under 18.04. Qemu now locks a QCOW2 file and thus can't be snapshotted while the VM is running in order to protect the QCOW2 file.

@wido @rhtyd I was able to snapshot a stopped VM. Probably some caching issue in my UI else it should have stopped the operation. Checking that. This should be a non-issue otherwise.

@blueorangutan package

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔debian. JID-1260

Trillian test result (tid-1597)
Environment: kvm-ubuntu18 (x2), Advanced Networking with Mgmt server 7
Total time taken: 59276 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4069-t1597-kvm-ubuntu18.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_extra_config_data.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 78 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_deploy_vm_with_extraconfig_kvm	Error	35.90	test_deploy_vm_extra_config_data.py
test_03_update_vm_with_extraconfig_kvm	Error	387.26	test_deploy_vm_extra_config_data.py
test_01_deploy_kubernetes_cluster	Error	3615.35	test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster	Error	0.04	test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster	Failure	2301.75	test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster	Error	0.03	test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster	Error	3610.04	test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster	Error	0.03	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	Error	34.72	test_kubernetes_clusters.py
test_01_snapshot_root_disk	Error	8.25	test_snapshots.py
test_01_snapshot_usage	Error	6.21	test_usage.py
test_06_download_detached_volume	Error	9.31	test_volumes.py

@weizhouapache can you check and close, we used your commits as reference and included here for consolidated fixes and testing - #4068

Closing due to cherry-picked in #4068

LGTM, here is what I've tested:

KVM operations:

VM lifecycle - deploy, console view, stop/start on different host, reinstall, reboot, save/restore VM snapshot, migrate, attach/detach ISO, deploy VM from ISO/template, reset SSH pair, scale VM, delete
Add/remove host

LGTM, here is what I've tested:

KVM operations:

VM lifecycle - deploy, console view, stop/start on different host, reinstall, reboot, save/restore VM snapshot, migrate, attach/detach ISO, deploy VM from ISO/template, reset SSH pair, scale VM, delete

Add/remove host

@vladimirpetrov thanks for testing.
However, this pr has been closed, as 3 of 4 commits have been merged into mainstream already.
What have you tested ? mainsteam, or 4.14 + this pr ?

Hi @weizhouapache, this was tested on the latest mainstream, the idea was to double-check the feature.

ustcweizhou added 4 commits May 11, 2020 08:37

Ubuntu 20.04: install python2 instead of python

476d701

Ubuntu 20.04: Specify image format of backing file

bcc384a

For more information: https://libvirt.org/kbase/backing_chains.html

Ubuntu 20.04: setBackingFormat in vm migration

36518b9

weizhouapache mentioned this pull request May 11, 2020

kvm: newer qemu-img and libvirt 6.0 expects backing file format #4066

Closed

1 of 5 tasks

yadvr reviewed May 11, 2020

View reviewed changes

yadvr mentioned this pull request May 11, 2020

Adding Centos8, Ubuntu 20.04, XCPNG8.1 Support #4068

Merged

1 of 5 tasks

wido requested changes May 11, 2020

View reviewed changes

weizhouapache mentioned this pull request May 12, 2020

[4.15] Support of next versions #4081

Closed

DaanHoogland changed the title [WIP] Ubuntu 20.04 support (as hypervisor) Ubuntu 20.04 support (as hypervisor) May 18, 2020

weizhouapache mentioned this pull request Jun 29, 2020

python3-mysql.connector package needed for running on Ubuntu20.04 #4185

Closed

yadvr closed this Jul 4, 2020

                       s.add("-f");
                       if (backingFile != null) {
+                          s.add(backingFile.getFormat().toString());
+                          s.add("-F");

+                  # Create symbolic link for python
+                  if [ ! -f "/usr/bin/python" ]; then
+                      ln -s /usr/bin/python2 /usr/bin/python
+                  fi

               Package: cloudstack-management
               Architecture: all
-              Depends: ${python:Depends}, openjdk-11-jre-headless | java11-runtime-headless | java11-runtime | openjdk-11-jre-headless | zulu-11, cloudstack-common (= ${source:Version}), sudo, python-mysql.connector, augeas-tools, mysql-client | mariadb-client, adduser, bzip2, ipmitool, file, gawk, iproute2, qemu-utils, python-dnspython, lsb-release, init-system-helpers (>= 1.14~)
+              Depends: ${python:Depends}, openjdk-11-jre-headless | java11-runtime-headless | java11-runtime | openjdk-11-jre-headless | zulu-11, cloudstack-common (= ${source:Version}), sudo, python-mysql.connector, augeas-tools, mysql-client | mariadb-client, adduser, bzip2, ipmitool, file, gawk, iproute2, qemu-utils, python-dnspython, lsb-release, init-system-helpers (>= 1.14~), python | python2

+                          // Verify Format of backing file
+                          for (DiskDef disk : disks) {
+                              if (disk.getDeviceType() == DiskDef.DeviceType.DISK

Ubuntu 20.04 support (as hypervisor) by weizhouapache · Pull Request #4069 · apache/cloudstack · GitHub

Repository navigation

Ubuntu 20.04 support (as hypervisor) - #4069

Ubuntu 20.04 support (as hypervisor)#4069weizhouapache wants to merge 4 commits intoapache:masterapache/cloudstack:masterfrom weizhouapache:4.15-ubuntu2004weizhouapache/cloudstack:4.15-ubuntu2004Copy head branch name to clipboard

Conversation

weizhouapache commented May 11, 2020

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weizhouapache May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr commented May 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr commented May 17, 2020

Uh oh!

yadvr commented May 17, 2020

Uh oh!

yadvr commented May 17, 2020

Uh oh!

yadvr commented May 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wido commented May 17, 2020

Uh oh!

weizhouapache commented May 17, 2020

Uh oh!

weizhouapache commented May 17, 2020

Uh oh!

yadvr commented May 18, 2020

Uh oh!

wido commented May 18, 2020

Uh oh!

yadvr commented May 18, 2020

Uh oh!

shwstppr commented May 18, 2020

Uh oh!

shwstppr commented Jun 1, 2020

Uh oh!

blueorangutan commented Jun 1, 2020

Uh oh!

blueorangutan commented Jun 1, 2020

Uh oh!

blueorangutan commented Jun 2, 2020

Uh oh!

yadvr commented Jun 17, 2020

Uh oh!

yadvr commented Jul 4, 2020

Uh oh!

Ubuntu 20.04 support (as hypervisor)#4069
weizhouapache wants to merge 4 commits into
apache:masterfrom
weizhouapache:4.15-ubuntu2004

weizhouapache May 11, 2020 •
edited
Loading

yadvr commented May 17, 2020 •
edited
Loading