Re: [ovirt-devel] oVirt System Test configuration

On Mon, Dec 18, 2017 at 2:05 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > On Mon, Dec 18, 2017 at 3:01 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > wrote: > >> >> >> 2017-12-18 13:57 GMT+01:00 Eyal Edri <eedri(a)redhat.com&gt;: >> >>> >>> >>> On Mon, Dec 18, 2017 at 2:53 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> 2017-12-18 12:42 GMT+01:00 Yaniv Kaul <ykaul(a)redhat.com&gt;: >>>> >>>>> >>>>> >>>>> On Mon, Dec 18, 2017 at 12:43 PM, Sandro Bonazzola < >>>>> sbonazzo(a)redhat.com&gt; wrote: >>>>> >>>>>> Hi, I'd like to discuss what's being tested by oVirt System Test. >>>>>> >>>>>> I'm investigating on a sanlock issue that affects hosted engine hc >>>>>> suite. >>>>>> I installed a CentOS minimal VM and set repositories as in >>>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic- >>>>>> suite-master/128/artifact/exported-artifacts/reposync-config.repo >>>>>> >>>>>> Upgrade from CentOS 1708 (7.4) minimal is: >>>>>> >>>>>> Aggiornamento: >>>>>> bind-libs-lite x86_64 >>>>>> 32:9.9.4-51.el7_4.1 >>>>>> centos-updates-el7 733 k >>>>>> bind-license noarch >>>>>> 32:9.9.4-51.el7_4.1 >>>>>> centos-updates-el7 84 k >>>>>> nss x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 849 k >>>>>> nss-softokn x86_64 >>>>>> 3.28.3-8.el7_4 >>>>>> centos-updates-el7 310 k >>>>>> nss-softokn-freebl x86_64 >>>>>> 3.28.3-8.el7_4 >>>>>> centos-updates-el7 214 k >>>>>> nss-sysinit x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 60 k >>>>>> nss-tools x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 501 k >>>>>> selinux-policy noarch >>>>>> 3.13.1-166.el7_4.7 >>>>>> centos-updates-el7 437 k >>>>>> selinux-policy-targeted noarch >>>>>> 3.13.1-166.el7_4.7 >>>>>> centos-updates-el7 6.5 M >>>>>> systemd x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 5.2 M >>>>>> systemd-libs x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 376 k >>>>>> systemd-sysv x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 70 k >>>>>> >>>>>> Enabling the CentOS repos: >>>>>> >>>>>> grub2 x86_64 >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 29 k >>>>>> in sostituzione di grub2.x86_64 1:2.02-0.64.el7.centos >>>>>> grub2-tools x86_64 >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 1.8 M >>>>>> in sostituzione di grub2-tools.x86_64 1:2.02-0.64.el7.centos >>>>>> grub2-tools-extra x86_64 >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 993 k >>>>>> in sostituzione di grub2-tools.x86_64 1:2.02-0.64.el7.centos >>>>>> grub2-tools-minimal x86_64 >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 170 k >>>>>> in sostituzione di grub2-tools.x86_64 1:2.02-0.64.el7.centos >>>>>> kernel x86_64 >>>>>> 3.10.0-693.11.1.el7 >>>>>> updates 43 M >>>>>> Aggiornamento: >>>>>> NetworkManager x86_64 >>>>>> 1:1.8.0-11.el7_4 >>>>>> updates 1.6 M >>>>>> NetworkManager-libnm x86_64 >>>>>> 1:1.8.0-11.el7_4 >>>>>> updates 1.2 M >>>>>> NetworkManager-team x86_64 >>>>>> 1:1.8.0-11.el7_4 >>>>>> updates 156 k >>>>>> NetworkManager-tui x86_64 >>>>>> 1:1.8.0-11.el7_4 >>>>>> updates 224 k >>>>>> NetworkManager-wifi x86_64 >>>>>> 1:1.8.0-11.el7_4 >>>>>> updates 184 k >>>>>> bash x86_64 >>>>>> 4.2.46-29.el7_4 >>>>>> updates 1.0 M >>>>>> bind-libs-lite x86_64 >>>>>> 32:9.9.4-51.el7_4.1 >>>>>> centos-updates-el7 733 k >>>>>> bind-license noarch >>>>>> 32:9.9.4-51.el7_4.1 >>>>>> centos-updates-el7 84 k >>>>>> binutils x86_64 >>>>>> 2.25.1-32.base.el7_4.1 >>>>>> updates 5.4 M >>>>>> cpio x86_64 >>>>>> 2.11-25.el7_4 >>>>>> updates 210 k >>>>>> cryptsetup-libs x86_64 >>>>>> 1.7.4-3.el7_4.1 >>>>>> updates 223 k >>>>>> curl x86_64 >>>>>> 7.29.0-42.el7_4.1 >>>>>> updates 267 k >>>>>> glibc x86_64 >>>>>> 2.17-196.el7_4.2 >>>>>> updates 3.6 M >>>>>> glibc-common x86_64 >>>>>> 2.17-196.el7_4.2 >>>>>> updates 11 M >>>>>> grub2-common noarch >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 726 k >>>>>> grub2-pc x86_64 >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 29 k >>>>>> grub2-pc-modules noarch >>>>>> 1:2.02-0.65.el7.centos.2 >>>>>> updates 845 k >>>>>> iptables x86_64 >>>>>> 1.4.21-18.2.el7_4 >>>>>> updates 428 k >>>>>> kernel-tools x86_64 >>>>>> 3.10.0-693.11.1.el7 >>>>>> updates 5.1 M >>>>>> kernel-tools-libs x86_64 >>>>>> 3.10.0-693.11.1.el7 >>>>>> updates 5.0 M >>>>>> kexec-tools x86_64 >>>>>> 2.0.14-17.2.el7 >>>>>> updates 333 k >>>>>> kmod x86_64 >>>>>> 20-15.el7_4.6 >>>>>> updates 120 k >>>>>> kmod-libs x86_64 >>>>>> 20-15.el7_4.6 >>>>>> updates 50 k >>>>>> libblkid x86_64 >>>>>> 2.23.2-43.el7_4.2 >>>>>> updates 176 k >>>>>> libcurl x86_64 >>>>>> 7.29.0-42.el7_4.1 >>>>>> updates 219 k >>>>>> libgcc x86_64 >>>>>> 4.8.5-16.el7_4.1 >>>>>> updates 98 k >>>>>> libgomp x86_64 >>>>>> 4.8.5-16.el7_4.1 >>>>>> updates 154 k >>>>>> libmount x86_64 >>>>>> 2.23.2-43.el7_4.2 >>>>>> updates 178 k >>>>>> libpciaccess x86_64 >>>>>> 0.13.4-3.1.el7_4 >>>>>> updates 26 k >>>>>> libstdc++ x86_64 >>>>>> 4.8.5-16.el7_4.1 >>>>>> updates 301 k >>>>>> libuuid x86_64 >>>>>> 2.23.2-43.el7_4.2 >>>>>> updates 79 k >>>>>> ncurses x86_64 >>>>>> 5.9-14.20130511.el7_4 >>>>>> updates 304 k >>>>>> ncurses-base noarch >>>>>> 5.9-14.20130511.el7_4 >>>>>> updates 68 k >>>>>> ncurses-libs x86_64 >>>>>> 5.9-14.20130511.el7_4 >>>>>> updates 316 k >>>>>> nss x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 849 k >>>>>> nss-softokn x86_64 >>>>>> 3.28.3-8.el7_4 >>>>>> centos-updates-el7 310 k >>>>>> nss-softokn-freebl x86_64 >>>>>> 3.28.3-8.el7_4 >>>>>> centos-updates-el7 214 k >>>>>> nss-sysinit x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 60 k >>>>>> nss-tools x86_64 >>>>>> 3.28.4-15.el7_4 >>>>>> centos-updates-el7 501 k >>>>>> openssh x86_64 >>>>>> 7.4p1-13.el7_4 >>>>>> updates 509 k >>>>>> openssh-clients x86_64 >>>>>> 7.4p1-13.el7_4 >>>>>> updates 654 k >>>>>> openssh-server x86_64 >>>>>> 7.4p1-13.el7_4 >>>>>> updates 458 k >>>>>> python-gobject-base x86_64 >>>>>> 3.22.0-1.el7_4.1 >>>>>> updates 294 k >>>>>> python-perf x86_64 >>>>>> 3.10.0-693.11.1.el7 >>>>>> updates 5.1 M >>>>>> selinux-policy noarch >>>>>> 3.13.1-166.el7_4.7 >>>>>> centos-updates-el7 437 k >>>>>> selinux-policy-targeted noarch >>>>>> 3.13.1-166.el7_4.7 >>>>>> centos-updates-el7 6.5 M >>>>>> sudo x86_64 >>>>>> 1.8.19p2-11.el7_4 >>>>>> updates 1.1 M >>>>>> systemd x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 5.2 M >>>>>> systemd-libs x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 376 k >>>>>> systemd-sysv x86_64 >>>>>> 219-42.el7_4.4 >>>>>> centos-updates-el7 70 k >>>>>> tzdata noarch >>>>>> 2017c-1.el7 >>>>>> updates 468 k >>>>>> util-linux x86_64 >>>>>> 2.23.2-43.el7_4.2 >>>>>> updates 2.0 M >>>>>> wpa_supplicant x86_64 >>>>>> 1:2.6-5.el7_4.1 >>>>>> updates 1.2 M >>>>>> >>>>>> meaning this environement is not receiving updates to core packages >>>>>> like the kernel. >>>>>> >>>>>> Restricting to libvirt, with the repos used in the job libvirt >>>>>> packages doesn't even exists, making yum install libvirt just fail. >>>>>> >>>>>> >>>>>> I think you already know I'm against filtering packages from the >>>>>> repos even if I understand it saves a huge amount of space and download >>>>>> time. >>>>>> I may be wrong, but I tend to not trust OST results since it's not >>>>>> testing real life environments. Any chance we can improve OST to match what >>>>>> users are going to have on their systems? >>>>>> >>>>> >>>>> Specifically for the hosted-engine use case, I had the same >>>>> suspicion, so I patched Lago to use '--network' in virt-sysprep and added >>>>> 'update' command to ensure all hosts are fully up-to-date when deployed. >>>>> It did not solve the issue. >>>>> >>>>> Generally: >>>>> 1. Updating the images take quite some time, per run. >>>>> 2. I agree every X months we should be updating the images. Every Q >>>>> for example. I've already discussed this with Gal. If CentOS would have >>>>> released a new baseline, that would have been more real life scenario, btw. >>>>> 3. That being said, if you need a dependency, you need to depend on >>>>> it, and we'll pull it. We do not ask (I think?) to 'yum update' your system >>>>> before deploying oVirt. >>>>> >>>> >>>> Actually yes, we require it. See https://ovirt.org/document >>>> ation/install-guide/chap-Installing_oVirt/ >>>> It explicitly says to yum update before proceeding with the deployment >>>> and reboot the machine if kernel has been updated. >>>> >>> >>> Isn't that just for engine side? do we have the same requirement for >>> hypervisors as well? >>> >> >> Not 100% sure but ovirt-engine in 4.2 should be performing yum update on >> host upgrade and this obviously don't pull in new packages if they are >> filtered out at repository level. >> > > Nothing is filtered out, unless you disable the external repos, which is > not the default. > However, we do not test upgrading a host (which we probably should add to > o-s-t!) > ​Yeah, we could try to add that

> Y. > > Adding Martin to keep me honest on this. >> > ​Ho​st upgrade manager in 4.2 updates all available packages and perform reboot by default afterwards. But this is happening only after host is successfully added to engine and also admin needs to execute this action (we only check for upgrades automatically). But we don't update all packages before or during host installation. >> >> >> >>> >>> >>>> >>>> >>>>> 4. There were, luckily, very few o-s-t only issues thus far. It may >>>>> happen, and we will run into those, but we have not so far encountered >>>>> many. Some we did not bother pursue, though are clear bugs (I remember an >>>>> annoying one with 'mount' command not getting back at all, etc.) >>>>> Y. >>>>> >>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> SANDRO BONAZZOLA >>>>>> >>>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>>>>> >>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>> <https://red.ht/sig> >>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> SANDRO BONAZZOLA >>>> >>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> >>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>> <https://redhat.com/trusted> >>> phone: +972-9-7692018 <+972%209-769-2018> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> >> -- >> >> SANDRO BONAZZOLA >> >> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> >> > -- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.