- Devel - oVirt List Archives

[VDSM] Test pass, build fail during cleanup
by Nir Soffer 26 Jan '21

26 Jan '21

We seem to have an issue in the CI, starting this week. All tests pass, but creating coverage report fail: + generate_combined_coverage_report + pushd tests ~/tests ~ + pwd /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests + ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt + python3 -m coverage combine .coverage-gluster .coverage-lib .coverage-network .coverage-nose .coverage-storage .coverage-virt No usable data files Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte Coverage.py warning: Couldn't read data from '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt': UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 106: invalid start byte This looks like a bug in coverage, or maybe we use the wrong version of this tool? + teardown + res=1 + '[' 1 -ne 0 ']' + echo '*** err: 1' *** err: 1 Additionally, we have this very strange error: + teardown_storage + python3 tests/storage/userstorage.py teardown python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No such file or directory This error is impossible, since tests/storage/userstorage.py is part of the source. Is it possible that some other code has deleted the source while a build was running? + echo 'WARNING: Ingoring error while tearing down user storage' WARNING: Ingoring error while tearing down user storage We were careful not to fail the build because this issue, but it failed because coverage report could not be created. There are multiple instance of this issue: - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/che… - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/che… - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/che… I think Eyal reported another instance, but I cannot find it now. 2 weeks ago we updated pytest and pytest-cov to latest version, but we did not see this issue in the CI when we tested the change: https://gerrit.ovirt.org/c/vdsm/+/112604 And CI was fine since then, so this may not be related, unless this is a regression in latest coverage or pytest-cov. Nir

2 2

OST fails, please review gerrit #113140
by Vojtech Juranek 25 Jan '21

25 Jan '21

Hi, OST fails AFAICT due to flake8 issue. Please review [1], it should fix the issue. Thanks Vojta [1] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113140/

1 0

Gerrit very slow again - reload takes many seconds
by Nir Soffer 24 Jan '21

24 Jan '21

This is not a new issue, we see this every few months. In the past I think this was solved by restarting gerrit. I compared our gerrit to other installations: https://go-review.googlesource.com/ https://review.opendev.org/ Both reloads in about 1 second. Our gerrit takes 10-15 seconds. Go gerrit is using: 3.3.1-812-g388176bbb4 Openstack is using: 3.2.6-28-g4a9ffec296-dirty We use: 2.16.23 Looking in https://www.gerritcodereview.com/ current version is 3.3.1. Looks like we are using an old version. Nir

1 0

OST on Stream
by Michal Skrivanek 22 Jan '21

22 Jan '21

There seems to be multiple issues really 1) First it fails on “unknown state” of ovirt-engine-notifier service. Seems the OST code is not handling service states well and just explodes 2) Either way, the problem is it's stopped. It should be running 3) if you start it manually it gets to the engine-config test and restarts engine. It explodes here again, apparently same reason as #1 4) host installation fails - didi’s https://gerrit.ovirt.org/#/c/ovirt-engine/+/113101/ is fixing it 5) verify_engine_notifier fails when we try to stop it. it’s not running due to #2 and it explodes anyway due to #1 rest seems to work fine except few more #1 issues, so that’s good… Marcin, can you please prioritize #1? Martin, any idea about notifier? Thanks, michal

2 3

Re: [ovirt-users] Re: [ANN] oVirt 4.4.4 is now generally available
by Benny Zlotnik 22 Jan '21

22 Jan '21

On Tue, Dec 22, 2020 at 6:33 PM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote: > > Sandro, FYI we are not against cinderlib integration, more than we are upgrade 4.3 to 4.4 due movement to cinderlib. > > But (!) current Managed Storage Block realization support only krbd (kernel RBD) driver - it's also not a option, because kernel client is always lagging behind librbd, and every update\bugfix we should reboot whole host instead simple migration of all VMs and then migrate it back. Also with krbd host will be use kernel page cache, and will not be unmounted if VM will crash (qemu with librbd is one userland process). > There was rbd-nbd support at some point in cinderlib[1] which addresses your concerns, but it was removed because of some issues +Gorka, are there any plans to pick it up again? [1] https://github.com/Akrog/cinderlib/commit/a09a7e12fe685d747ed390a59cd42d0ac… > So for me current situation look like this: > > 1. We update deprecated OpenStack code? Why, Its for delete?.. Nevermind, just update this code... > > 2. Hmm... auth tests doesn't work, to pass test just disable any OpenStack project_id related things... and... Done... > > 3. I don't care how current cinder + qemu code works, just write new one for linux kernel, it's optimal to use userland apps, just add wrappers (no, it's not); > > 4. Current Cinder integration require zero configuration on oVirt hosts. It's lazy, why oVirt administrator do nothing? just write manual how-to install packages - oVirt administrators love anything except "reinstall" from engine (no, it's not); > > 5. We broke old code. New features is "Cinderlib is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production". > > 6. Oh, we broke old code. Let's deprecate them and close PRODUCTION issues (we didn't see anything). > > > And again, we are not hate new cinderlib integration. We just want that new technology don't break all PRODUCTION clustes. Almost two years ago I write on this issue https://bugzilla.redhat.com/show_bug.cgi?id=1539837#c6 about "before deprecate, let's help to migrate". For now I see that oVirt totally will disable QEMU RBD support and want to use kernel RBD module + python os-brick + userland mappers + shell wrappers. > > > Thanks, I hope I am writing this for a reason and it will help build bridges between the community and the developers. We have been with oVirt for almost 10 years and now it is a crossroads towards a different virtualization manager. > > k > > > So I see only regressions for now, hope we'll found some code owner who can catch this oVirt 4.4 only bugs. > I looked at the bugs and I see you've already identified the problem and have patches attached, if you can submit the patches and verify them perhaps we can merge the fixes

5 6

[ANN] oVirt 4.4.5 Second Release Candidate is now available for testing
by Sandro Bonazzola 22 Jan '21

22 Jan '21

oVirt 4.4.5 Second Release Candidate is now available for testing The oVirt Project is pleased to announce the availability of oVirt 4.4.5 Second Release Candidate for testing, as of January 21st, 2021. This update is the fifth in a series of stabilization updates to the 4.4 series. How to prevent hosts entering emergency mode after upgrade from oVirt 4.4.1 Note: Upgrading from 4.4.2 GA or later should not require re-doing these steps, if already performed while upgrading from 4.4.1 to 4.4.2 GA. These are only required to be done once. Due to Bug 1837864 <https://bugzilla.redhat.com/show_bug.cgi?id=1837864> - Host enter emergency mode after upgrading to latest build If you have your root file system on a multipath device on your hosts you should be aware that after upgrading from 4.4.1 to 4.4.5 you may get your host entering emergency mode. In order to prevent this be sure to upgrade oVirt Engine first, then on your hosts: 1. Remove the current lvm filter while still on 4.4.1, or in emergency mode (if rebooted). 2. Reboot. 3. Upgrade to 4.4.5 (redeploy in case of already being on 4.4.5). 4. Run vdsm-tool config-lvm-filter to confirm there is a new filter in place. 5. Only if not using oVirt Node: - run "dracut --force --add multipath” to rebuild initramfs with the correct filter configuration 6. Reboot. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. Important notes before you try it Please note this is a pre-release build. The oVirt Project makes no guarantees as to its suitability or usefulness. This pre-release must not be used in production. Installation instructions For installation instructions and additional information please refer to: https://ovirt.org/documentation/ This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer * oVirt Node 4.4 based on CentOS Linux 8.3 (available for x86_64 only) See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS Linux 8 - oVirt Node NG is already available for CentOS Linux 8 - We found a few issues while testing on CentOS Stream so we are still basing oVirt 4.4.5 Node and Appliance on CentOS Linux. Additional Resources: * Read more about the oVirt 4.4.5 release highlights: http://www.ovirt.org/release/4.4.5/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.5/ [2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

"Exception: null" after "Check if post tasks file exists" (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1892 - Failure!)
by Yedidyah Bar David 21 Jan '21

21 Jan '21

Hi all, On Thu, Jan 21, 2021 at 4:56 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1892/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1892… : 2021-01-21 03:25:52,254+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host lago-he-basic-suite-master-host-0.lago.local. Check if post tasks file exists. 2021-01-21 03:25:52,258+01 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] Exception: null 2021-01-21 03:25:52,296+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host lago-he-basic-suite-master-host-0.lago.local. Gather the rpm package facts. 2021-01-21 03:25:52,312+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host lago-he-basic-suite-master-host-0.lago.local. Check if post tasks file exists. 2021-01-21 03:25:52,316+01 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] Host installation failed for host '736e10b6-0445-4a24-935d-87ba15cae4c4', 'lago-he-basic-suite-master-host-0.lago.local': null 2021-01-21 03:25:52,321+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] START, SetVdsStatusVDSCommand(HostName = lago-he-basic-suite-master-host-0.lago.local, SetVdsStatusVDSCommandParameters:{hostId='736e10b6-0445-4a24-935d-87ba15cae4c4', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 198cd7e7 2021-01-21 03:25:52,337+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] FINISH, SetVdsStatusVDSCommand, return: , log id: 198cd7e7 2021-01-21 03:25:52,363+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [6668b493] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-he-basic-suite-master-host-0.lago.local installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1892… ends with: 2021-01-21 03:25:52 CET - TASK [Executing post tasks defined by user] ************************************ Any idea? Best regards, > Build Number: 1892 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1892 > [Andrej Cernek] pylint: Fix searched directories > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

2 3

ovirt-engine has been tagged (ovirt-engine-4.4.5.2)
by Tal Nisan 20 Jan '21

20 Jan '21

1 0

bz 1915329: [Stream] Add host fails with: Destination /etc/pki/ovirt-engine/requests not writable
by Yedidyah Bar David 20 Jan '21

20 Jan '21

Hi all, Now filed $Subject [1]. Any clues are most welcome. Thanks. Best regards, [1] https://bugzilla.redhat.com/show_bug.cgi?id=1915329 -- Didi

3 11

ovirt-engine has been tagged (ovirt-engine-4.4.5.1)
by Tal Nisan 20 Jan '21

20 Jan '21

2 1