- Devel - oVirt List Archives

Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1818 - Failure!
by Yedidyah Bar David 15 Nov '20

15 Nov '20

On Sat, Nov 14, 2020 at 5:29 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1818/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1818… Error Message 'module' object has no attribute 'guest_os_image_name' Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 88, in <module> GUEST_IMAGE_NAME = versioning.guest_os_image_name() AttributeError: 'module' object has no attribute 'guest_os_image_name' Seems to be caused by: https://gerrit.ovirt.org/112196 Can you please have a look? Thanks. Best regards, -- Didi

1 0

ovirt-engine has been tagged (ovirt-engine-4.4.3.12)
by Tal Nisan 13 Nov '20

13 Nov '20

2 1

Travis builds
by Vojtech Juranek 11 Nov '20

11 Nov '20

Hi, recently, I noticed Travis builds waits in the queue for a very long time. Looking around I found out Travis probably reduced resources for OSS projects. Also I found out all projects should migrate from travis-ci.org to travis- ci.com by end of December 2020 [1] (see Q. When will the migration from travis-ci.org to travis-ci.com be completed?). Recently they announce new pricing model [2] when OSS projects will get some initial credit and after that either has to pay or ask the support for another credit (see section "Building on a public repositories only" in [2]). Maybe time to migrate away from travis-ci to something else, e.g. GH Actions? Thoughts? Vojta [1] https://docs.travis-ci.com/user/migrate/open-source-repository-migration [2] https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

3 4

ovirt-engine has been tagged (ovirt-engine-4.4.4)
by Tal Nisan 11 Nov '20

11 Nov '20

1 0

[ANN] oVirt 4.4.3 is now generally available
by Sandro Bonazzola 10 Nov '20

10 Nov '20

oVirt 4.4.3 is now generally available The oVirt project is excited to announce the general availability of oVirt 4.4.3 , as of November 10th, 2020. This release unleashes an altogether more powerful and flexible open source virtualization solution that encompasses hundreds of individual changes and a wide range of enhancements across the engine, storage, network, user interface, and analytics, as compared to oVirt 4.3. Important notes before you install / upgrade Please note that oVirt 4.4 only supports clusters and data centers with compatibility version 4.2 and above. If clusters or data centers are running with an older compatibility version, you need to upgrade them to at least 4.2 (4.3 is recommended). Please note that in RHEL 8 / CentOS 8 several devices that worked on EL7 are no longer supported. For example, the megaraid_sas driver is removed. If you use Enterprise Linux 8 hosts you can try to provide the necessary drivers for the deprecated hardware using the DUD method (See the users’ mailing list thread on this at https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NDSVUZSESOXEFJ… ) Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. What’s new in oVirt 4.4.3 Release? This update is the third in a series of stabilization updates to the 4.4 series. This release is available now on x86_64 architecture for: - Red Hat Enterprise Linux 8.2 / 8.3 - CentOS Linux (or similar) 8.2 - CentOS Stream (tech preview) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: - Red Hat Enterprise Linux 8.2 / 8.3 - CentOS Linux (or similar) 8.2 - CentOS Stream (tech preview) Please note that this release introduces cluster compatibility level 4.5 which is supported only with Red Hat Enterprise Linux 8.3 or equivalent. oVirt Node and Appliance have been updated, including: - oVirt 4.4.3: https://www.ovirt.org/release/4.4.3/ - Ansible 2.9.15: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.… - Glusterfs 7.8: https://docs.gluster.org/en/latest/release-notes/7.8/ See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS Linux 8 - oVirt Node NG will be available soon for CentOS Linux 8 Additional resources: - Read more about the oVirt 4.4.3 release highlights: https://www.ovirt.org/release/4.4.3/ - Get more oVirt project updates on Twitter: https://twitter.com/ovirt - Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/release/4.4.3/ <https://www.ovirt.org/release/4.4.3/> [2] https://resources.ovirt.org/pub/ovirt-4.4/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Change compatibility level for data center/cluster during upgrade - what exactly are we changing?
by Patrick Chiang 10 Nov '20

10 Nov '20

Hi all, From the upgrade procedure, when we completed most upgrade tasks, we have to change the compatibility level in cluster and data center, and then restart all guest VM afterwards.It seems no down time expected with these 2 changes from the procedure. I wonder what exactly we are changing (a setting in the database/file??) and most important of all, whether there will be a few seconds down time when we made those changes to "data center"? Because I worried if a data center change could lead to storage domain's down time and eventually force hypervisors, guest VM restart before expected time. (My hosts/guest VM directly use HP 3par's storage as boot disks/data disks...) Hope anyone can share any experience on this. Thank you, Patrick Chiang

2 1

[OST] Network suites fails, 6+ days runtime?
by Nir Soffer 09 Nov '20

09 Nov '20

I have a trivial patch: https://gerrit.ovirt.org/c/112068 This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_sta… - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_sta… The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results! Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable. Can someone look at the network suite failure? Should we remove it from the test matrix? Nir

3 4

How to set up a (rh)el8 machine for running OST
by Marcin Sobczyk 09 Nov '20

09 Nov '20

Hi All, there are multiple pieces of information floating around on how to set up a machine for running OST. Some of them outdated (like dealing with el7), some of them more recent, but still a bit messy. Not long ago, in some email conversation, Milan presented an ansible playbook that provided the steps necessary to do that. We've picked up the playbook, tweaked it a bit, made a convenience shell script wrapper that runs it, and pushed that into OST project [1]. This script, along with the playbook, should be our single-source-of-truth, one-stop solution for the job. It's been tested by a couple of persons and proved to be able to set up everything on a bare (rh)el8 machine. If you encounter any problems with the script please either report it on the devel mailing list, directly to me, or simply file a patch. Let's keep it maintained. Regards, Marcin [1] https://gerrit.ovirt.org/#/c/111749/

6 14

virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
by Yedidyah Bar David 09 Nov '20

09 Nov '20

Hi all, On Mon, Oct 12, 2020 at 5:17 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Above failed with: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… vdsm.log has: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… 2020-10-11 22:05:14,695-0400 INFO (jsonrpc/1) [api.host] FINISH getJobs return={'jobs': {'05eaea44-7e4c-4442-9926-2bcb696520f1': {'id': '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed', 'description': 'sparsify_volume', 'job_type': 'storage', 'error': {'code': 100, 'message': 'General Exception: (\'Command [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\', \\\'--in-place\\\', \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\'] failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error: guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command again. For further information, read:\\\\\\\\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the *complete* output\\\\\\\\ninto a bug report or message to the libguestfs mailing list.", "timestamp": "2020-10-11T22:05:08.397538670-04:00", "type": "error" }\\\\n\\\' err=b"virt-sparsify: error: libguestfs error: guestfs_launch failed.\\\\nThis usually means the libguestfs appliance failed to start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\nand run the command again. For further information, read:\\\\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou can also run \\\'libguestfs-test-tool\\\' and post the *complete* output\\\\ninto a bug report or message to the libguestfs mailing list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify -v -x [...]\\\\n"\',)'}}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.201.4,43318, flow_id=365642f4-2fe2-45df-937a-f4ca435eea38 (api:54) 2020-10-11 22:05:14,695-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] Return 'Host.getJobs' in bridge with {'05eaea44-7e4c-4442-9926-2bcb696520f1': {'id': '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed', 'description': 'sparsify_volume', 'job_type': 'storage', 'error': {'code': 100, 'message': 'General Exception: (\'Command [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\', \\\'--in-place\\\', \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\'] failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error: guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command again. For further information, read:\\\\\\\\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the *complete* output\\\\\\\\ninto a bug report or message to the libguestfs mailing list.", "timestamp": "2020-10-11T22:05:08.397538670-04:00", "type": "error" }\\\\n\\\' err=b"virt-sparsify: error: libguestfs error: guestfs_launch failed.\\\\nThis usually means the libguestfs appliance failed to start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\nand run the command again. For further information, read:\\\\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou can also run \\\'libguestfs-test-tool\\\' and post the *complete* output\\\\ninto a bug report or message to the libguestfs mailing list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify -v -x [...]\\\\n"\',)'}}} (__init__:356) /var/log/messages has: Oct 11 22:04:51 lago-basic-suite-master-host-0 kvm[80601]: 1 guest now active Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain id=1 name='guestfs-hl0ntvn92rtkk2u0' uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: custom-argv Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain id=1 name='guestfs-hl0ntvn92rtkk2u0' uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: host-cpu Oct 11 22:05:06 lago-basic-suite-master-host-0 kvm[80801]: 2 guests now active Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: internal error: End of file from qemu monitor Oct 11 22:05:08 lago-basic-suite-master-host-0 kvm[80807]: 1 guest now active Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot resolve symlink /tmp/libguestfseTG8xF/console.sock: No such file or directoryKN<F3>L^? Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot resolve symlink /tmp/libguestfseTG8xF/guestfsd.sock: No such file or directoryKN<F3>L^? Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot lookup default selinux label for /tmp/libguestfs1WkcF7/overlay1.qcow2 Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/kernel Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/initrd Oct 11 22:05:08 lago-basic-suite-master-host-0 vdsm[74096]: ERROR Job '05eaea44-7e4c-4442-9926-2bcb696520f1' failed#012Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/vdsm/jobs.py", line 159, in run#012 self._run()#012 File "/usr/lib/python3.6/site-packages/vdsm/storage/sdm/api/sparsify_volume.py", line 57, in _run#012 virtsparsify.sparsify_inplace(self._vol_info.path)#012 File "/usr/lib/python3.6/site-packages/vdsm/virtsparsify.py", line 40, in sparsify_inplace#012 commands.run(cmd)#012 File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 101, in run#012 raise cmdutils.Error(args, p.returncode, out, err)#012vdsm.common.cmdutils.Error: Command ['/usr/bin/virt-sparsify', '--machine-readable', '--in-place', '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67'] failed with rc=1 out=b'3/12\n{ "message": "libguestfs error: guestfs_launch failed.\\nThis usually means the libguestfs appliance failed to start or crashed.\\nDo:\\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\nand run the command again. For further information, read:\\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\nYou can also run \'libguestfs-test-tool\' and post the *complete* output\\ninto a bug report or message to the libguestfs mailing list.", "timestamp": "2020-10-11T22:05:08.397538670-04:00", "type": "error" }\n' err=b"virt-sparsify: error: libguestfs error: guestfs_launch failed.\nThis usually means the libguestfs appliance failed to start or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand run the command again. For further information, read:\n http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can also run 'libguestfs-test-tool' and post the *complete* output\ninto a bug report or message to the libguestfs mailing list.\n\nIf reporting bugs, run virt-sparsify with debugging enabled and include the \ncomplete output:\n\n virt-sparsify -v -x [...]\n" The next run of the job (480) did finish successfully. No idea if it was already fixed by a patch, or is simply a random/env issue. Is it possible to pass LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 without patching vdsm? Not sure. In any case, if this does not cause too much excess debug logging, perhaps better always pass it, to help retroactively analyze such failures. Or, patch virt-sparsify/libguestfs/whatever to always log at least enough information on failure even without passing these. Best regards, > Build Number: 479 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #479 > [hbraha] network: bond active slave test > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_sparsify_disk1 > > Error Message: > AssertionError: False != True after 600 seconds > > Stack Trace: > api_v4 = <ovirtsdk4.Connection object at 0x7fe717c60e50> > > @order_by(_TEST_LIST) > def test_sparsify_disk1(api_v4): > engine = api_v4.system_service() > disk_service = test_utils.get_disk_service(engine, DISK1_NAME) > with test_utils.TestEvent(engine, 1325): # USER_SPARSIFY_IMAGE_START event > disk_service.sparsify() > > with test_utils.TestEvent(engine, 1326): # USER_SPARSIFY_IMAGE_FINISH_SUCCESS > > pass > > ../basic-suite-master/test-scenarios/004_basic_sanity.py:295: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > /usr/lib64/python2.7/contextlib.py:24: in __exit__ > self.gen.next() > ../ost_utils/ost_utils/engine_utils.py:44: in wait_for_event > lambda: > ../ost_utils/ost_utils/assertions.py:97: in assert_true_within_long > assert_equals_within_long(func, True, allowed_exceptions) > ../ost_utils/ost_utils/assertions.py:82: in assert_equals_within_long > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > func = <function <lambda> at 0x7fe7176dfb18>, value = True, timeout = 600 > allowed_exceptions = [], initial_wait = 0 > error_message = 'False != True after 600 seconds' > > def assert_equals_within( > func, value, timeout, allowed_exceptions=None, initial_wait=10, > error_message=None > ): > allowed_exceptions = allowed_exceptions or [] > with _EggTimer(timeout) as timer: > while not timer.elapsed(): > try: > res = func() > if res == value: > return > except Exception as exc: > if _instance_of_any(exc, allowed_exceptions): > time.sleep(3) > continue > > LOGGER.exception("Unhandled exception in %s", func) > raise > > if initial_wait == 0: > time.sleep(3) > else: > time.sleep(initial_wait) > initial_wait = 0 > try: > if error_message is None: > error_message = '%s != %s after %s seconds' % (res, value, timeout) > > raise AssertionError(error_message) > E AssertionError: False != True after 600 seconds > > ../ost_utils/ost_utils/assertions.py:60: AssertionError_______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/65FECSE7EBWNS… -- Didi

5 14

OST check-patch jobs are stuck on archive artifacts stage
by Amit Bawer 08 Nov '20

08 Nov '20

Hi, Exemplified here, started last night and still hasn't finished archiving: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/13195… Thanks.

2 2