March 2019 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

OST's basic suite UI sanity tests optimization
by Marcin Sobczyk 07 Aug '19

07 Aug '19

Hi, _TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario (https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…) 01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45) Starting the Selenium grid takes 2:15 out of 3:35 of total running time! I've investigated a lot of approaches and came up with something like this: * install 'chromium-headless' package on engine VM * download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM * run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh * connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' * make screenshots This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+statu…. This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/): 20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… _The pros:_ * we cut the running time by more than 3 minutes _The cons:_ * we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) * we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests _Some design choices explained:_ Q: Why engine VM? A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot. Q: Why Chromium? A: Because it has a separate 'headless' package. Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds? A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us. _What still needs to be polished with the patches:_ * Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. * 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored. Questions, comments, reviews are welcome. Regards, Marcin

7 10

[ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]
by Dafna Ron 29 Apr '19

29 Apr '19

Hi, We are failing branch 4.3 for test: 002_bootstrap.add_master_storage_domain It seems that in one of the hosts, the vdsm is not starting there is nothing in vdsm.log or in supervdsm.log CQ identified this patch as the suspected root cause: https://gerrit.ovirt.org/#/c/98748/ - vdsm: client: Add support for flow id Milan, Marcin, can you please have a look? full logs: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/326/artifact/bas… the only error I can see is about host not being up (makes sense as vdsm is not running) Stacktrace File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 417, in add_master_storage_domain add_iscsi_storage_domain(prefix) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 561, in add_iscsi_storage_domain host=_random_host_from_dc(api, DC_NAME), File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 122, in _random_host_from_dc return _hosts_in_dc(api, dc_name, True) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 119, in _hosts_in_dc raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name) 'Could not find hosts that are up in DC test-dc\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:937bdea7-a2a3-47ad-9383-36647ea37ddf:Get ssh client for lago-basic-suite-4-3-engine:\nlago.ssh: DEBUG: end task:937bdea7-a2a3-47ad-9383-36647ea37ddf:Get ssh client for lago-basic-suite-4-3-engine:\nlago.ssh: DEBUG: Running c07b5ee2 on lago-basic-suite-4-3-engine: cat /root/multipath.txt\nlago.ssh: DEBUG: Command c07b5ee2 on lago-basic-suite-4-3-engine returned with 0\nlago.ssh: DEBUG: Command c07b5ee2 on lago-basic-suite-4-3-engine output:\n 3600140516f88cafa71243648ea218995\n360014053e28f60001764fed9978ec4b3\n360014059edc777770114a6484891dcf1\n36001405d93d8585a50d43a4ad0bd8d19\n36001405e31361631de14bcf87d43e55a\n\n-----------

8 31

Python 3 vdsm RPM packages
by Marcin Sobczyk 10 Apr '19

10 Apr '19

Hi, On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages. Some facts: - it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap Two non-exclusive propositions were raised: - let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage. The steps needed to be done are: - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only) The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...). There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K6…) so we can align the new spec file as we build it. I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us. Questions, comments are welcome. Regards, Marcin

5 12

[ OST Failure Report ] [ oVirt master (ovirt-engine) ] [ 22-03-2019 ] [004_basic_sanity.hotplug_cpu ]
by Dafna Ron 02 Apr '19

02 Apr '19

Hi, We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure. Adding Marcin, Milan and Dan as well as I think it may be netwrok related. You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifa… you can see the full logs here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac… Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm Thanks, Dafna error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script 2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory 2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected

8 17

How write OpenID Connect clients
by du_hongyu＠yeah.net 31 Mar '19

31 Mar '19

hi， I write a OpenID Connect clients follow https://github.com/cqcmdwym/passport-oidc-example https://bugzilla.redhat.com/show_bug.cgi?id=1416491 I fixed passport.use('oidc', new OidcStrategy({ issuer: 'https://10.110.128.120:443/ovirt-engine/sso/openid', authorizationURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/authorize', tokenURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/token', userInfoURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/userinfo', clientID: 'openid-test', clientSecret: '34bACHKj8VN1WOImGUKphzE4fPk3WPvAaKlQFsrQ79g', callbackURL: 'http://10.110.128.195:3000/authorization-code/callback', scope: 'openid ovirt-app-portal ovirt-app-admin ovirt-app-api ovirt-ext=token:password-access ovirt-ext=auth:sequence-priority ovirt-ext=token:login-on-behalf ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=revoke:revoke-all' error： id token not issued by correct OpenID provider - expected: https://10.110.128.120:443/ovirt-engine/sso/openid | from: https://10.110.128.120:443 Error: id token not issued by correct OpenID provider - expected: https://10.110.128.120:443/ovirt-engine/sso/openid | from: https://10.110.128.120:443 at /home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/passport-openidconnect/lib/strategy.js:113:62 at /home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:209:7 at passBackControl (/home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:134:9) at IncomingMessage.<anonymous> (/home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:157:7) at IncomingMessage.emit (events.js:194:15) at endReadableNT (_stream_readable.js:1125:12) at process._tickCallback (internal/process/next_tick.js:63:19) Regards Hongyu Du

1 0

Failed to restart vdsmd due to kdump.service
by Eyal Shenitzky 31 Mar '19

31 Mar '19

Hi, Does someone ever encountered on failure to restart vdsmd due to the following issue: # journalctl -xe Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com kdumpctl[8926]: kexec: failed to load kdump kernel Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com kdumpctl[8926]: Starting kdump: [FAILED] Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: kdump.service: Failed with result 'exit-code'. Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: Failed to start Crash recovery kernel arming. -- Subject: Unit kdump.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit kdump.service has failed. -- -- The result is failed. Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kdump comm="systemd" exe="/usr/lib/systemd/systemd" > Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: root : problem with defaults entries ; TTY=pts/0 ; PWD=/root/git/qemu/build ; USER=root ; Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_ACCT pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/b> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: root : TTY=pts/0 ; PWD=/root/git/qemu/build ; USER=root ; COMMAND=/usr/bin/chcon --reference=/usr/bin/qemu-xxx /u> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_CMD pid=9143 uid=0 auid=0 ses=1 msg='cwd="/root/git/qemu/build" cmd=6368636F6E202D2D7265666572656E63653D2F75737> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: CRED_REFR pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sud> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_systemd(sudo:session): Cannot create session: Already running in a session Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_unix(sudo:session): session opened for user root by root(uid=0) Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_START pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limi> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_unix(sudo:session): session closed for user root Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_END pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limit> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: CRED_DISP pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sud> lines 2486-2509/2509 (END) # systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2019-03-28 14:46:45 IST; 5min ago Process: 9059 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 9048 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=1/FAILURE) Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Service hold-off time over, scheduling restart. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Scheduled restart job, restart counter is at 5. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: Stopped Virtual Desktop Server Manager. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Start request repeated too quickly. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Failed with result 'exit-code'. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: Failed to start Virtual Desktop Server Manager. Occurred to me on a Fedora 28 virtual machine that runs on rhel 7.6 host. Thanks -- Regards, Eyal Shenitzky

3 3

[Ovirt] [CQ weekly status] [29-03-2019]
by Dafna Ron 29 Mar '19

29 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 25-03-2019 on project ovirt-ansible-hosted-engine-setup due missing polkit package. *CQ-4.3*: RED (#1) failures in 4.3 and master this week were caused by two issues: 1. we have had random failures on 4 different tests which we are working on debugging with Marcin, Martin and several more people. this has been disruptive this week but did not cause any delays as I was re-triggering failed projects to insure they have their packages buit in tested. Currently we suspect api issue or changes to poastgres are causing the failures 2. we had libcgroup-tools package missing which cause failures for all projects in initialize engine. I merged a patch to fix the issue and provide the package. *CQ-Master:* RED (#1) We have had the same issues as in 4.3 this week. Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

OST: add_master_storage_domain failed, Could not find hosts that are up in DC test-dc
by Yedidyah Bar David 28 Mar '19

28 Mar '19

Hi all, I want to verify [1]. So I ran the manual job, basic suite 4.3 [2]. It failed [3] with $subject. Right before that, verify_add_hosts did succeed, and took 59 seconds. I gave a brief look at the code of verify_add_hosts, and it checks that at least one host is UP. The patch [1] *might* have caused ansible-host-deploy to take longer, still not sure. In any case, ansible-host-deploy finished at 05:48:45 [5], 14 seconds before verify_add_hosts finished at 09:48:59 [4]. Can it be, that a host is considered UP (from the POV of verify_add_hosts), but is still not ready for creating storage? Also, host-1, at the point when OST was killed and collected logs, was right after finishing host-deploy, and didn't start ansible-host-deploy yet, according to engine.log. But host-0 should have been enough, I think. Thanks, [1] https://gerrit.ovirt.org/99000 [2] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [3] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [5] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… -- Didi

3 3

planned Jenkins restart
by Evgheni Dereveanchin 28 Mar '19

28 Mar '19

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. -- Regards, Evgheni Dereveanchin

1 1

Question regarding support of incremental backups in 4.3.?
by Anton E 27 Mar '19

27 Mar '19

Hi everyone, I'm quite confusing regarding Incremental backups support in 4.3. Could you please clarify it for me. The REST API contains a /backups link in links section of a VM, also the UI allows to enable incremental backup for a disk. However, the /backups returns empty result for any VM. It looks like it is a stump for future development, is that correct? Thanks in advance. Regards, Anton.

3 3

Question regarding support of incremental backups in 4.3.2
by Anton E 27 Mar '19

27 Mar '19

Hi, It seems there is an incomplete or missing support for incremental backup in 4.3.2. Need your clarifications in this regard. The API for a VM (/ovirt-engine/api/vms/<vm-id>) shows the links to <vm-id>/backups service. Also there is an option to enable incremental backup for a disk in the user interface. This makes me believe the incremental backups are supported, however accessing the <vm-id>/backups service gives an empty output. Could you please clarify, whether incremental backup feature is supported in 4.3.2 and I'm trying to use it incorrectly or is this service just a stub for now? Many thanks and regards, Anton Erdman.

2 1

[NOTICE] Failing randomly on get_host_hooks with a NullPointerException
by Dafna Ron 26 Mar '19

26 Mar '19

Hi, we are randomly failing on get_host_hooks test for at least 3 weeks. its not a specific branch or project and there are no commonalities that I can see, aside from not being able to communicate with the host. this week its started happening at least once a day (this morning, 2 out of 3 failures were due to that test). This test has been added by Yaniv Kaul over a year ago and he is no longer working on ovirt I think someone else should take ownership of this test and fix it. Please let me know if you are intending to investigate and either fix the failure or fix the test if not I will add a skip to the test, Thanks, Dafna

5 13

the mess with uploading files
by Hetz Ben Hamo 26 Mar '19

26 Mar '19

Hi, Since both export and ISO domain are being deprecated, the only place to upload remains the data domain. This method works, however, it doesn't "look" good. Uploading few ISO files and you won't see any problem. Upload 50-70 files, and the "directory" looks very messy. So, my question, will there be any "structure" then can be created, like "folders" to put those uploaded files? Thanks Hetz

2 1

ovirt-engine has been tagged (ovirt-engine-4.3.3.1)
by Tal Nisan 26 Mar '19

26 Mar '19

1 0

ovirt-engine has been tagged (ovirt-engine-4.3.3)
by Tal Nisan 26 Mar '19

26 Mar '19

1 0

vdsm has been tagged (v4.30.12)
by Milan Zamazal 26 Mar '19

26 Mar '19

1 0

FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
by Eyal Edri 26 Mar '19

26 Mar '19

Not sure if all the same issue, but seems to failing around the same time: ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57 min integ-tests ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr - #1045 51 min integ-tests ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec integ-tests ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min integ-tests ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 hr - #873 1 hr 36 min integ-tests ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr - #1149 1 hr 26 min integ-tests ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr - #729 1 hr 54 min integ-tests ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929 45 min integ-tests ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 hr - #190 45 min integ-tests Links to jobs can be found here: https://jenkins.ovirt.org/ -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

7 13

Lago's local sync repo not available in OST
by Sandro Bonazzola 25 Mar '19

25 Mar '19

Hi, just seen: 2019-03-25 10:41:36,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-hostUpdatesChecker-Thread-2) [] EVENT_ID: HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host lago-basic-suite-master-host-1 with message 'Failed to run check-update of host 'lago-basic-suite-master-host-1'. Error: fatal: [lago-basic-suite-master-host-1]: FAILED! => {"changed": false, "msg": " http://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\n\n\n One of the configured repositories failed (Latest oVirt nightly),\n and yum doesn't have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work \"fix\" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=alocalsync ...\n\n 4. Disable the repository permanently, so yum won't use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable alocalsync\n or\n subscription-manager repos --disable=alocalsync\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=alocalsync.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from alocalsync: [Errno 256] No more mirrors to try.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nCannot upload enabled repos report, is this client registered?\n", "rc": 1, "results": []} while looking at https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3867/… for https://gerrit.ovirt.org/98862 Can you please have a look? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 3

[OST upgrade suite] Looking for maintainer
by Ales Musil 25 Mar '19

25 Mar '19

Hi, I am not sure who is currently maintainer for the upgrade suite, but please take a look at [1]. This patch is testing the engine upgrade issue with database vm_id and vmt_id in interface table [2]. And at the same time serves as opening patch to put more sophisticated tests into upgrade suite. Thank you. Regards, Ales [1] https://gerrit.ovirt.org/c/98656/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1547038 -- ALES MUSIL Associate Software Engineer - rhv network Red Hat EMEA <https://www.redhat.com/> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

3 6

[OST][master] hc-basic-suite-master failing on OVF_STORE not ready
by Sandro Bonazzola 25 Mar '19

25 Mar '19

Job is: https://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/1054… Logs show: *03:19:46* TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE disk content] **************03:19:47* FAILED - RETRYING: Wait for OVF_STORE disk content (12 retries left).*03:19:58* FAILED - RETRYING: Wait for OVF_STORE disk content (11 retries left).*03:20:09* FAILED - RETRYING: Wait for OVF_STORE disk content (10 retries left).*03:20:19* FAILED - RETRYING: Wait for OVF_STORE disk content (9 retries left).*03:20:30* FAILED - RETRYING: Wait for OVF_STORE disk content (8 retries left).*03:20:41* FAILED - RETRYING: Wait for OVF_STORE disk content (7 retries left).*03:20:52* FAILED - RETRYING: Wait for OVF_STORE disk content (6 retries left).*03:21:03* FAILED - RETRYING: Wait for OVF_STORE disk content (5 retries left).*03:21:14* FAILED - RETRYING: Wait for OVF_STORE disk content (4 retries left).*03:21:25* FAILED - RETRYING: Wait for OVF_STORE disk content (3 retries left).*03:21:36* FAILED - RETRYING: Wait for OVF_STORE disk content (2 retries left).*03:21:46* FAILED - RETRYING: Wait for OVF_STORE disk content (1 retries left).*03:21:57* failed: [localhost] (item={u'image_id': u'7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0', u'name': u'OVF_STORE', u'id': u'c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1'}) => {"attempts": 12, "changed": true, "cmd": "vdsm-client Image prepare storagepoolID=b9c1cfba-4ea2-11e9-b48c-00163e24d363 storagedomainID=a35d1e26-790d-4da6-90d1-df654cbfc0f3 imageID=c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1 volumeID=7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf", "delta": "0:00:00.578484", "end": "2019-03-24 22:21:57.660496", "item": {"id": "c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1", "image_id": "7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0", "name": "OVF_STORE"}, "msg": "non-zero return code", "rc": 2, "start": "2019-03-24 22:21:57.082012", "stderr": "tar: 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\n20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied, 0.00214885 s, 4.8 MB/s", "stderr_lines": ["tar: 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf: Not found in archive", "tar: Exiting with failure status due to previous errors", "20+0 records in", "20+0 records out", "10240 bytes (10 kB) copied, 0.00214885 s, 4.8 MB/s"], "stdout": "", "stdout_lines": []} -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 1

[Ovirt] [CQ weekly status] [22-03-2019]
by Dafna Ron 22 Mar '19

22 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 21-03-2019 on project ovirt-hosted-engine-setup due to access to repo centos-cr-el7 in mirrors. this can be solved by projects setting their repo list to work with CI local mirrors. *CQ-4.3*: RED (#1) vdsm is failing on 4.3 due to failure in add_master_storage_domain. this issue is currently investigated by Marcin Sobczyk *CQ-Master:* RED (#1) we are failing on project ovirt-engine for test hotplug_cpu. this was debugged by Evgheni and myself to determine that the issue is indeed in ovirt and not in infra. The test is failing as the internal vm is not responding and we cannot ssh to vm0. *There is a random failure for test get_host_hooks. * It fails on all branches and I can see a NPE in the engine log. The test owner was Yaniv Kaul and Marcin Sobczyk is now looking into the failure. I created a skiptest patch but I am hoping that this would be resolved soon so we do not have to use it. Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

virtual floppy disk bug
by Hetz Ben Hamo 22 Mar '19

22 Mar '19

Hi, Since the ISO domain is being deprecated, it seems that when uploading a VFD to the data domain, it won't show when trying to select it in either VM configuration or Run Once windows. Which component on bugzilla should I report this bug into? Thanks

2 1

where is the vdsm-hook-nestedvt package?
by Hetz Ben Hamo 22 Mar '19

22 Mar '19

Hi, I'm testing the master branch and looking here <https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/x86_64/> it seems the package vdsm-hook-nestedvt is missing.. Thanks

1 2

Is only master branch of ovirt-engine analyzed on coverity scan?
by simtiaz＠ncsu.edu 21 Mar '19

21 Mar '19

I see the github project ovirt-engine (https://github.com/oVirt/ovirt-engine) is registered on Coverity Scan for routine analysis (https://scan.coverity.com/projects/ovirt-engine?tab=project_settings) My question is - 1) do we only build and analyze the master branch on Coverity? 1a) If yes, why do we not analyze the other branches?

3 3

Usage of ntpdate in OST suites
by Sandro Bonazzola 21 Mar '19

21 Mar '19

Hi, just noticed today we are calling "ntpdate -4 <engine fqdn>" within OST suites but then it fails with "ntpdate: command not found" and we ignore it. I think "chronyc -4 makestep" should be used instead after having configure the engine as server. Any insight on this? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 1

Is only master branch of ovirt-engine analyzed on coverity scan?
by simtiaz＠ncsu.edu 20 Mar '19

20 Mar '19

I see the github project ovirt-engine (https://github.com/oVirt/ovirt-engine) is registered on Coverity Scan for routine analysis (https://scan.coverity.com/projects/ovirt-engine?tab=project_settings) My question is - 1) do we only build and analyze the master branch on Coverity? 1a) If yes, why do we not analyze the other branches? 1b) If no, how to detect which alert on coverity is for which branch? Thanks, Nasif

1 0

ovirt-jboss-modules-maven-plugin package
by Sandro Bonazzola 20 Mar '19

20 Mar '19

Hi, just noticed today we are building on master a package named ovirt-jboss-modules-maven-plugin. The package is not required by any other package at spec file / rpm level and it has never been shipped in an official release as far as I can tell. Grep says we are using a java package through maven called ovirt-jboss-modules-maven-plugin in: - ovirt-engine - ovirt-engine-api-metamodel It doesn't seem we are bundling the library so it looks like a build time dependency only. Am I correct? Why aren't we adding it as BuildRequires to ovirt-engine spec file and keeping downloading it from maven central at every build instead of using system installed library? Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 3

Increase limits on concurrent https://jenkins.ovirt.org/job/standard-manual-runner/ jobs
by Nir Soffer 20 Mar '19

20 Mar '19

We started to use this job to build artifacts for running OST. Looks like this job is limited to one concurrent job, which make it less useful than it could be. If we don't make it easy to run OST, people will not run it, and the change queue will break. Please change the limit to use the same limit used for regular builds. We seem to be able to create more then 10 concurrent builds by uploading patches to gerrit. The manual runner should this as well. Nir

1 0

Check-Merged Fails for ovirt-provider-ovn
by Anton Marchukov 20 Mar '19

20 Mar '19

Hello. I note that check-merged for ovirt-provider-ovn [1] fails on the following: > assert len(networks) == 1 E AssertionError: assert 2 == 1 E + where 2 = len([{'id': 'e0e73fc4-88e1-41e4-91c9-c9519c02f319', 'mtu': 1442, 'name': 'net1', 'port_security_enabled': False, ...}, {'id': '34d40d1f-b2e3-4eb9-9202-6d311162ea5a', 'mtu': 1442, 'name': 'ls0', 'port_security_enabled': False, ...}]) test_provider_api.py:72: AssertionError seems like the unit tests are failing. This is not OST failure itself, but it happens right before that on the build stage and hence this project fails in the Change Queue, so we need to check it. [1] https://jenkins.ovirt.org/job/ovirt-provider-ovn_standard-on-merge/263/ Anton. -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat

5 14

ovirt-vdsmfake
by Sandro Bonazzola 20 Mar '19

20 Mar '19

Hi, on master we have ovirt-vdsmfake package. Is there any reason for branching it and/or add it to 4.3 pipeline? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

4 7

oVirt Orb 4.3.2 build failure importing image
by Sandro Bonazzola 20 Mar '19

20 Mar '19

Hi, I was trying to build oVirt Orb with 4.3.2 GA content and it failed with: 2019-03-20 13:53:13,210::utils.py::_ret_via_queue::62::lago.utils::DEBUG::Error while running thread Thread-11 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-demo-tool_4.3_check-patch-el7-x86_64/ovirt-demo-tool/deploy/sdk_scripts/deploy/002_bootstrap.py", line 875, in import_non_template_from_glance generic_import_from_glance(prefix) File "/home/jenkins/workspace/ovirt-demo-tool_4.3_check-patch-el7-x86_64/ovirt-demo-tool/deploy/sdk_scripts/deploy/002_bootstrap.py", line 711, in generic_import_from_glance target_image.import_image(import_action) AttributeError: 'NoneType' object has no attribute 'import_image' Lago log is here: https://jenkins.ovirt.org/job/ovirt-demo-tool_4.3_check-patch-el7-x86_64/15… If I understood it correctly, it's importing an image from glance to export domain, but if I remember correctly, export domains are deprecated in 4.3 so no reason to do this here. Any insight? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 1

How to debug engine backend code
by du_hongyu＠yeah.net 20 Mar '19

20 Mar '19

hi I can step through frontend code follow https://www.ovirt.org/blog/2017/08/ovirt-webadmin-gwt-debug-quick-refresh.h…, I want to step through backend code, now I only print log to debug backend code, But i do not how to debug backend code by other method , can you give me some advise? Regards Hongyu Du

2 1

Re: Cannot add fedora 28 host to oVirt
by Eyal Shenitzky 20 Mar '19

20 Mar '19

Added devel On Wed, Mar 20, 2019 at 9:05 AM Yedidyah Bar David <didi(a)redhat.com> wrote: > On Tue, Mar 19, 2019 at 3:49 PM Eyal Shenitzky <eshenitz(a)redhat.com> > wrote: > > > > Hi, > > > > I tried to add Fedora 28 host to ovirt and failed due to missing > ovirt-host-deploy: > > > > 2019-03-19 14:27:34,795+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (VdsDeploy) [1ce188c7] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An > error has occurred during installation of Host 10.35.1.234: Internal > error: No module named ovirt_host_deploy. > > 2019-03-19 14:27:34,823+02 ERROR > [org.ovirt.engine.core.uutils.ssh.SSHDialog] > (EE-ManagedThreadFactory-engine-Thread-2355) [1ce188c7] SSH error running > command root(a)10.35.1.234:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" > mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar > --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy > DIALOG/dialect=str:machine DIALOG/customization=bool:True': IOException: > Command returned failure code 2 during SSH session 'root(a)10.35.1.234' > > 2019-03-19 14:27:34,824+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1ce188c7] > Error during deploy dialog > > 2019-03-19 14:27:34,838+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-2355) [1ce188c7] Error during host > 10.35.1.234 install > > > > When I tried to install ovirt-host-deploy on the Fedora 28 machine I > failed because otopi was missing. > > > > Do we have a known issue for Fedora 28 host? > > Nothing that I can currently recall, although it's been some time since I > tried. > > The missing ovirt_host_deploy/otopi are not something you should > install on the host. They are included in the bundle. > > Where is the engine (OS)? Is it current master? dev-env or RPM? Which > otopi/ovirt-host-deploy you have there? Current master, both? > The engine is also on Fedora 28, this is the current master (dev-env) host-deploy: python2-ovirt-host-deploy-1.8.0-0.0.master.20181016080559.gitff6ea7d.fc28.noarch ovirt-host-deploy -common-1.8.0-0.0.master.20181016080559.gitff6ea7d.fc28.noarch otopi: python2-otopi-1.8.0-0.0.master.20181017100341.git487c83a.fc28.noarch python3-otopi-1.8.0-0.0.master.20181017100341.git487c83a.fc28.noarch otopi-common-1.8.0-0.0.master.20181017100341.git487c83a.fc28.noarch > > Please check/share content of the bundle. After you try 'Add host', it > should be in > /var/cache/ovirt-engine/ovirt-host-deploy.tar/var/cache/ovirt-engine/ovirt-host-deploy.tar > . It's > auto-generated, so you can remove it and try again. > What should I look for there? > Also, IMO you can open such threads on devel(a)ovirt.org.il. > > Best regards, > -- > Didi > -- Regards, Eyal Shenitzky

2 4

dwh failing on master, perhaps because engine is 4.4
by Yedidyah Bar David 20 Mar '19

20 Mar '19

Hi all, https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… fails with: ... lambda: engine.service('ovirt-engine-dwhd').alive() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 282, in assert_true_within_short assert_equals_within_short(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 266, in assert_equals_within_short func, value, SHORT_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) 'False != True after 180 seconds dwhd log has: 2019-03-20 03:07:06|5iRyfJ|KapTFh|KapTFh|OVIRT_ENGINE_DWH|MinimalVersionCheck|Default|5|tDie|tDie_1|2019-03-20 03:07:06|You have upgraded your oVirt Engine and now require an upgrade of the ovirt-engine-dwh package. Please run engine-setup to upgrade the version. Service will now exit.|4 I assume that this is because the engine is now 4.4 and dwh master is still 4.3. I see that dwh already branched 4.3, should we bump master to 4.4? Thanks and best regards, -- Didi

1 0

Re: OST failures (Update to use gluster 5)
by Sahina Bose 20 Mar '19

20 Mar '19

On Wed, Mar 20, 2019 at 12:31 AM Greg Sheremeta <gshereme(a)redhat.com> wrote: > Hey, > > Is someone looking at all the OST failures? > he, hc, and network are all failing with the 409 error. Started after the > merge of > "Update to use gluster 5" > I'm not positive that's the problem, but seems likely. > > Do you mean https://gerrit.ovirt.org/#/c/98470/ ? this should have affected only the hc-master suite Both hc and he suites are failing with this error..this is probably not related to gluster, as he suite does not use gluster. Adding Simone and devel ml for insights "tar: ebaaaefb-5d8c-4059-a59e-b81cdb09e001.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\n20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied, 0.00144612 s, 7.1 MB/s", "stderr_lines": ["tar: ebaaaefb-5d8c-4059-a59e-b81cdb09e001.ovf: Not found in archive" > Best wishes, > Greg > > -- > > GREG SHEREMETA > > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX > > Red Hat NA > > <https://www.redhat.com/> > > gshereme(a)redhat.com IRC: gshereme > <https://red.ht/sig> >

2 3

oVirt Engine 4.3 has been branched
by Tal Nisan 19 Mar '19

19 Mar '19

Which means that if you want your content in Engine 4.3.3 and onwards you should backport to the ovirt-engine-4.3 branch, master now basically serves as the future branch (4.4)

2 1

Are we ready to branch ovirt-engine for 4.3?
by Sandro Bonazzola 18 Mar '19

18 Mar '19

Hi, we built 4.3.2 RC2 this week and we have a planned GA for it next week. Since branching engine requires a bit of coordination, are we ready to branch it? Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

3 8

Missed ovirt-openshift-extensions release?
by Sandro Bonazzola 18 Mar '19

18 Mar '19

Hi, just seen https://twitter.com/RealGophersShip/status/1105791600762986496 Are we missing a release of ovirt-openshift-extensions? It wasn't included in 4.3.2 RC2 Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 7

oci: command line tool for oVirt CI
by Nir Soffer 18 Mar '19

18 Mar '19

I want to share a nice little tool that can make your life easier. https://github.com/nirs/oci With this PR: https://github.com/nirs/oci/pull/1 You will be able to do this: $ python system-tests.py 98535 2019-03-15 21:17:04,579 INFO [system-tests] [ 1/8 ] Getting build info for change 98535 2019-03-15 21:17:05,295 INFO [system-tests] [ 2/8 ] Starting build-artifacts job for {'url': u'git://gerrit.ovirt.org/vdsm', 'ref': u'refs/changes/35/98535/3'} 2019-03-15 21:17:06,134 INFO [system-tests] [ 3/8 ] Waiting for queue item https://jenkins.ovirt.org/queue/item/382806/ 2019-03-15 21:17:07,960 INFO [system-tests] [ 4/8 ] Waiting for job http://jenkins.ovirt.org/job/standard-manual-runner/89/ 2019-03-15 21:26:21,381 INFO [system-tests] [ 5/8 ] Starting oVirt system tests basic suite with custom repos http://jenkins.ovirt.org/job/standard-manual-runner/89/ 2019-03-15 21:26:22,242 INFO [system-tests] [ 6/8 ] Waiting for queue item https://jenkins.ovirt.org/queue/item/382817/ 2019-03-15 21:30:43,558 INFO [system-tests] [ 7/8 ] Waiting for job http://jenkins.ovirt.org/job/ovirt-system-tests_manual/4330/ 2019-03-15 22:10:52,891 INFO [system-tests] [ 8/8 ] System tests completed with SUCCESS Yes, one command from your favorite shell. It takes about 53 minutes, but on Fedora you get notifications when commands finish. No need to to add "ci please build" comment, wait until the job is complete, copy the job url, open OST page, wait (jenkins is slow), paste the build artifacts url, start the job, wait, wait more, check if OST is done, (not yet), wait more..., OST finished, check the logs, did you paste the wrong url? The tool is not finished yet. If you like to contribute to this fun little project please check the issues: https://github.com/nirs/oci/issues Most of the issues are easy. Some harder like: - https://github.com/nirs/oci/issues/11 - https://github.com/nirs/oci/issues/12 Feel free to open new issues if you have an idea for this tool. Nir

5 5

Dependencies issue on manual OST
by Eyal Shenitzky 18 Mar '19

18 Mar '19

Hi, I encounter this dependency issue while trying to run OST manually: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4343/ *09:49:35* @ Deploy oVirt environment: *09:49:35* # Deploy environment: *09:49:35* * [Thread-2] Deploy VM lago-basic-suite-master-host-0: *09:49:35* * [Thread-3] Deploy VM lago-basic-suite-master-host-1: *09:49:35* * [Thread-4] Deploy VM lago-basic-suite-master-engine: *09:49:54* * [Thread-3] Deploy VM lago-basic-suite-master-host-1: Success (in 0:00:18) *09:49:56* STDERR*09:49:56* + yum -y install ovirt-host*09:49:56* Error: Package: ovirt-host-dependencies-4.4.0-0.0.master.20190308162916.git4c2aa13.el7.x86_64 (alocalsync)*09:49:56* Requires: vdsm-client >= 4.40.0*09:49:56* Available: vdsm-client-4.30.11-10.gitf23ba52.el7.noarch (alocalsync)*09:49:56* vdsm-client = 4.30.11-10.gitf23ba52.el7*09:49:56* Error: Package: ovirt-host-dependencies-4.4.0-0.0.master.20190308162916.git4c2aa13.el7.x86_64 (alocalsync)*09:49:56* Requires: vdsm >= 4.40.0*09:49:56* Installing: vdsm-4.30.11-10.gitf23ba52.el7.x86_64 (alocalsync)*09:49:56* vdsm = 4.30.11-10.gitf23ba52.el7*09:49:56* Error: Package: ovirt-host-dependencies-4.4.0-0.0.master.20190308162916.git4c2aa13.el7.x86_64 (alocalsync)*09:49:56* Requires: vdsm-client >= 4.40.0*09:49:56* Installing: vdsm-client-4.30.11-10.gitf23ba52.el7.noarch (alocalsync)*09:49:56* vdsm-client = 4.30.11-10.gitf23ba52.el7 Does someone encounter it? Thanks -- Regards, Eyal Shenitzky

3 3

OST hyperconverged master fails
by Greg Sheremeta 17 Mar '19

17 Mar '19

Hi, I'm trying to run OST hyperconverged master locally, but it's missing some ansible inventory file and fails. Is this a known issue? + lago copy-to-vm lago-hc-basic-suite-master-host-0 /home/greg/projects/ovirt-system-tests/hc-basic-suite-master/generate-hc-answerfile.sh /root/generate-hc-answerfile.sh @ Copy /home/greg/projects/ovirt-system-tests/hc-basic-suite-master/generate-hc-answerfile.sh to lago-hc-basic-suite-master-host-0:/root/generate-hc-answerfile.sh: @ Copy /home/greg/projects/ovirt-system-tests/hc-basic-suite-master/generate-hc-answerfile.sh to lago-hc-basic-suite-master-host-0:/root/generate-hc-answerfile.sh: Success (in 0:00:00) + lago shell lago-hc-basic-suite-master-host-0 /root/exec_playbook.sh lago-hc-basic-suite-master-host-0 lago-hc-basic-suite-master-host-1 lago-hc-basic-suite-master-host-2 */root/exec_playbook.sh: line 9: /usr/share/doc/gluster.ansible/playbooks/hc-ansible-deployment/ohc_gluster_inventory.yml: No such file or directory* + RET_CODE=1 + '[' 1 -ne 0 ']' + echo 'ansible setup on lago-hc-basic-suite-master-host-0 failed with status 1.' ansible setup on lago-hc-basic-suite-master-host-0 failed with status 1. + exit 1 + on_exit + [[ 1 -ne 0 ]] + logger.error 'on_exit: Exiting with a non-zero status' + logger.log ERROR 'on_exit: Exiting with a non-zero status' + set +x 2019-03-15 06:16:36.021775858-0400 run_suite.sh::on_exit::ERROR:: on_exit: Exiting with a non-zero status -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 4

[Ovirt] [CQ weekly status] [15-03-2019]
by Dafna Ron 15 Mar '19

15 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 14-03-2019 on project v2v-conversion-host due to missing failed build artifacts which was fixed the same day. *CQ-4.3*: RED (#1) Last job to failed today. it had 13 changes and CQ is still bisecting the result. *CQ-Master:* RED (#1) Last failure was today due to a build-artifacts failure. a message was sent to patch owner. Current running jobs for 4.2 [1] and master [2] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

Re: [URGENT] stop merging to vdsm master
by Sandro Bonazzola 13 Mar '19

13 Mar '19

Adding missing devel list. Il giorno mar 12 mar 2019 alle ore 10:24 Dafna Ron <dron(a)redhat.com> ha scritto: > Hi, > > We have been continuously failing CQ on the vdsm project since 07-03-2019. > > The latest failure is still in vm migration test and root cause is still > pointing to: > > https://gerrit.ovirt.org/#/c/97381/22 - virt: In FIPS mode, use SASL auth > instead of qemu passwords > > Please stop merging until issue is resolved as we are risking further > regressions being merged. > > Thanks, > Dafna > > -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

5 6

[hyperv] hvinfo: command line tool to get the HyperV enlightements status from within the guest
by Francesco Romani 12 Mar '19

12 Mar '19

Hi all, lately I've been involved again in hyperv support. I was tasked to help improve the testability of the hyperv guest configuration. Hyperv support is a set of optimizations that libvirt/qemu offer to improve the runtime behaviour (stability, performance) of Windows guets. See for example: https://libvirt.org/formatdomain.html#elementsFeatures oVirt does support them (see for example https://github.com/oVirt/vdsm/blob/master/lib/vdsm/virt/libvirtxml.py#L243) and will keep supporting them because they are a key feature for windows guests. Up until now the easiest (only?) way to check if a hyperv optmization was really enabled was to inspect the libvirt XML and/or the QEMU command line flags But we wanted to have another check. Enter hvinfo: https://github.com/fromanirh/hvinfo hvinfo is a simple tool that decodes CPUID informations according to the publicly-available HyperV specs (https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/live/tlfs) and report what the guest sees. It takes no arguments - just run it!-, it requires no special privileges and emits easy to consume JSON. It is designed to help and integrate into fully automated CI/QA. Being a commandline tool, is hard to give a "screenshot", so let me just report sample output (admin privileges not actually required) Windows PowerShell Copyright (C) Microsoft Corporation. All rights reserved. PS C:\Users\admin> cd .\Downloads\ PS C:\Users\admin\Downloads> .\hvinfo.exe { "HyperVsupport": true, "Features": { "GuestDebugging": false, "PerformanceMonitor": false, "PCPUDynamicPartitioningEvents": true, "HypercallInputParamsXMM": false, "VirtualGuestIdleState": false, "HypervisorSleepState": false, "NUMADistanceQuery": false, "TimerFrequenciesQuery": false, "SytheticMCEInjection": false, "GuestCrashMSR": false, "DebugMSR": false, "NPIEP": false, "DisableHypervisorAvailable": false, "ExtendedGvaRangesForFlushVirtualAddressList": false, "HypercallOutputReturnXMM": false, "SintPollingMode": false, "HypercallMsrLock": false, "UseDirectSyntheticTimers": false }, "Recommendations": { "HypercallAddressSpaceSwitch": false, "HypercallLocalTLBFlush": false, "HypercallRemoteTLBFlush": false, "MSRAPICRegisters": true, "MSRSystemReset": false, "RelaxedTiming": true, "DMARemapping": false, "InterruptRemapping": false, "X2APICMSR": false, "DeprecatingAutoEOI": false, "SyntheticClusterIPI": false, "ExProcessorMasks": false, "Nested": false, "INTForMBECSyscalls": false, "NestedEVMCS": false, "SyncedTimeline": false, "DirectLocalFlushEntire": false, "NoNonArchitecturalCoreSharing": false, "SpinlockRetries": 8191 } } PS C:\Users\admin\Downloads> Caveat: the name of the features are the same of the spec, so we need mappings for oVirt flags, libvirt flags and so on. For example libvirt xml domain.features.hyperv.relaxed[state="on"] maps to hvinfo json Features.Recommendations.RelaxedTiming=true Feel free to test it and report any issue Bests, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

2 1

ovirt-engine has been tagged (ovirt-engine-4.3.2.1)
by Tal Nisan 12 Mar '19

12 Mar '19

1 0

vdsm has been tagged (v4.30.11)
by Milan Zamazal 11 Mar '19

11 Mar '19

3 3

unboundid-ldapsdk-4.0.10 is now available for testing
by Sandro Bonazzola 11 Mar '19

11 Mar '19

On Fedora 28 we have 4.0.7. On Fedora 30 we had 4.0.9. The new releases 4.0.10 will be soon available within repositories for Fedora >= 30 and includes several bug fixes and enhancements. You can see release notes for 4.0.10 here: https://github.com/pingidentity/ldapsdk/releases In oVirt this package is required by ovirt-engine-extension-aaa-ldap. If you're going to test the package, please provide karma at https://bodhi.fedoraproject.org/updates/unboundid-ldapsdk-4.0.10-1.fc30 Martin, Ravi, let me know if you see value in backporting this version to Fedora 28 and EL7 for oVirt 4.3 consumption. Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

[ OST Failure Report ] [ oVirt 4.3 (vdsm-jsonrpc-java) ] [ 10-3-19 ] [ Upgrade Suite ]
by Liora Milbaum 11 Mar '19

11 Mar '19

Link and headline of suspected patches: Link to Job: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/153/ Link to all logs: (Relevant) error snippet from the log: <error> [36m # 001_upgrade_engine.test_initialize_engine: [0m [0m [0m [36m * Copy /home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/upgrade-engine-answer-file.conf to lago-upgrade-from-prevrelease-suite-4-3-engine:/tmp/answer-file-post: [0m [0m [0m [36m * Copy /home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/upgrade-engine-answer-file.conf to lago-upgrade-from-prevrelease-suite-4-3-engine:/tmp/answer-file-post: [32mSuccess [0m (in 0:00:00) [0m [36m * Collect artifacts: [0m [0m [0m [36m ~ [Thread-8] Copy from lago-upgrade-from-prevrelease-suite-4-3-engine:/tmp/otopi* to /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.3/default/test_logs/001_upgrade_engine.test_initialize_engine-20190307160642/lago-upgrade-from-prevrelease-suite-4-3-engine/_tmp_otopi*: [31mERROR [0m (in 0:00:00) [0m [36m - [Thread-8] lago-upgrade-from-prevrelease-suite-4-3-engine: [31mERROR [0m (in 0:00:00) [0m [36m * Collect artifacts: [31mERROR [0m (in 0:00:01) [0m [36m # 001_upgrade_engine.test_initialize_engine: [31mERROR [0m (in 0:00:49) [0m [36m # Results located at /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.3/001_upgrade_engine.py.junit.xml [0m [36m@ Run test: 001_upgrade_engine.py: [31mERROR [0m (in 0:00:49) [0m [31mError occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 383, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 549, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 560, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 107, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed [0m </error> -- *Liora Milbaum* Senior Principal Software Engineer RHV/CNV DevOps EMEA VIRTUALIZATION R&D T: +972-54-6560051 <https://red.ht/sig>

3 2