March 2019 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

OST's basic suite UI sanity tests optimization
by Marcin Sobczyk 07 Aug '19

07 Aug '19

Hi, _TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario (https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…) 01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45) Starting the Selenium grid takes 2:15 out of 3:35 of total running time! I've investigated a lot of approaches and came up with something like this: * install 'chromium-headless' package on engine VM * download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM * run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh * connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' * make screenshots This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+statu…. This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/): 20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… _The pros:_ * we cut the running time by more than 3 minutes _The cons:_ * we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) * we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests _Some design choices explained:_ Q: Why engine VM? A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot. Q: Why Chromium? A: Because it has a separate 'headless' package. Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds? A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us. _What still needs to be polished with the patches:_ * Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. * 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored. Questions, comments, reviews are welcome. Regards, Marcin

7 10

[ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]
by Dafna Ron 29 Apr '19

29 Apr '19

Hi, We are failing branch 4.3 for test: 002_bootstrap.add_master_storage_domain It seems that in one of the hosts, the vdsm is not starting there is nothing in vdsm.log or in supervdsm.log CQ identified this patch as the suspected root cause: https://gerrit.ovirt.org/#/c/98748/ - vdsm: client: Add support for flow id Milan, Marcin, can you please have a look? full logs: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/326/artifact/bas… the only error I can see is about host not being up (makes sense as vdsm is not running) Stacktrace File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 417, in add_master_storage_domain add_iscsi_storage_domain(prefix) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 561, in add_iscsi_storage_domain host=_random_host_from_dc(api, DC_NAME), File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 122, in _random_host_from_dc return _hosts_in_dc(api, dc_name, True) File "/home/jenkins/workspace/ovirt-4.3_change-queue-tester/ovirt-system-tests/basic-suite-4.3/test-scenarios/002_bootstrap.py", line 119, in _hosts_in_dc raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name) 'Could not find hosts that are up in DC test-dc\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:937bdea7-a2a3-47ad-9383-36647ea37ddf:Get ssh client for lago-basic-suite-4-3-engine:\nlago.ssh: DEBUG: end task:937bdea7-a2a3-47ad-9383-36647ea37ddf:Get ssh client for lago-basic-suite-4-3-engine:\nlago.ssh: DEBUG: Running c07b5ee2 on lago-basic-suite-4-3-engine: cat /root/multipath.txt\nlago.ssh: DEBUG: Command c07b5ee2 on lago-basic-suite-4-3-engine returned with 0\nlago.ssh: DEBUG: Command c07b5ee2 on lago-basic-suite-4-3-engine output:\n 3600140516f88cafa71243648ea218995\n360014053e28f60001764fed9978ec4b3\n360014059edc777770114a6484891dcf1\n36001405d93d8585a50d43a4ad0bd8d19\n36001405e31361631de14bcf87d43e55a\n\n-----------

8 31

Python 3 vdsm RPM packages
by Marcin Sobczyk 10 Apr '19

10 Apr '19

Hi, On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages. Some facts: - it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap Two non-exclusive propositions were raised: - let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage. The steps needed to be done are: - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only) The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...). There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K6…) so we can align the new spec file as we build it. I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us. Questions, comments are welcome. Regards, Marcin

5 12

[ OST Failure Report ] [ oVirt master (ovirt-engine) ] [ 22-03-2019 ] [004_basic_sanity.hotplug_cpu ]
by Dafna Ron 02 Apr '19

02 Apr '19

Hi, We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure. Adding Marcin, Milan and Dan as well as I think it may be netwrok related. You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifa… you can see the full logs here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac… Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm Thanks, Dafna error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script 2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory 2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected

8 17

How write OpenID Connect clients
by du_hongyu＠yeah.net 31 Mar '19

31 Mar '19

hi， I write a OpenID Connect clients follow https://github.com/cqcmdwym/passport-oidc-example https://bugzilla.redhat.com/show_bug.cgi?id=1416491 I fixed passport.use('oidc', new OidcStrategy({ issuer: 'https://10.110.128.120:443/ovirt-engine/sso/openid', authorizationURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/authorize', tokenURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/token', userInfoURL: 'https://10.110.128.120:443/ovirt-engine/sso/openid/userinfo', clientID: 'openid-test', clientSecret: '34bACHKj8VN1WOImGUKphzE4fPk3WPvAaKlQFsrQ79g', callbackURL: 'http://10.110.128.195:3000/authorization-code/callback', scope: 'openid ovirt-app-portal ovirt-app-admin ovirt-app-api ovirt-ext=token:password-access ovirt-ext=auth:sequence-priority ovirt-ext=token:login-on-behalf ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=revoke:revoke-all' error： id token not issued by correct OpenID provider - expected: https://10.110.128.120:443/ovirt-engine/sso/openid | from: https://10.110.128.120:443 Error: id token not issued by correct OpenID provider - expected: https://10.110.128.120:443/ovirt-engine/sso/openid | from: https://10.110.128.120:443 at /home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/passport-openidconnect/lib/strategy.js:113:62 at /home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:209:7 at passBackControl (/home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:134:9) at IncomingMessage.<anonymous> (/home/duhy/workSpace/nodejs/duhy/passport-oidc-example/node_modules/oauth/lib/oauth2.js:157:7) at IncomingMessage.emit (events.js:194:15) at endReadableNT (_stream_readable.js:1125:12) at process._tickCallback (internal/process/next_tick.js:63:19) Regards Hongyu Du

1 0

Failed to restart vdsmd due to kdump.service
by Eyal Shenitzky 31 Mar '19

31 Mar '19

Hi, Does someone ever encountered on failure to restart vdsmd due to the following issue: # journalctl -xe Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com kdumpctl[8926]: kexec: failed to load kdump kernel Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com kdumpctl[8926]: Starting kdump: [FAILED] Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: kdump.service: Failed with result 'exit-code'. Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com systemd[1]: Failed to start Crash recovery kernel arming. -- Subject: Unit kdump.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit kdump.service has failed. -- -- The result is failed. Mar 28 14:46:46 dhcp-0-221.tlv.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kdump comm="systemd" exe="/usr/lib/systemd/systemd" > Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: root : problem with defaults entries ; TTY=pts/0 ; PWD=/root/git/qemu/build ; USER=root ; Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_ACCT pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/b> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: root : TTY=pts/0 ; PWD=/root/git/qemu/build ; USER=root ; COMMAND=/usr/bin/chcon --reference=/usr/bin/qemu-xxx /u> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_CMD pid=9143 uid=0 auid=0 ses=1 msg='cwd="/root/git/qemu/build" cmd=6368636F6E202D2D7265666572656E63653D2F75737> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: CRED_REFR pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sud> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_systemd(sudo:session): Cannot create session: Already running in a session Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_unix(sudo:session): session opened for user root by root(uid=0) Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_START pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limi> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com sudo[9143]: pam_unix(sudo:session): session closed for user root Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: USER_END pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limit> Mar 28 14:47:15 dhcp-0-221.tlv.redhat.com audit[9143]: CRED_DISP pid=9143 uid=0 auid=0 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sud> lines 2486-2509/2509 (END) # systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2019-03-28 14:46:45 IST; 5min ago Process: 9059 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 9048 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=1/FAILURE) Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Service hold-off time over, scheduling restart. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Scheduled restart job, restart counter is at 5. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: Stopped Virtual Desktop Server Manager. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Start request repeated too quickly. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: vdsmd.service: Failed with result 'exit-code'. Mar 28 14:46:45 dhcp-0-221.tlv.redhat.com systemd[1]: Failed to start Virtual Desktop Server Manager. Occurred to me on a Fedora 28 virtual machine that runs on rhel 7.6 host. Thanks -- Regards, Eyal Shenitzky

3 3

[Ovirt] [CQ weekly status] [29-03-2019]
by Dafna Ron 29 Mar '19

29 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 25-03-2019 on project ovirt-ansible-hosted-engine-setup due missing polkit package. *CQ-4.3*: RED (#1) failures in 4.3 and master this week were caused by two issues: 1. we have had random failures on 4 different tests which we are working on debugging with Marcin, Martin and several more people. this has been disruptive this week but did not cause any delays as I was re-triggering failed projects to insure they have their packages buit in tested. Currently we suspect api issue or changes to poastgres are causing the failures 2. we had libcgroup-tools package missing which cause failures for all projects in initialize engine. I merged a patch to fix the issue and provide the package. *CQ-Master:* RED (#1) We have had the same issues as in 4.3 this week. Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

OST: add_master_storage_domain failed, Could not find hosts that are up in DC test-dc
by Yedidyah Bar David 28 Mar '19

28 Mar '19

Hi all, I want to verify [1]. So I ran the manual job, basic suite 4.3 [2]. It failed [3] with $subject. Right before that, verify_add_hosts did succeed, and took 59 seconds. I gave a brief look at the code of verify_add_hosts, and it checks that at least one host is UP. The patch [1] *might* have caused ansible-host-deploy to take longer, still not sure. In any case, ansible-host-deploy finished at 05:48:45 [5], 14 seconds before verify_add_hosts finished at 09:48:59 [4]. Can it be, that a host is considered UP (from the POV of verify_add_hosts), but is still not ready for creating storage? Also, host-1, at the point when OST was killed and collected logs, was right after finishing host-deploy, and didn't start ansible-host-deploy yet, according to engine.log. But host-0 should have been enough, I think. Thanks, [1] https://gerrit.ovirt.org/99000 [2] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [3] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [5] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… -- Didi

3 3

planned Jenkins restart
by Evgheni Dereveanchin 28 Mar '19

28 Mar '19

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. -- Regards, Evgheni Dereveanchin

1 1

Question regarding support of incremental backups in 4.3.?
by Anton E 27 Mar '19

27 Mar '19

Hi everyone, I'm quite confusing regarding Incremental backups support in 4.3. Could you please clarify it for me. The REST API contains a /backups link in links section of a VM, also the UI allows to enable incremental backup for a disk. However, the /backups returns empty result for any VM. It looks like it is a stump for future development, is that correct? Thanks in advance. Regards, Anton.

3 3

Question regarding support of incremental backups in 4.3.2
by Anton E 27 Mar '19

27 Mar '19

Hi, It seems there is an incomplete or missing support for incremental backup in 4.3.2. Need your clarifications in this regard. The API for a VM (/ovirt-engine/api/vms/<vm-id>) shows the links to <vm-id>/backups service. Also there is an option to enable incremental backup for a disk in the user interface. This makes me believe the incremental backups are supported, however accessing the <vm-id>/backups service gives an empty output. Could you please clarify, whether incremental backup feature is supported in 4.3.2 and I'm trying to use it incorrectly or is this service just a stub for now? Many thanks and regards, Anton Erdman.

2 1

[NOTICE] Failing randomly on get_host_hooks with a NullPointerException
by Dafna Ron 26 Mar '19

26 Mar '19

Hi, we are randomly failing on get_host_hooks test for at least 3 weeks. its not a specific branch or project and there are no commonalities that I can see, aside from not being able to communicate with the host. this week its started happening at least once a day (this morning, 2 out of 3 failures were due to that test). This test has been added by Yaniv Kaul over a year ago and he is no longer working on ovirt I think someone else should take ownership of this test and fix it. Please let me know if you are intending to investigate and either fix the failure or fix the test if not I will add a skip to the test, Thanks, Dafna

5 13

the mess with uploading files
by Hetz Ben Hamo 26 Mar '19

26 Mar '19

Hi, Since both export and ISO domain are being deprecated, the only place to upload remains the data domain. This method works, however, it doesn't "look" good. Uploading few ISO files and you won't see any problem. Upload 50-70 files, and the "directory" looks very messy. So, my question, will there be any "structure" then can be created, like "folders" to put those uploaded files? Thanks Hetz

2 1

ovirt-engine has been tagged (ovirt-engine-4.3.3.1)
by Tal Nisan 26 Mar '19

26 Mar '19

1 0

ovirt-engine has been tagged (ovirt-engine-4.3.3)
by Tal Nisan 26 Mar '19

26 Mar '19

1 0

vdsm has been tagged (v4.30.12)
by Milan Zamazal 26 Mar '19

26 Mar '19

1 0

FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
by Eyal Edri 26 Mar '19

26 Mar '19

Not sure if all the same issue, but seems to failing around the same time: ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57 min integ-tests ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr - #1045 51 min integ-tests ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec integ-tests ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min integ-tests ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 hr - #873 1 hr 36 min integ-tests ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr - #1149 1 hr 26 min integ-tests ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr - #729 1 hr 54 min integ-tests ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929 45 min integ-tests ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 hr - #190 45 min integ-tests Links to jobs can be found here: https://jenkins.ovirt.org/ -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

7 13

Lago's local sync repo not available in OST
by Sandro Bonazzola 25 Mar '19

25 Mar '19

Hi, just seen: 2019-03-25 10:41:36,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-hostUpdatesChecker-Thread-2) [] EVENT_ID: HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host lago-basic-suite-master-host-1 with message 'Failed to run check-update of host 'lago-basic-suite-master-host-1'. Error: fatal: [lago-basic-suite-master-host-1]: FAILED! => {"changed": false, "msg": " http://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\nhttp:// 192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nTrying other mirror.\n\n\n One of the configured repositories failed (Latest oVirt nightly),\n and yum doesn't have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work \"fix\" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=alocalsync ...\n\n 4. Disable the repository permanently, so yum won't use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable alocalsync\n or\n subscription-manager repos --disable=alocalsync\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=alocalsync.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from alocalsync: [Errno 256] No more mirrors to try.\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nhttp://192.168.201.1:8585/default/el7/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 192.168.201.1:8585; Connection refused\"\nCannot upload enabled repos report, is this client registered?\n", "rc": 1, "results": []} while looking at https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3867/… for https://gerrit.ovirt.org/98862 Can you please have a look? -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 3

[OST upgrade suite] Looking for maintainer
by Ales Musil 25 Mar '19

25 Mar '19

Hi, I am not sure who is currently maintainer for the upgrade suite, but please take a look at [1]. This patch is testing the engine upgrade issue with database vm_id and vmt_id in interface table [2]. And at the same time serves as opening patch to put more sophisticated tests into upgrade suite. Thank you. Regards, Ales [1] https://gerrit.ovirt.org/c/98656/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1547038 -- ALES MUSIL Associate Software Engineer - rhv network Red Hat EMEA <https://www.redhat.com/> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

3 6

[OST][master] hc-basic-suite-master failing on OVF_STORE not ready
by Sandro Bonazzola 25 Mar '19

25 Mar '19

Job is: https://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/1054… Logs show: *03:19:46* TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE disk content] **************03:19:47* FAILED - RETRYING: Wait for OVF_STORE disk content (12 retries left).*03:19:58* FAILED - RETRYING: Wait for OVF_STORE disk content (11 retries left).*03:20:09* FAILED - RETRYING: Wait for OVF_STORE disk content (10 retries left).*03:20:19* FAILED - RETRYING: Wait for OVF_STORE disk content (9 retries left).*03:20:30* FAILED - RETRYING: Wait for OVF_STORE disk content (8 retries left).*03:20:41* FAILED - RETRYING: Wait for OVF_STORE disk content (7 retries left).*03:20:52* FAILED - RETRYING: Wait for OVF_STORE disk content (6 retries left).*03:21:03* FAILED - RETRYING: Wait for OVF_STORE disk content (5 retries left).*03:21:14* FAILED - RETRYING: Wait for OVF_STORE disk content (4 retries left).*03:21:25* FAILED - RETRYING: Wait for OVF_STORE disk content (3 retries left).*03:21:36* FAILED - RETRYING: Wait for OVF_STORE disk content (2 retries left).*03:21:46* FAILED - RETRYING: Wait for OVF_STORE disk content (1 retries left).*03:21:57* failed: [localhost] (item={u'image_id': u'7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0', u'name': u'OVF_STORE', u'id': u'c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1'}) => {"attempts": 12, "changed": true, "cmd": "vdsm-client Image prepare storagepoolID=b9c1cfba-4ea2-11e9-b48c-00163e24d363 storagedomainID=a35d1e26-790d-4da6-90d1-df654cbfc0f3 imageID=c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1 volumeID=7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf", "delta": "0:00:00.578484", "end": "2019-03-24 22:21:57.660496", "item": {"id": "c55ba7d1-10cf-4f71-a9d2-59cf4ccb08e1", "image_id": "7df4f0be-0192-4e8b-bb6d-ee4cfd2a45d0", "name": "OVF_STORE"}, "msg": "non-zero return code", "rc": 2, "start": "2019-03-24 22:21:57.082012", "stderr": "tar: 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\n20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied, 0.00214885 s, 4.8 MB/s", "stderr_lines": ["tar: 5f87c4b6-2141-4d0c-8ce6-8bb1da97aeb6.ovf: Not found in archive", "tar: Exiting with failure status due to previous errors", "20+0 records in", "20+0 records out", "10240 bytes (10 kB) copied, 0.00214885 s, 4.8 MB/s"], "stdout": "", "stdout_lines": []} -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 1

[Ovirt] [CQ weekly status] [22-03-2019]
by Dafna Ron 22 Mar '19

22 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 21-03-2019 on project ovirt-hosted-engine-setup due to access to repo centos-cr-el7 in mirrors. this can be solved by projects setting their repo list to work with CI local mirrors. *CQ-4.3*: RED (#1) vdsm is failing on 4.3 due to failure in add_master_storage_domain. this issue is currently investigated by Marcin Sobczyk *CQ-Master:* RED (#1) we are failing on project ovirt-engine for test hotplug_cpu. this was debugged by Evgheni and myself to determine that the issue is indeed in ovirt and not in infra. The test is failing as the internal vm is not responding and we cannot ssh to vm0. *There is a random failure for test get_host_hooks. * It fails on all branches and I can see a NPE in the engine log. The test owner was Yaniv Kaul and Marcin Sobczyk is now looking into the failure. I created a skiptest patch but I am hoping that this would be resolved soon so we do not have to use it. Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

virtual floppy disk bug
by Hetz Ben Hamo 22 Mar '19

22 Mar '19

Hi, Since the ISO domain is being deprecated, it seems that when uploading a VFD to the data domain, it won't show when trying to select it in either VM configuration or Run Once windows. Which component on bugzilla should I report this bug into? Thanks

2 1

where is the vdsm-hook-nestedvt package?
by Hetz Ben Hamo 22 Mar '19

22 Mar '19

Hi, I'm testing the master branch and looking here <https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/x86_64/> it seems the package vdsm-hook-nestedvt is missing.. Thanks

1 2