January 2017 - Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 195
by jenkins＠jenkins.phx.ovirt.org 09 Jan '17

09 Jan '17

------=_Part_206_639461436.1484002804607 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/195//artifact/exported-a… Cheers, Jenkins ------=_Part_206_639461436.1484002804607 Content-Type: text/html; charset=us-ascii; name=upstream_report.html Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=upstream_report.html Content-ID: <upstream_report.html> <!DOCTYPE html><head><style type="text/css"> table.gridtable { border-collapse: collapse; table-layout:fixed; width:1600px; font-family: monospace; font-size:13px; } .head { font-size:20px; font-family: arial; } .sub { font-size:18px; background-color:#e5e5e5; font-family: arial; } pre { font-family: monospace; display: inline; white-space: pre-wrap; white-space: -moz-pre-wrap !important; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } </style> </head> <body> <table class="gridtable" border=2> <tr><th colspan=2 class=head> RHEVM CI Jenkins Daily Report - 09/01/2017 </th></tr><tr><th colspan=2 class=sub> <font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font> </th></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_3.6/">deploy-to-ovirt_experimental_3.6</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_build-…">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-fc24-x86…">ovirt-system-tests_master_check-patch-fc24-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/">ovirt_3.6_he-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_3.6_image-ng-system-tests/">ovirt_3.6_image-ng-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/">ovirt_4.1_system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_4.0_check-closure_fc23_merged/">repos_4.0_check-closure_fc23_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_4.1_check-closure_el7_merged/">repos_4.1_check-closure_el7_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_4.1_check-closure_fc24_merged/">repos_4.1_check-closure_fc24_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_master_check-closure_el7_merged/">repos_master_check-closure_el7_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/">system-backup_jenkins_old_ovirt_org</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/">test-repo_ovirt_experimental_master</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> ------=_Part_206_639461436.1484002804607--

1 0

Fwd: ** PROBLEM Service Alert: ovirt-mirrorchecker/www.gtlib.gatech.edu/pub/oVirt/pub mirror site last sync is WARNING **
by Nadav Goldin 09 Jan '17

09 Jan '17

Hi, Seems like the ovirt mirror went out of sync again, can you have a look? Thanks, Nadav. ---------- Forwarded message ---------- From: icinga <icinga(a)monitoring.ovirt.org> Date: Mon, Jan 9, 2017 at 11:51 PM Subject: ** PROBLEM Service Alert: ovirt-mirrorchecker/www.gtlib.gatech.edu/pub/oVirt/pub mirror site last sync is WARNING ** To: ngoldin(a)redhat.com ***** Icinga ***** Notification Type: PROBLEM Service: www.gtlib.gatech.edu/pub/oVirt/pub mirror site last sync Host: ovirt-mirrorchecker Address: 66.187.230.105 State: WARNING Date/Time: Mon Jan 9 21:51:19 UTC 2017 Additional Info: WARNING - 237938 seconds since last sync, which are 66.0939 hours.

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4643 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 09 Jan '17

09 Jan '17

------=_Part_182_2036524778.1483975245955 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/, Build Number: 4643, Build Status: FAILURE ------=_Part_182_2036524778.1483975245955--

1 3

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #174 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 09 Jan '17

09 Jan '17

------=_Part_188_202513061.1483983491038 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/174/, Build Number: 174, Build Status: FAILURE ------=_Part_188_202513061.1483983491038--

1 3

New failure in OST - master branch: add secondary storage domains fails
by Nadav Goldin 09 Jan '17

09 Jan '17

Hi, There is a new failure on on master in experimental flow, the failing test is 'add_secondary_storage_domain', the engine.log has few exceptions: 2017-01-09 10:07:24,943-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Command 'PollVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call 2017-01-09 10:07:24,943-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] ... 2017-01-09 10:10:23,323-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-09 10:10:23,323-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] ... 2017-01-09 10:10:43,704-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name at [Source: [B@6a84a0d0; line: 1, column: 889]: org.codehaus.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name at [Source: [B@6a84a0d0; line: 1, column: 889] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName2(ReaderBasedParser.java:1042) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName(ReaderBasedParser.java:1008) [jackson-core-asl-1.9.13.jar:1.9.13] .... <JsonRpcRequest id: "7711f770-dbef-44be-9f9e-2d8a2bfae937", method: Host.getAllVmStats, params: {}> 2017-01-09 10:11:33,336-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParameters Base:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Unrecognized message received 2017-01-09 10:11:33,336-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNe tworkException: Unrecognized message received at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] VDSM logs on host1: 2017-01-09 10:11:27,120 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) 2017-01-09 10:11:27,452 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for unfetched domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:151) 2017-01-09 10:11:27,453 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:168) 2017-01-09 10:11:27,552 WARN (jsonrpc/4) [storage.LVM] lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).' , ' Volume group "80985016-bdd8-4778-abd9-becc8fedcab4" not found', ' Cannot process volume group 80985016-bdd8-4778-abd9-becc8fedcab4'] (lvm:377) 2017-01-09 10:11:27,559 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) 2017-01-09 10:11:27,560 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='e2381f1f-eee5-4922-a56d-f6ca40d76eec') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1159, in attachStorageDomain pool.attachSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 924, in attachSD dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 112, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) .... 2017-01-09 10:19:31,467 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2212, in getAllTasksInfo raise se.SpmStatusError() SpmStatusError: Not SPM: () 2017-01-09 10:19:31,471 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175) 2017-01-09 10:19:31,471 ERROR (jsonrpc/6) [storage.Dispatcher] {'status': {'message': 'Not SPM: ()', 'code': 654}} (dispatcher:77) 2017-01-09 10:19:31,472 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksInfo failed (error 654) in 0.01 seconds (__init__:515) 2017-01-09 10:19:31,479 INFO (jsonrpc/7) [dispatcher] Run and protect: getAllTasksStatuses(spUUID=None, options=None) (logUtils:49) 2017-01-09 10:19:31,479 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2172, in getAllTasksStatuses raise se.SpmStatusError() SpmStatusError: Not SPM: () 2017-01-09 10:19:31,480 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175) Full engine logs can be found here: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/artif… VDSM host1 logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/artif… Rest of the logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/artif… Could someone take a look? Thanks, Nadav.

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #172 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 09 Jan '17

09 Jan '17

------=_Part_184_297438184.1483980082426 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/172/, Build Number: 172, Build Status: FAILURE ------=_Part_184_297438184.1483980082426--

1 1

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #169 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 09 Jan '17

09 Jan '17

------=_Part_178_1890642216.1483971936570 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/169/, Build Number: 169, Build Status: FAILURE ------=_Part_178_1890642216.1483971936570--

1 1

[JIRA] (OVIRT-1012) ovirt-engine-sdk is build on el6 in following branches master, sdk_4.1 and sdk_4.0
by Ondra Machacek (oVirt JIRA) 09 Jan '17

09 Jan '17

Ondra Machacek created OVIRT-1012: ------------------------------------- Summary: ovirt-engine-sdk is build on el6 in following branches master, sdk_4.1 and sdk_4.0 Key: OVIRT-1012 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1012 Project: oVirt - virtualization made easy Issue Type: Bug Components: Jenkins Reporter: Ondra Machacek Assignee: infra Hello, The ovirt-engine-sdk is build on el6 in following branches master, sdk_4.1 and sdk_4.0. This isn't correct, it specified branches the project should be build only for el7, fc24 and fc25. Thank you. CC [~jhernand(a)redhat.com] -- This message was sent by Atlassian JIRA (v1000.670.2#100024)

1 0

jsonrpc build failures
by Piotr Kliczewski 09 Jan '17

09 Jan '17

I pushed a patch and I can see that all the jobs failed for it: http://jenkins.ovirt.org/job/vdsm-jsonrpc-java_master_check-patch-el7-x86_6… : FAILURE http://jenkins.ovirt.org/job/vdsm-jsonrpc-java_master_check-patch-fc24-x86_… : FAILURE http://jenkins.ovirt.org/job/vdsm-jsonrpc-java_4.1_check-patch-fc24-x86_64/… : FAILURE http://jenkins.ovirt.org/job/vdsm-jsonrpc-java_4.1_check-patch-el7-x86_64/3/ : FAILURE Can someone please take a look? Thanks, Piotr

1 0

The feature everyone was asking for is finally here...
by Eyal Edri 09 Jan '17

09 Jan '17

FYI, After many requests from multiple developers and testers, the oVirt CI added a new simple job that lets you run the full fledged end-to-end oVirt system tests with a click of a button. You can read all the details and how-to in the new oVirt blog [1]. We wanted to allow running oVirt system tests on EVERY open patch from ANY oVirt project, without relaying on complex building code inside the job. Luckily we just added the 'build-on-demand' so together with it you can build any rpms you'd like and use them to run the manual job. So the 2 steps you'll need to do are: 1. Write 'ci please build' inside a comment on an open oVirt patch ( make sure the feature is enabled for that project first, its already available for ovirt-engine,vdsm,dashboard and vdsm-jsonrpc-java) 2. Run the manual OST job for the version you'd like to test with the URLs you got from #1 You'll get and email once the job is done and you can browse the results and check for logs from engine and the hosts. Please feel free to ask questions on infra(a)ovirt.org as usual. [1] https://www.ovirt.org/blog/2017/01/ovirt-system-tests-to-the-rescue/ -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)

4 3