<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 27, 2018 at 2:15 PM, Sandro Bonazzola <span dir="ltr"><<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">2018-02-22 22:42 GMT+01:00 oVirt Jenkins <span dir="ltr"><<a href="mailto:jenkins@ovirt.org" target="_blank">jenkins@ovirt.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">A system test invoked by the "ovirt-master" change queue including change<br>
87428,2 (vdsm) failed. However, this change seems not to be the root cause for<br>
this failure. Change 87944,3 (vdsm) that this change depends on or is based on,<br>
was detected as the cause of the testing failures.<br>
<br>
This change had been removed from the testing queue. Artifacts built from this<br>
change will not be released until either change 87944,3 (vdsm) is fixed and<br>
this change is updated to refer to or rebased on the fixed version, or this<br>
change is modified to no longer depend on it.<br>
<br>
For further details about the change see:<br>
<a href="https://gerrit.ovirt.org/#/c/87428/2" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>7428/2</a><br>
<br>
For further details about the change that seems to be the root cause behind the<br>
testing failures see:<br>
<a href="https://gerrit.ovirt.org/#/c/87944/3" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>7944/3</a><br>
<br>
For failed test results see:<br>
<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt-master_change-queue-teste<wbr>r/5841/</a></blockquote><div><br></div><div><br></div></span><div>This fails due to multiple reason.</div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-merged-el7-<wbr>x86_64/3224/</a> failed on<br></div><div><pre class="m_2860475710338256743gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span class="m_2860475710338256743gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">22:23:15</b> </span>E OSError: [Errno 24] Too many open files</pre>which seems to be an infra issue on the slave / dirty slave</div></div></div></div></blockquote><div><br></div><div><br></div><div>Just FYI, check-merged jobs has nothing to do with OST/CQ results, they are running functional tests written by VDSM developers. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div><a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/5841/testReport/junit/(<wbr>root)/002_bootstrap/add_hosts/</a><br></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Host lago-upgrade-from-release-<wbr>suite-master-host0 is in non responsive state</pre><br></div><div>The host has vdsm failures with:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=<wbr>None) from=internal, task_id=f45a0864-9bd6-4116-<wbr>add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service (check:104)
2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-<wbr>add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/vdsmd.py", line 156, in run
serve_clients(log)
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/vdsmd.py", line 103, in serve_clients
cif = clientIF.getInstance(irs, log, scheduler)
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/clientIF.py", line 251, in getInstance
cls._instance = clientIF(irs, log, scheduler)
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/clientIF.py", line 121, in __init__
self.mom = MomClient(config.get("mom", "socket_path"))
File "/usr/lib/python2.7/site-<wbr>packages/vdsm/momIF.py", line 51, in __init__
raise MomNotAvailableError()
MomNotAvailableError</pre>and failure on mom side:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
File "/usr/lib/python2.7/site-<wbr>packages/mom/__init__.py", line 29, in run
hypervisor_iface = self.get_hypervisor_interface(<wbr>)
File "/usr/lib/python2.7/site-<wbr>packages/mom/__init__.py", line 217, in get_hypervisor_interface
return module.instance(self.config)
File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcbulkInterface.py", line 47, in instance
return JsonRpcVdsmBulkInterface()
File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcbulkInterface.py", line 29, in __init__
super(<wbr>JsonRpcVdsmBulkInterface, self).__init__()
File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcInterface.py", line 41, in __init__
.orRaise(RuntimeError, 'No connection to VDSM.')
File "/usr/lib/python2.7/site-<wbr>packages/mom/optional.py", line 28, in orRaise
raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.</pre></div><div>vdsm upgrade log shows:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">MainThread::DEBUG::2018-02-22 16:31:34,113::<wbr>libvirtconnection::167::root::<wbr>(get) trying to connect libvirt
MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::<wbr>root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::<wbr>root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::<wbr>root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::<wbr>root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::<wbr>root::(exec_cmd) /usr/share/openvswitch/<wbr>scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::<wbr>root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::INFO::2018-02-22 16:31:34,286::<wbr>netconfpersistence::198::root:<wbr>:(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-<wbr>22 16:31:34,287::fileutils::51::<wbr>root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-<wbr>22 16:31:34,287::fileutils::96::<wbr>root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22 16:31:34,287::<wbr>netconfpersistence::150::root:<wbr>:(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/<wbr>nets and /var/lib/vdsm/staging/netconf/<wbr>bonds
MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::<wbr>root::(exec_cmd) /usr/share/openvswitch/<wbr>scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::<wbr>root::(exec_cmd) FAILED: <err> = ''; <rc> = 1</pre><br></div><div>Involved versions:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">vdsm-4.30.0-70.git7d8b4e5.el7.<wbr>centos.x86_64</pre><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">mom-0.5.12-0.0.master.el7.<wbr>centos.noarch</pre><br></div><span class=""><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org" target="_blank">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/infra</a><br>
</blockquote></span></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_2860475710338256743gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div></div></div>
</div></div>