<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2018-02-22 22:42 GMT+01:00 oVirt Jenkins <span dir="ltr"><<a href="mailto:jenkins@ovirt.org" target="_blank">jenkins@ovirt.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">A system test invoked by the "ovirt-master" change queue including change<br>
87428,2 (vdsm) failed. However, this change seems not to be the root cause for<br>
this failure. Change 87944,3 (vdsm) that this change depends on or is based on,<br>
was detected as the cause of the testing failures.<br>
<br>
This change had been removed from the testing queue. Artifacts built from this<br>
change will not be released until either change 87944,3 (vdsm) is fixed and<br>
this change is updated to refer to or rebased on the fixed version, or this<br>
change is modified to no longer depend on it.<br>
<br>
For further details about the change see:<br>
<a href="https://gerrit.ovirt.org/#/c/87428/2" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>87428/2</a><br>
<br>
For further details about the change that seems to be the root cause behind the<br>
testing failures see:<br>
<a href="https://gerrit.ovirt.org/#/c/87944/3" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>87944/3</a><br>
<br>
For failed test results see:<br>
<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/5841/</a></blockquote><div><br></div><div><br></div><div>This fails due to multiple reason.</div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/">http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/</a> failed on<br></div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">22:23:15</b> </span>E OSError: [Errno 24] Too many open files</pre>which seems to be an infra issue on the slave / dirty slave</div><div><br></div><div><br></div><div><a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/</a><br></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Host lago-upgrade-from-release-suite-master-host0 is in non responsive state</pre><br></div><div>The host has vdsm failures with:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service (check:104)
2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
serve_clients(log)
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients
cif = clientIF.getInstance(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance
cls._instance = clientIF(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__
self.mom = MomClient(config.get("mom", "socket_path"))
File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
raise MomNotAvailableError()
MomNotAvailableError</pre>and failure on mom side:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
hypervisor_iface = self.get_hypervisor_interface()
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface
return module.instance(self.config)
File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance
return JsonRpcVdsmBulkInterface()
File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__
super(JsonRpcVdsmBulkInterface, self).__init__()
File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__
.orRaise(RuntimeError, 'No connection to VDSM.')
File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.</pre></div><div>vdsm upgrade log shows:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1</pre><br></div><div>Involved versions:</div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64</pre><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">mom-0.5.12-0.0.master.el7.centos.noarch</pre><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>