[CQ]: 87428, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause

Tue Feb 27 12:23:01 UTC 2018

On Tue, Feb 27, 2018 at 2:15 PM, Sandro Bonazzola <sbonazzo at redhat.com>
wrote:

>
>
> 2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins at ovirt.org>:
>
>> A system test invoked by the "ovirt-master" change queue including change
>> 87428,2 (vdsm) failed. However, this change seems not to be the root
>> cause for
>> this failure. Change 87944,3 (vdsm) that this change depends on or is
>> based on,
>> was detected as the cause of the testing failures.
>>
>> This change had been removed from the testing queue. Artifacts built from
>> this
>> change will not be released until either change 87944,3 (vdsm) is fixed
>> and
>> this change is updated to refer to or rebased on the fixed version, or
>> this
>> change is modified to no longer depend on it.
>>
>> For further details about the change see:
>> https://gerrit.ovirt.org/#/c/87428/2
>>
>> For further details about the change that seems to be the root cause
>> behind the
>> testing failures see:
>> https://gerrit.ovirt.org/#/c/87944/3
>>
>> For failed test results see:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
>
>
>
> This fails due to multiple reason.
> http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/
> failed on
>
> *22:23:15* E       OSError: [Errno 24] Too many open files
>
> which seems to be an infra issue on the slave / dirty slave
>

Just FYI, check-merged jobs has nothing to do with OST/CQ results, they are
running functional tests written by VDSM developers.

>
>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
> tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/
>
> Host lago-upgrade-from-release-suite-master-host0 is in non responsive state
>
>
> The host has vdsm failures with:
>
> 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50)
> 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148)
> 2018-02-22 16:34:04,208-0500 INFO  (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
> 2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222)
> 2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.check] Stopping check service (check:104)
> 2018-02-22 16:34:04,221-0500 INFO  (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220)
> 2018-02-22 16:34:04,221-0500 INFO  (MainThread) [storage.udev] Stopping multipath event listener (udev:149)
> 2018-02-22 16:34:04,221-0500 INFO  (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
> 2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
>     serve_clients(log)
>   File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients
>     cif = clientIF.getInstance(irs, log, scheduler)
>   File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance
>     cls._instance = clientIF(irs, log, scheduler)
>   File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__
>     self.mom = MomClient(config.get("mom", "socket_path"))
>   File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
>     raise MomNotAvailableError()
> MomNotAvailableError
>
> and failure on mom side:
>
> 2018-02-22 16:34:00,168 - mom - INFO - MOM starting
> 2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
> 2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
> 2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
> 2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
>     hypervisor_iface = self.get_hypervisor_interface()
>   File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface
>     return module.instance(self.config)
>   File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance
>     return JsonRpcVdsmBulkInterface()
>   File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__
>     super(JsonRpcVdsmBulkInterface, self).__init__()
>   File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__
>     .orRaise(RuntimeError, 'No connection to VDSM.')
>   File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
>     raise exception(*args, **kwargs)
> RuntimeError: No connection to VDSM.
>
> vdsm upgrade log shows:
>
> MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt
> MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None)
> MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
> MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
> MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
> MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
> MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
> MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
> MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed
> MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed
> MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
> MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
> MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
>
>
> Involved versions:
>
> vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64
>
> mom-0.5.12-0.0.master.el7.centos.noarch
>
>
>
>
>
>
>>
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>
>
>
> --
>
> SANDRO BONAZZOLA
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>

-- 

Eyal edri

MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180227/40d29466/attachment-0001.html>