[CQ]: 87428, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause

Sandro Bonazzola sbonazzo at redhat.com
Tue Feb 27 12:15:09 UTC 2018 

2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins at ovirt.org>:

> A system test invoked by the "ovirt-master" change queue including change
> 87428,2 (vdsm) failed. However, this change seems not to be the root cause
> for
> this failure. Change 87944,3 (vdsm) that this change depends on or is
> based on,
> was detected as the cause of the testing failures.
>
> This change had been removed from the testing queue. Artifacts built from
> this
> change will not be released until either change 87944,3 (vdsm) is fixed and
> this change is updated to refer to or rebased on the fixed version, or this
> change is modified to no longer depend on it.
>
> For further details about the change see:
> https://gerrit.ovirt.org/#/c/87428/2
>
> For further details about the change that seems to be the root cause
> behind the
> testing failures see:
> https://gerrit.ovirt.org/#/c/87944/3
>
> For failed test results see:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/



This fails due to multiple reason.
http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/
failed on

*22:23:15* E       OSError: [Errno 24] Too many open files

which seems to be an infra issue on the slave / dirty slave


http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/

Host lago-upgrade-from-release-suite-master-host0 is in non responsive state


The host has vdsm failures with:

2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC
interface is disabled (momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init
clientIF, shutting down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO  (MainThread) [vdsm.api] START
prepareForShutdown(options=None) from=internal,
task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.Monitor]
Shutting down domain monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.check]
Stopping check service (check:104)
2018-02-22 16:34:04,221-0500 INFO  (check/loop) [storage.asyncevent]
Stopping <EventLoop running=True closed=False at 0x38225104>
(asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO  (MainThread) [storage.udev]
Stopping multipath event listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO  (MainThread) [vdsm.api] FINISH
prepareForShutdown return=None from=internal,
task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised
(vdsmd:158)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
    serve_clients(log)
  File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in
serve_clients
    cif = clientIF.getInstance(irs, log, scheduler)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251,
in getInstance
    cls._instance = clientIF(irs, log, scheduler)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121,
in __init__
    self.mom = MomClient(config.get("mom", "socket_path"))
  File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
    raise MomNotAvailableError()
MomNotAvailableError

and failure on mom side:

2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to
VDSM! [Errno 111] Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
    hypervisor_iface = self.get_hypervisor_interface()
  File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217,
in get_hypervisor_interface
    return module.instance(self.config)
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 47, in instance
    return JsonRpcVdsmBulkInterface()
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 29, in __init__
    super(JsonRpcVdsmBulkInterface, self).__init__()
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py",
line 41, in __init__
    .orRaise(RuntimeError, 'No connection to VDSM.')
  File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
    raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.

vdsm upgrade log shows:

MainThread::DEBUG::2018-02-22
16:31:34,113::libvirtconnection::167::root::(get) trying to connect
libvirt
MainThread::DEBUG::2018-02-22
16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb
-disable pcmcia -disable isapnp -disable ide -disable scsi -disable
dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = '';
<rc> = 0
MainThread::DEBUG::2018-02-22
16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd
None)
MainThread::DEBUG::2018-02-22
16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = '';
<rc> = 0
MainThread::DEBUG::2018-02-22
16:31:34,273::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc>
= 1
MainThread::INFO::2018-02-22
16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing
netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-22
16:31:34,287::fileutils::51::root::(rm_file) File:
/var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-22
16:31:34,287::fileutils::96::root::(rm_tree) Directory:
/var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22
16:31:34,287::netconfpersistence::150::root::(save) Saved new config
RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and
/var/lib/vdsm/staging/netconf/bonds
MainThread::DEBUG::2018-02-22
16:31:34,288::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc>
= 1


Involved versions:

vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64

mom-0.5.12-0.0.master.el7.centos.noarch






>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>



-- 

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180227/2831f760/attachment-0001.html>

More information about the Infra mailing list