Re: [CQ]: 87428, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause

2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins(a)ovirt.org&gt;: This fails due to multiple reason. http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/ failed on *22:23:15* E OSError: [Errno 24] Too many open files which seems to be an infra issue on the slave / dirty slave http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testRe... Host lago-upgrade-from-release-suite-master-host0 is in non responsive state The host has vdsm failures with: 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50) 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148) 2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service (check:104) 2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52) 2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run serve_clients(log) File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients cif = clientIF.getInstance(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance cls._instance = clientIF(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__ self.mom = MomClient(config.get("mom", "socket_path")) File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__ raise MomNotAvailableError() MomNotAvailableError and failure on mom side: 2018-02-22 16:34:00,168 - mom - INFO - MOM starting 2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting 2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk 2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance return JsonRpcVdsmBulkInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__ super(JsonRpcVdsmBulkInterface, self).__init__() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__ .orRaise(RuntimeError, 'No connection to VDSM.') File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise raise exception(*args, **kwargs) RuntimeError: No connection to VDSM. vdsm upgrade log shows: MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1 MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1 Involved versions: vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64 mom-0.5.12-0.0.master.el7.centos.noarch -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>