[CQ]: 87428, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
A system test invoked by the "ovirt-master" change queue including change 87428,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 87944,3 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 87944,3 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/87428/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/87944/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
A system test invoked by the "ovirt-master" change queue including change 87428,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 87944,3 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 87944,3 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it.
For further details about the change see: https://gerrit.ovirt.org/#/c/87428/2
For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/87944/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
This fails due to multiple reason. http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/ failed on *22:23:15* E OSError: [Errno 24] Too many open files which seems to be an infra issue on the slave / dirty slave http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testRepor... Host lago-upgrade-from-release-suite-master-host0 is in non responsive state The host has vdsm failures with: 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50) 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148) 2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service (check:104) 2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52) 2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run serve_clients(log) File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients cif = clientIF.getInstance(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance cls._instance = clientIF(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__ self.mom = MomClient(config.get("mom", "socket_path")) File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__ raise MomNotAvailableError() MomNotAvailableError and failure on mom side: 2018-02-22 16:34:00,168 - mom - INFO - MOM starting 2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting 2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk 2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance return JsonRpcVdsmBulkInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__ super(JsonRpcVdsmBulkInterface, self).__init__() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__ .orRaise(RuntimeError, 'No connection to VDSM.') File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise raise exception(*args, **kwargs) RuntimeError: No connection to VDSM. vdsm upgrade log shows: MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1 MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1 Involved versions: vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64 mom-0.5.12-0.0.master.el7.centos.noarch
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Tue, Feb 27, 2018 at 2:15 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
A system test invoked by the "ovirt-master" change queue including change 87428,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 87944,3 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 87944,3 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it.
For further details about the change see: https://gerrit.ovirt.org/#/c/87428/2
For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/87944/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
This fails due to multiple reason. http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/ failed on
*22:23:15* E OSError: [Errno 24] Too many open files
which seems to be an infra issue on the slave / dirty slave
Just FYI, check-merged jobs has nothing to do with OST/CQ results, they are running functional tests written by VDSM developers.
http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/
Host lago-upgrade-from-release-suite-master-host0 is in non responsive state
The host has vdsm failures with:
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50) 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148) 2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222) 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service (check:104) 2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149) 2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52) 2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run serve_clients(log) File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients cif = clientIF.getInstance(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance cls._instance = clientIF(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__ self.mom = MomClient(config.get("mom", "socket_path")) File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__ raise MomNotAvailableError() MomNotAvailableError
and failure on mom side:
2018-02-22 16:34:00,168 - mom - INFO - MOM starting 2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting 2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk 2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance return JsonRpcVdsmBulkInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__ super(JsonRpcVdsmBulkInterface, self).__init__() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__ .orRaise(RuntimeError, 'No connection to VDSM.') File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise raise exception(*args, **kwargs) RuntimeError: No connection to VDSM.
vdsm upgrade log shows:
MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1 MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
Involved versions:
vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64
mom-0.5.12-0.0.master.el7.centos.noarch
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (3)
-
Eyal Edri -
oVirt Jenkins -
Sandro Bonazzola