2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:

A system test invoked by the "ovirt-master" change queue including change
87428,2 (vdsm) failed. However, this change seems not to be the root cause for
this failure. Change 87944,3 (vdsm) that this change depends on or is based on,
was detected as the cause of the testing failures.

This change had been removed from the testing queue. Artifacts built from this
change will not be released until either change 87944,3 (vdsm) is fixed and
this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.

For further details about the change see:
https://gerrit.ovirt.org/#/c/87428/2

For further details about the change that seems to be the root cause behind the
testing failures see:
https://gerrit.ovirt.org/#/c/87944/3

For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/

This fails due to multiple reason.

http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/ failed on

22:23:15 E       OSError: [Errno 24] Too many open files

which seems to be an infra issue on the slave / dirty slave

http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/

Host lago-upgrade-from-release-suite-master-host0 is in non responsive state

The host has vdsm failures with:

2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled (momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO  (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO  (MainThread) [storage.check] Stopping check service (check:104)
2018-02-22 16:34:04,221-0500 INFO  (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x38225104> (asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO  (MainThread) [storage.udev] Stopping multipath event listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO  (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
    serve_clients(log)
  File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients
    cif = clientIF.getInstance(irs, log, scheduler)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in getInstance
    cls._instance = clientIF(irs, log, scheduler)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in __init__
    self.mom = MomClient(config.get("mom", "socket_path"))
  File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
    raise MomNotAvailableError()
MomNotAvailableError

and failure on mom side:

2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
    hypervisor_iface = self.get_hypervisor_interface()
  File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface
    return module.instance(self.config)
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance
    return JsonRpcVdsmBulkInterface()
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__
    super(JsonRpcVdsmBulkInterface, self).__init__()
  File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 41, in __init__
    .orRaise(RuntimeError, 'No connection to VDSM.')
  File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
    raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.

vdsm upgrade log shows:

MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File: /var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1

Involved versions:

vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64

mom-0.5.12-0.0.master.el7.centos.noarch

 

_______________________________________________
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

--

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA

TRIED. TESTED. TRUSTED.

_______________________________________________
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra