3:42 p.m.
A system test invoked by the "ovirt-master" change queue including change
87428,2 (vdsm) failed. However, this change seems not to be the root cause for
this failure. Change 87944,3 (vdsm) that this change depends on or is based on,
was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from this
change will not be released until either change 87944,3 (vdsm) is fixed and
this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.
For further details about the change see:
https://gerrit.ovirt.org/#/c/87428/2
For further details about the change that seems to be the root cause behind the
testing failures see:
https://gerrit.ovirt.org/#/c/87944/3
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
6:15 a.m.
2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins(a)ovirt.org>:
A system test invoked by the "ovirt-master" change queue
including change
87428,2 (vdsm) failed. However, this change seems not to be the root cause
for
this failure. Change 87944,3 (vdsm) that this change depends on or is
based on,
was detected as the cause of the testing failures.
This change had been removed from the testing queue. Artifacts built from
this
change will not be released until either change 87944,3 (vdsm) is fixed and
this change is updated to refer to or rebased on the fixed version, or this
change is modified to no longer depend on it.
For further details about the change see:
https://gerrit.ovirt.org/#/c/87428/2
For further details about the change that seems to be the root cause
behind the
testing failures see:
https://gerrit.ovirt.org/#/c/87944/3
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
This fails due to multiple reason.
http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/
failed on
*22:23:15* E OSError: [Errno 24] Too many open files
which seems to be an infra issue on the slave / dirty slave
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/testRe...
Host lago-upgrade-from-release-suite-master-host0 is in non responsive state
The host has vdsm failures with:
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC
interface is disabled (momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init
clientIF, shutting down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START
prepareForShutdown(options=None) from=internal,
task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor]
Shutting down domain monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check]
Stopping check service (check:104)
2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent]
Stopping <EventLoop running=True closed=False at 0x38225104>
(asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev]
Stopping multipath event listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH
prepareForShutdown return=None from=internal,
task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised
(vdsmd:158)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
serve_clients(log)
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in
serve_clients
cif = clientIF.getInstance(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251,
in getInstance
cls._instance = clientIF(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121,
in __init__
self.mom = MomClient(config.get("mom", "socket_path"))
File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
raise MomNotAvailableError()
MomNotAvailableError
and failure on mom side:
2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to
VDSM! [Errno 111] Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
hypervisor_iface = self.get_hypervisor_interface()
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217,
in get_hypervisor_interface
return module.instance(self.config)
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 47, in instance
return JsonRpcVdsmBulkInterface()
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 29, in __init__
super(JsonRpcVdsmBulkInterface, self).__init__()
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py",
line 41, in __init__
.orRaise(RuntimeError, 'No connection to VDSM.')
File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.
vdsm upgrade log shows:
MainThread::DEBUG::2018-02-22
16:31:34,113::libvirtconnection::167::root::(get) trying to connect
libvirt
MainThread::DEBUG::2018-02-22
16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json -disable usb
-disable pcmcia -disable isapnp -disable ide -disable scsi -disable
dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = '';
<rc> = 0
MainThread::DEBUG::2018-02-22
16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd
None)
MainThread::DEBUG::2018-02-22
16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = '';
<rc> = 0
MainThread::DEBUG::2018-02-22
16:31:34,273::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED: <err> = '';
<rc>
= 1
MainThread::INFO::2018-02-22
16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing
netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-22
16:31:34,287::fileutils::51::root::(rm_file) File:
/var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-22
16:31:34,287::fileutils::96::root::(rm_tree) Directory:
/var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22
16:31:34,287::netconfpersistence::150::root::(save) Saved new config
RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and
/var/lib/vdsm/staging/netconf/bonds
MainThread::DEBUG::2018-02-22
16:31:34,288::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22
16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED: <err> = '';
<rc>
= 1
Involved versions:
vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64
mom-0.5.12-0.0.master.el7.centos.noarch
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
6:23 a.m.
On Tue, Feb 27, 2018 at 2:15 PM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenkins(a)ovirt.org>:
> A system test invoked by the "ovirt-master" change queue including change
> 87428,2 (vdsm) failed. However, this change seems not to be the root
> cause for
> this failure. Change 87944,3 (vdsm) that this change depends on or is
> based on,
> was detected as the cause of the testing failures.
>
> This change had been removed from the testing queue. Artifacts built from
> this
> change will not be released until either change 87944,3 (vdsm) is fixed
> and
> this change is updated to refer to or rebased on the fixed version, or
> this
> change is modified to no longer depend on it.
>
> For further details about the change see:
> https://gerrit.ovirt.org/#/c/87428/2
>
> For further details about the change that seems to be the root cause
> behind the
> testing failures see:
> https://gerrit.ovirt.org/#/c/87944/3
>
> For failed test results see:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/
This fails due to multiple reason.
http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/
failed on
*22:23:15* E OSError: [Errno 24] Too many open files
which seems to be an infra issue on the slave / dirty slave
Just FYI, check-merged jobs has nothing to do with OST/CQ results, they are
running functional tests written by VDSM developers.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/
Host lago-upgrade-from-release-suite-master-host0 is in non responsive state
The host has vdsm failures with:
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is disabled
(momIF:50)
2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init clientIF, shutting
down storage dispatcher (clientIF:148)
2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START
prepareForShutdown(options=None) from=internal,
task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting down domain
monitors (monitor:222)
2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping check service
(check:104)
2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping
<EventLoop running=True closed=False at 0x38225104> (asyncevent:220)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping multipath event
listener (udev:149)
2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown
return=None from=internal, task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52)
2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised (vdsmd:158)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run
serve_clients(log)
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in
serve_clients
cif = clientIF.getInstance(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in
getInstance
cls._instance = clientIF(irs, log, scheduler)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in
__init__
self.mom = MomClient(config.get("mom", "socket_path"))
File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__
raise MomNotAvailableError()
MomNotAvailableError
and failure on mom side:
2018-02-22 16:34:00,168 - mom - INFO - MOM starting
2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting
2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk
2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! [Errno 111]
Connection refused
2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run
hypervisor_iface = self.get_hypervisor_interface()
File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in
get_hypervisor_interface
return module.instance(self.config)
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 47, in instance
return JsonRpcVdsmBulkInterface()
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
line 29, in __init__
super(JsonRpcVdsmBulkInterface, self).__init__()
File
"/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py",
line 41, in __init__
.orRaise(RuntimeError, 'No connection to VDSM.')
File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise
raise exception(*args, **kwargs)
RuntimeError: No connection to VDSM.
vdsm upgrade log shows:
MainThread::DEBUG::2018-02-22 16:31:34,113::libvirtconnection::167::root::(get) trying to
connect libvirt
MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) lshw -json
-disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi
-disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) SUCCESS:
<err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) /sbin/tc
qdisc show (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) SUCCESS:
<err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) FAILED:
<err> = ''; <rc> = 1
MainThread::INFO::2018-02-22 16:31:34,286::netconfpersistence::198::root::(_clearDisk)
Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) File:
/var/lib/vdsm/staging/netconf already removed
MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) Directory:
/var/lib/vdsm/staging/netconf already removed
MainThread::INFO::2018-02-22 16:31:34,287::netconfpersistence::150::root::(save) Saved
new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and
/var/lib/vdsm/staging/netconf/bonds
MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd)
/usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) FAILED:
<err> = ''; <rc> = 1
Involved versions:
vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64
mom-0.5.12-0.0.master.el7.centos.noarch
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
2249
days inactive
2254
days old
2 comments
3 participants
participants (3)
-
Eyal Edri -
oVirt Jenkins -
Sandro Bonazzola