2018-02-23 3:29 GMT+01:00 <jenkins@jenkins.phx.ovirt.org>:

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic- ansible-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic- ansible-suite-master/47/

This fails on: *03:28:49* [ INFO ] TASK [Wait for the engine to come up on the target VM] *03:29:11* [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"' failed. The error was: error while evaluating conditional (health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"): No first item, sequence was empty."}*03:29:11* [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook Messages also show some errors probably not related to this one in a vdsm hook: Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Starting oVirt Hosted Engine High Availability Monitoring Agent... Feb 22 21:28:49 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config' Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163 Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:28:50-8927 (dup of Python-2018-02-22-21:22:17-5163) Feb 22 21:28:54 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config' Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 abrt-server: Not saving repeating crash in '/usr/libexec/vdsm/hooks/openstacknet-get-config' Feb 22 21:29:05 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None Feb 22 21:29:08 lago-he-basic-ansible-suite-master-host0 vdsm[3767]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__#012 self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__#012 stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012 ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012 stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats#012 stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012 result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request#012 verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012 response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse#012 response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 data = self._sock.recv(self._rbufsize) Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config' Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 111] Connection refused#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, port)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/socket.py", line 571, in create_connection#012 raise err#012error: [Errno 111] Connection refused Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163 Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:29:10-9067 (dup of Python-2018-02-22-21:22:17-5163) Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs Feb 22 21:29:14 lago-he-basic-ansible-suite-master-host0 python: ansible-setup Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10 Feb 22 21:29:15 lago-he-basic-ansible-suite-master-host0 python: ansible-file Invoked with directory_mode=None force=False remote_src=None path=/var/tmp/localvmK67L7V owner=None follow=False group=None unsafe_writes=None state=absent content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=None attributes=None backup=None Which should correspond to vdsm log: 2018-02-22 21:29:00,767-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 1.06 seconds (__init__:311) 2018-02-22 21:29:08,114-0500 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__ self._func() File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__ stats = hostapi.get_stats(self._cif, self._samples.stats()) File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats ret['haStats'] = _getHaInfo() File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo stats = instance.get_all_stats() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats stats = broker.get_stats_from_storage() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage result = self._proxy.get_stats() File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request response = h.getresponse(buffering=True) File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse response.begin() File: "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File: "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) (executor:363) 2018-02-22 21:29:08,378-0500 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,44366 (api:46) Is it possible to extract logs from the HE VM, even using libguestfs if needed, to understand why the engine was not up?

Build Number: 47 Build Status: Failure Triggered By: Started by timer

------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #47 [Your Name] networking: Introducing mac pools and overlap range usage tests

[Barak Korren] Install/Update mock from global_setup.sh

[Sandro Bonazzola] ovirt-image-uploader: drop master jobs

----------------- Failed Tests: ----------------- No tests ran.

-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>