2018-02-23 3:29 GMT+01:00 <jenkins@jenkins.phx.ovirt.org>:
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/47/


This fails on:
03:28:49 [ INFO  ] TASK [Wait for the engine to come up on the target VM]
03:29:11 [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"' failed. The error was: error while evaluating conditional (health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"): No first item, sequence was empty."}
03:29:11 [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook

Messages also show some errors probably not related to this one in a vdsm hook:

Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent.
Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Starting oVirt Hosted Engine High Availability Monitoring Agent...
Feb 22 21:28:49 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:28:50-8927 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:28:54 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 abrt-server: Not saving repeating crash in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:05 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:08 lago-he-basic-ansible-suite-master-host0 vdsm[3767]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012  self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012  self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012  self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012  ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012  self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012  task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012  self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__#012  self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__#012  stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012  ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012  stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats#012  stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012  result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012  return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request#012  verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012  return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012  response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse#012  response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012  version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012  line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012  data = self._sock.recv(self._rbufsize)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 111] Connection refused#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012    timeout=float(cfg["smtp-timeout"]))#012  File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012    (code, msg) = self.connect(host, port)#012  File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012    self.sock = self._get_socket(host, port, self.timeout)#012  File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket#012    return socket.create_connection((host, port), timeout)#012  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection#012    raise err#012error: [Errno 111] Connection refused
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:29:10-9067 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:14 lago-he-basic-ansible-suite-master-host0 python: ansible-setup Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10
Feb 22 21:29:15 lago-he-basic-ansible-suite-master-host0 python: ansible-file Invoked with directory_mode=None force=False remote_src=None path=/var/tmp/localvmK67L7V owner=None follow=False group=None unsafe_writes=None state=absent content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=None attributes=None backup=None
Which should correspond to vdsm log:

2018-02-22 21:29:00,767-0500 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 1.06 seconds (__init__:311)
2018-02-22 21:29:08,114-0500 WARN  (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
  self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
  self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run
  ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
  self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task
  task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
  self._callable()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__
  self._func()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__
  stats = hostapi.get_stats(self._cif, self._samples.stats())
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats
  ret['haStats'] = _getHaInfo()
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo
  stats = instance.get_all_stats()
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats
  stats = broker.get_stats_from_storage()
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage
  result = self._proxy.get_stats()
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
  return self.__send(self.__name, args)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
  verbose=self.__verbose
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
  return self.single_request(host, handler, request_body, verbose)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request
  response = h.getresponse(buffering=True)
File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
  response.begin()
File: "/usr/lib64/python2.7/httplib.py", line 444, in begin
  version, status, reason = self._read_status()
File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
  line = self.fp.readline(_MAXLINE + 1)
File: "/usr/lib64/python2.7/socket.py", line 476, in readline
  data = self._sock.recv(self._rbufsize) (executor:363)
2018-02-22 21:29:08,378-0500 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::1,44366 (api:46)


Is it possible to extract logs from the HE VM, even using libguestfs if needed, to understand why the engine was not up?

 

Build Number: 47
Build Status:  Failure
Triggered By: Started by timer

-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #47
[Your Name] networking: Introducing mac pools and overlap range usage tests

[Barak Korren] Install/Update mock from global_setup.sh

[Sandro Bonazzola] ovirt-image-uploader: drop master jobs




-----------------
Failed Tests:
-----------------
No tests ran.



--

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA