<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2018-02-23 3:29 GMT+01:00 <span dir="ltr"><<a href="mailto:jenkins@jenkins.phx.ovirt.org" target="_blank">jenkins@jenkins.phx.ovirt.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Project: <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_he-basic-<wbr>ansible-suite-master/</a><br>
Build: <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/47/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_he-basic-<wbr>ansible-suite-master/47/</a></blockquote><div><br></div><div><br></div><div>This fails on:</div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><b style="box-sizing:border-box">03:28:49</b> <span style="font-family:arial,sans-serif">[ INFO ] TASK [Wait for the engine to come up on the target VM]</span><br></pre></pre></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">03:29:11</b> </span>[ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"' failed. The error was: error while evaluating conditional (health_result.rc == 0 and health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"): No first item, sequence was empty."}
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">03:29:11</b> </span>[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook</pre></pre><br></div><div>Messages also show some errors probably not related to this one in a vdsm hook:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent.
Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Starting oVirt Hosted Engine High Availability Monitoring Agent...
Feb 22 21:28:49 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:28:50-8927 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:28:54 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 abrt-server: Not saving repeating crash in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:05 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:08 lago-he-basic-ansible-suite-master-host0 vdsm[3767]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__#012 self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__#012 stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012 ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012 stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats#012 stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012 result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request#012 verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012 response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse#012 response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 data = self._sock.recv(self._rbufsize)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in '/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 111] Connection refused#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, port)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/socket.py", line 571, in create_connection#012 raise err#012error: [Errno 111] Connection refused
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:29:10-9067 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:14 lago-he-basic-ansible-suite-master-host0 python: ansible-setup Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10
Feb 22 21:29:15 lago-he-basic-ansible-suite-master-host0 python: ansible-file Invoked with directory_mode=None force=False remote_src=None path=/var/tmp/localvmK67L7V owner=None follow=False group=None unsafe_writes=None state=absent content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=None attributes=None backup=None</pre>Which should correspond to vdsm log:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 21:29:00,767-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 1.06 seconds (__init__:311)
2018-02-22 21:29:08,114-0500 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at 0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at 0x3cc90d0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task
task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
self._callable()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in __call__
self._func()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__
stats = hostapi.get_stats(self._cif, self._samples.stats())
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats
ret['haStats'] = _getHaInfo()
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo
stats = instance.get_all_stats()
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 92, in get_all_stats
stats = broker.get_stats_from_storage()
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage
result = self._proxy.get_stats()
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request
response = h.getresponse(buffering=True)
File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File: "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File: "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize) (executor:363)
2018-02-22 21:29:08,378-0500 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,44366 (api:46)</pre><br></div><div><br></div><div>Is it possible to extract logs from the HE VM, even using libguestfs if needed, to understand why the engine was not up?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Build Number: 47<br>
Build Status: Failure<br>
Triggered By: Started by timer<br>
<br>
------------------------------<wbr>-------<br>
Changes Since Last Success:<br>
------------------------------<wbr>-------<br>
Changes for Build #47<br>
[Your Name] networking: Introducing mac pools and overlap range usage tests<br>
<br>
[Barak Korren] Install/Update mock from global_setup.sh<br>
<br>
[Sandro Bonazzola] ovirt-image-uploader: drop master jobs<br>
<br>
<br>
<br>
<br>
-----------------<br>
Failed Tests:<br>
-----------------<br>
No tests ran.</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>