<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2018-02-23 3:29 GMT+01:00  <span dir="ltr">&lt;<a href="mailto:jenkins@jenkins.phx.ovirt.org" target="_blank">jenkins@jenkins.phx.ovirt.org</a>&gt;</span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Project: <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_he-basic-<wbr>ansible-suite-master/</a><br>
Build: <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-master/47/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_he-basic-<wbr>ansible-suite-master/47/</a></blockquote><div><br></div><div><br></div><div>This fails on:</div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><b style="box-sizing:border-box">03:28:49</b> <span style="font-family:arial,sans-serif">[ INFO  ] TASK [Wait for the engine to come up on the target VM]</span><br></pre></pre></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">03:29:11</b> </span>[ ERROR ] fatal: [localhost]: FAILED! =&gt; {&quot;msg&quot;: &quot;The conditional check &#39;health_result.rc == 0 and health_result.stdout|from_json|json_query(&#39;*.\&quot;engine-status\&quot;.\&quot;health\&quot;&#39;)|first==\&quot;good\&quot;&#39; failed. The error was: error while evaluating conditional (health_result.rc == 0 and health_result.stdout|from_json|json_query(&#39;*.\&quot;engine-status\&quot;.\&quot;health\&quot;&#39;)|first==\&quot;good\&quot;): No first item, sequence was empty.&quot;}
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">03:29:11</b> </span>[ ERROR ] Failed to execute stage &#39;Closing up&#39;: Failed executing ansible-playbook</pre></pre><br></div><div>Messages also show some errors probably not related to this one in a vdsm hook:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent.
Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd: Starting oVirt Hosted Engine High Availability Monitoring Agent...
Feb 22 21:28:49 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in &#39;/usr/libexec/vdsm/hooks/openstacknet-get-config&#39;
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:28:50-8927 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:28:54 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in &#39;/usr/libexec/vdsm/hooks/openstacknet-get-config&#39;
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 abrt-server: Not saving repeating crash in &#39;/usr/libexec/vdsm/hooks/openstacknet-get-config&#39;
Feb 22 21:29:05 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:08 lago-he-basic-ansible-suite-master-host0 vdsm[3767]: WARN Worker blocked: &lt;Worker name=periodic/1 running &lt;Task &lt;Operation action=&lt;vdsm.virt.sampling.HostMonitor object at 0x3cdf490&gt; at 0x3cdf4d0&gt; timeout=15, duration=15 at 0x3cc9410&gt; task#=84 at 0x3cc90d0&gt;, traceback:#012File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 785, in __bootstrap#012  self.__bootstrap_inner()#012File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 812, in __bootstrap_inner#012  self.run()#012File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 765, in run#012  self.__target(*self.__args, **self.__kwargs)#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py&quot;, line 194, in run#012  ret = func(*args, **kwargs)#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 301, in _run#012  self._execute_task()#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 315, in _execute_task#012  task()#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 391, in __call__#012  self._callable()#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py&quot;, line 232, in __call__#012  self._func()#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py&quot;, line 578, in __call__#012  stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 77, in get_stats#012  ret[&#39;haStats&#39;] = _getHaInfo()#012File: &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 182, in _getHaInfo#012  stats = instance.get_all_stats()#012File: &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py&quot;, line 92, in get_all_stats#012  stats = broker.get_stats_from_storage()#012File: &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 135, in get_stats_from_storage#012  result = self._proxy.get_stats()#012File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1233, in __call__#012  return self.__send(self.__name, args)#012File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1587, in __request#012  verbose=self.__verbose#012File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1273, in request#012  return self.single_request(host, handler, request_body, verbose)#012File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1303, in single_request#012  response = h.getresponse(buffering=True)#012File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 1089, in getresponse#012  response.begin()#012File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 444, in begin#012  version, status, reason = self._read_status()#012File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 400, in _read_status#012  line = self.fp.readline(_MAXLINE + 1)#012File: &quot;/usr/lib64/python2.7/socket.py&quot;, line 476, in readline#012  data = self._sock.recv(self._rbufsize)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: detected unhandled Python exception in &#39;/usr/libexec/vdsm/hooks/openstacknet-get-config&#39;
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 111] Connection refused#012Traceback (most recent call last):#012  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py&quot;, line 26, in send_email#012    timeout=float(cfg[&quot;smtp-timeout&quot;]))#012  File &quot;/usr/lib64/python2.7/smtplib.py&quot;, line 255, in __init__#012    (code, msg) = self.connect(host, port)#012  File &quot;/usr/lib64/python2.7/smtplib.py&quot;, line 315, in connect#012    self.sock = self._get_socket(host, port, self.timeout)#012  File &quot;/usr/lib64/python2.7/smtplib.py&quot;, line 290, in _get_socket#012    return socket.create_connection((host, port), timeout)#012  File &quot;/usr/lib64/python2.7/socket.py&quot;, line 571, in create_connection#012    raise err#012error: [Errno 111] Connection refused
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Duplicate: core backtrace
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server: Deleting problem directory Python-2018-02-22-21:29:10-9067 (dup of Python-2018-02-22-21:22:17-5163)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=hosted-engine --vm-status --json removes=None creates=None chdir=None stdin=None
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
Feb 22 21:29:14 lago-he-basic-ansible-suite-master-host0 python: ansible-setup Invoked with filter=* gather_subset=[&#39;all&#39;] fact_path=/etc/ansible/facts.d gather_timeout=10
Feb 22 21:29:15 lago-he-basic-ansible-suite-master-host0 python: ansible-file Invoked with directory_mode=None force=False remote_src=None path=/var/tmp/localvmK67L7V owner=None follow=False group=None unsafe_writes=None state=absent content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=None attributes=None backup=None</pre>Which should correspond to vdsm log:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">2018-02-22 21:29:00,767-0500 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 1.06 seconds (__init__:311)
2018-02-22 21:29:08,114-0500 WARN  (vdsm.Scheduler) [Executor] Worker blocked: &lt;Worker name=periodic/1 running &lt;Task &lt;Operation action=&lt;vdsm.virt.sampling.HostMonitor object at 0x3cdf490&gt; at 0x3cdf4d0&gt; timeout=15, duration=15 at 0x3cc9410&gt; task#=84 at 0x3cc90d0&gt;, traceback:
File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 785, in __bootstrap
  self.__bootstrap_inner()
File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 812, in __bootstrap_inner
  self.run()
File: &quot;/usr/lib64/python2.7/threading.py&quot;, line 765, in run
  self.__target(*self.__args, **self.__kwargs)
File: &quot;/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py&quot;, line 194, in run
  ret = func(*args, **kwargs)
File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 301, in _run
  self._execute_task()
File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 315, in _execute_task
  task()
File: &quot;/usr/lib/python2.7/site-packages/vdsm/executor.py&quot;, line 391, in __call__
  self._callable()
File: &quot;/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py&quot;, line 232, in __call__
  self._func()
File: &quot;/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py&quot;, line 578, in __call__
  stats = hostapi.get_stats(self._cif, self._samples.stats())
File: &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 77, in get_stats
  ret[&#39;haStats&#39;] = _getHaInfo()
File: &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 182, in _getHaInfo
  stats = instance.get_all_stats()
File: &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py&quot;, line 92, in get_all_stats
  stats = broker.get_stats_from_storage()
File: &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 135, in get_stats_from_storage
  result = self._proxy.get_stats()
File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1233, in __call__
  return self.__send(self.__name, args)
File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1587, in __request
  verbose=self.__verbose
File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1273, in request
  return self.single_request(host, handler, request_body, verbose)
File: &quot;/usr/lib64/python2.7/xmlrpclib.py&quot;, line 1303, in single_request
  response = h.getresponse(buffering=True)
File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 1089, in getresponse
  response.begin()
File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 444, in begin
  version, status, reason = self._read_status()
File: &quot;/usr/lib64/python2.7/httplib.py&quot;, line 400, in _read_status
  line = self.fp.readline(_MAXLINE + 1)
File: &quot;/usr/lib64/python2.7/socket.py&quot;, line 476, in readline
  data = self._sock.recv(self._rbufsize) (executor:363)
2018-02-22 21:29:08,378-0500 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::1,44366 (api:46)</pre><br></div><div><br></div><div>Is it possible to extract logs from the HE VM, even using libguestfs if needed, to understand why the engine was not up?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Build Number: 47<br>
Build Status:  Failure<br>
Triggered By: Started by timer<br>
<br>
------------------------------<wbr>-------<br>
Changes Since Last Success:<br>
------------------------------<wbr>-------<br>
Changes for Build #47<br>
[Your Name] networking: Introducing mac pools and overlap range usage tests<br>
<br>
[Barak Korren] Install/Update mock from global_setup.sh<br>
<br>
[Sandro Bonazzola] ovirt-image-uploader: drop master jobs<br>
<br>
<br>
<br>
<br>
-----------------<br>
Failed Tests:<br>
-----------------<br>
No tests ran.</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&amp;D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>