8:29 p.m.
------=_Part_45_1872337951.1519352989993
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build Number: 47
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #47
[Your Name] networking: Introducing mac pools and overlap range usage tests
[Barak Korren] Install/Update mock from global_setup.sh
[Sandro Bonazzola] ovirt-image-uploader: drop master jobs
-----------------
Failed Tests:
-----------------
No tests ran.
------=_Part_45_1872337951.1519352989993--
1:12 a.m.
2018-02-23 3:29 GMT+01:00 <jenkins(a)jenkins.phx.ovirt.org>:
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-
ansible-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-
ansible-suite-master/47/
This fails on:
*03:28:49* [ INFO ] TASK [Wait for the engine to come up on the target VM]
*03:29:11* [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The
conditional check 'health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"'
failed. The error was: error while evaluating conditional
(health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"):
No first item, sequence was empty."}*03:29:11* [ ERROR ] Failed to
execute stage 'Closing up': Failed executing ansible-playbook
Messages also show some errors probably not related to this one in a vdsm
hook:
Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd:
Started oVirt Hosted Engine High Availability Monitoring Agent.
Feb 22 21:28:48 lago-he-basic-ansible-suite-master-host0 systemd:
Starting oVirt Hosted Engine High Availability Monitoring Agent...
Feb 22 21:28:49 lago-he-basic-ansible-suite-master-host0 python:
ansible-command Invoked with warn=True executable=None
_uses_shell=False _raw_params=hosted-engine --vm-status --json
removes=None creates=None chdir=None stdin=None
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 python:
detected unhandled Python exception in
'/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server:
Duplicate: core backtrace
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server:
DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:28:50 lago-he-basic-ansible-suite-master-host0 abrt-server:
Deleting problem directory Python-2018-02-22-21:28:50-8927 (dup of
Python-2018-02-22-21:22:17-5163)
Feb 22 21:28:54 lago-he-basic-ansible-suite-master-host0 python:
ansible-command Invoked with warn=True executable=None
_uses_shell=False _raw_params=hosted-engine --vm-status --json
removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python:
ansible-command Invoked with warn=True executable=None
_uses_shell=False _raw_params=hosted-engine --vm-status --json
removes=None creates=None chdir=None stdin=None
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 python:
detected unhandled Python exception in
'/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:00 lago-he-basic-ansible-suite-master-host0 abrt-server:
Not saving repeating crash in
'/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:05 lago-he-basic-ansible-suite-master-host0 python:
ansible-command Invoked with warn=True executable=None
_uses_shell=False _raw_params=hosted-engine --vm-status --json
removes=None creates=None chdir=None stdin=None
Feb 22 21:29:08 lago-he-basic-ansible-suite-master-host0 vdsm[3767]:
WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation
action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at
0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at
0x3cc90d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py",
line 785, in __bootstrap#012 self.__bootstrap_inner()#012File:
"/usr/lib64/python2.7/threading.py", line 812, in
__bootstrap_inner#012 self.run()#012File:
"/usr/lib64/python2.7/threading.py", line 765, in run#012
self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
194, in run#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
_run#012 self._execute_task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task#012 task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 232, in
__call__#012 self._func()#012File:
"/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in
__call__#012 stats = hostapi.get_stats(self._cif,
self._samples.stats())#012File:
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in
get_stats#012 ret['haStats'] = _getHaInfo()#012File:
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in
_getHaInfo#012 stats = instance.get_all_stats()#012File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 92, in get_all_stats#012 stats =
broker.get_stats_from_storage()#012File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 135, in get_stats_from_storage#012 result =
self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py",
line 1233, in __call__#012 return self.__send(self.__name,
args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in
__request#012 verbose=self.__verbose#012File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return
self.single_request(host, handler, request_body, verbose)#012File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012
response = h.getresponse(buffering=True)#012File:
"/usr/lib64/python2.7/httplib.py", line 1089, in getresponse#012
response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444,
in begin#012 version, status, reason = self._read_status()#012File:
"/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012 line
= self.fp.readline(_MAXLINE + 1)#012File:
"/usr/lib64/python2.7/socket.py", line 476, in readline#012 data =
self._sock.recv(self._rbufsize)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal:
ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config
ERROR Unable to identify the OVF_STORE volume, falling back to initial
vm.conf. Please ensure you already added your first data domain for
regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python:
detected unhandled Python exception in
'/usr/libexec/vdsm/hooks/openstacknet-get-config'
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal:
ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config
ERROR Unable to identify the OVF_STORE volume, falling back to initial
vm.conf. Please ensure you already added your first data domain for
regular VMs
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal:
ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno
111] Connection refused#012Traceback (most recent call last):#012
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012
File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012
(code, msg) = self.connect(host, port)#012 File
"/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File
"/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket#012
return socket.create_connection((host, port), timeout)#012 File
"/usr/lib64/python2.7/socket.py", line 571, in create_connection#012
raise err#012error: [Errno 111] Connection refused
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server:
Duplicate: core backtrace
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server:
DUP_OF_DIR: /var/tmp/abrt/Python-2018-02-22-21:22:17-5163
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 abrt-server:
Deleting problem directory Python-2018-02-22-21:29:10-9067 (dup of
Python-2018-02-22-21:22:17-5163)
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 python:
ansible-command Invoked with warn=True executable=None
_uses_shell=False _raw_params=hosted-engine --vm-status --json
removes=None creates=None chdir=None stdin=None
Feb 22 21:29:10 lago-he-basic-ansible-suite-master-host0 journal:
ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config
ERROR Unable to identify the OVF_STORE volume, falling back to initial
vm.conf. Please ensure you already added your first data domain for
regular VMs
Feb 22 21:29:14 lago-he-basic-ansible-suite-master-host0 python:
ansible-setup Invoked with filter=* gather_subset=['all']
fact_path=/etc/ansible/facts.d gather_timeout=10
Feb 22 21:29:15 lago-he-basic-ansible-suite-master-host0 python:
ansible-file Invoked with directory_mode=None force=False
remote_src=None path=/var/tmp/localvmK67L7V owner=None follow=False
group=None unsafe_writes=None state=absent
content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None
selevel=None original_basename=None regexp=None validate=None src=None
seuser=None recurse=False delimiter=None mode=None attributes=None
backup=None
Which should correspond to vdsm log:
2018-02-22 21:29:00,767-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer]
RPC call Host.getCapabilities succeeded in 1.06 seconds (__init__:311)
2018-02-22 21:29:08,114-0500 WARN (vdsm.Scheduler) [Executor] Worker
blocked: <Worker name=periodic/1 running <Task <Operation
action=<vdsm.virt.sampling.HostMonitor object at 0x3cdf490> at
0x3cdf4d0> timeout=15, duration=15 at 0x3cc9410> task#=84 at
0x3cc90d0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py",
line 194, in run
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task
task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__
self._callable()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line
232, in __call__
self._func()
File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line
578, in __call__
stats = hostapi.get_stats(self._cif, self._samples.stats())
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in
get_stats
ret['haStats'] = _getHaInfo()
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182,
in _getHaInfo
stats = instance.get_all_stats()
File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 92, in get_all_stats
stats = broker.get_stats_from_storage()
File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 135, in get_stats_from_storage
result = self._proxy.get_stats()
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request
response = h.getresponse(buffering=True)
File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File: "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File: "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize) (executor:363)
2018-02-22 21:29:08,378-0500 INFO (jsonrpc/7) [api.host] START
getAllVmStats() from=::1,44366 (api:46)
Is it possible to extract logs from the HE VM, even using libguestfs if
needed, to understand why the engine was not up?
Build Number: 47
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #47
[Your Name] networking: Introducing mac pools and overlap range usage tests
[Barak Korren] Install/Update mock from global_setup.sh
[Sandro Bonazzola] ovirt-image-uploader: drop master jobs
-----------------
Failed Tests:
-----------------
No tests ran.
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
4:26 a.m.
On Fri, Feb 23, 2018 at 8:12 AM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
2018-02-23 3:29 GMT+01:00 <jenkins(a)jenkins.phx.ovirt.org>:
> Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans
> ible-suite-master/
> Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans
> ible-suite-master/47/
This fails on:
*03:28:49* [ INFO ] TASK [Wait for the engine to come up on the target VM]
*03:29:11* [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The
conditional check 'health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"'
failed. The error was: error while evaluating conditional (health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"):
No first item, sequence was empty."}*03:29:11* [ ERROR ] Failed to execute stage
'Closing up': Failed executing ansible-playbook
Is it possible to extract logs from the HE VM, even using libguestfs if
needed, to understand why the engine was not up?
Maybe it's a bit different:
the task didn't failed on timeout waiting for the engine but it failed
parsing the json output of 'hosted-engine --vm-status --json' although it
completed with rc=0.
I think we have to investigate it on that side.
>
> Build Number: 47
> Build Status: Failure
> Triggered By: Started by timer
>
> -------------------------------------
> Changes Since Last Success:
> -------------------------------------
> Changes for Build #47
> [Your Name] networking: Introducing mac pools and overlap range usage
> tests
>
> [Barak Korren] Install/Update mock from global_setup.sh
>
> [Sandro Bonazzola] ovirt-image-uploader: drop master jobs
>
>
>
>
> -----------------
> Failed Tests:
> -----------------
> No tests ran.
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
11:30 a.m.
On Fri, Feb 23, 2018 at 11:26 AM, Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
On Fri, Feb 23, 2018 at 8:12 AM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
>
>
> 2018-02-23 3:29 GMT+01:00 <jenkins(a)jenkins.phx.ovirt.org>:
>
>> Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans
>> ible-suite-master/
>> Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans
>> ible-suite-master/47/
>
>
>
> This fails on:
>
> *03:28:49* [ INFO ] TASK [Wait for the engine to come up on the target VM]
>
> *03:29:11* [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The
conditional check 'health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"'
failed. The error was: error while evaluating conditional (health_result.rc == 0 and
health_result.stdout|from_json|json_query('*.\"engine-status\".\"health\"')|first==\"good\"):
No first item, sequence was empty."}*03:29:11* [ ERROR ] Failed to execute stage
'Closing up': Failed executing ansible-playbook
>
> Is it possible to extract logs from the HE VM, even using libguestfs if
> needed, to understand why the engine was not up?
>
>
Maybe it's a bit different:
the task didn't failed on timeout waiting for the engine but it failed
parsing the json output of 'hosted-engine --vm-status --json' although it
completed with rc=0.
I think we have to investigate it on that side.
I opened a bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1548508
and we have a patch:
git push origin HEAD:refs/for/master/ansible/
>
>
>>
>> Build Number: 47
>> Build Status: Failure
>> Triggered By: Started by timer
>>
>> -------------------------------------
>> Changes Since Last Success:
>> -------------------------------------
>> Changes for Build #47
>> [Your Name] networking: Introducing mac pools and overlap range usage
>> tests
>>
>> [Barak Korren] Install/Update mock from global_setup.sh
>>
>> [Sandro Bonazzola] ovirt-image-uploader: drop master jobs
>>
>>
>>
>>
>> -----------------
>> Failed Tests:
>> -----------------
>> No tests ran.
>
>
>
>
> --
>
> SANDRO BONAZZOLA
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
>
8:46 p.m.
New subject: [oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 48 - Fixed!
------=_Part_55_837297276.1519440407146
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-ma...
Build Number: 48
Build Status: Fixed
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #47
[Your Name] networking: Introducing mac pools and overlap range usage tests
[Barak Korren] Install/Update mock from global_setup.sh
[Sandro Bonazzola] ovirt-image-uploader: drop master jobs
Changes for Build #48
[Your Name] networking: Introducing mac pools and overlap range usage tests
-----------------
Failed Tests:
-----------------
All tests passed
------=_Part_55_837297276.1519440407146--
2466
days inactive
2467
days old
4 comments
3 participants
participants (3)
-
jenkins＠jenkins.phx.ovirt.org -
Sandro Bonazzola -
Simone Tiraboschi