9:33 a.m.
On Sun, Nov 8, 2020 at 6:08 AM <jenkins(a)jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
Build Number: 561
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #561
[Marcin Sobczyk] basic: Report start times and runnings time of tests
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_run_vms
Error Message:
Error: Fault reason is "Operation Failed". Fault detail is "[Network error
during communication with the Host.]". HTTP response code is 400.
06:05:28 ../basic-suite-master/test-scenarios/004_basic_sanity.py::test_run_vms
started at 04:05:28 FAILED (30s)
06:06:00
06:06:00 =================================== FAILURES
===================================
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
2020-11-07 23:05:57,825-05 ERROR
[org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-2)
[06bc4dc1-f61e-4004-8623-8e91e2e2db24] Command
'org.ovirt.engine.core.bll.RunVmOnceCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Connection timeout for host
'lago-basic-suite-master-host-1', last response arrived 23158 ms ago.
(Failed with error VDS_NETWORK_ERROR and code 5022)
Logs on host-1 seem ok to me, but finish a bit before this, so perhaps
indeed we lost communication with it (temporarily?):
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
Last line is:
2020-11-07 23:05:04,434-0500 DEBUG (jsonrpc/6) [jsonrpc.JsonRpcServer]
Return 'Host.getStats' in bridge with (suppressed) (__init__:360)
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
Last line is:
Nov 7 23:04:10 lago-basic-suite-master-host-1 systemd[1]: Started
Session c177 of user root.
This run (561) isn't the last one - there were a few successful runs
after it, and the following (and last) failed one, 565, is due to
virt-sparsify (see other thread).
But this also happened to me now:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te...
Not copying/pasting again - it looks very similar. engine.log has same
error, vdsm.log looks ok but finishes a bit earlier.
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te...
2020-11-08 20:51:57,693::log_utils.py::__exit__::611::lago.prefix::INFO::
# Deploy environment: [32mSuccess [0m (in 0:14:33)
2020-11-08 20:51:57,957::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@
Deploy oVirt environment: [32mSuccess [0m (in 0:14:33)
2020-11-08 21:32:41,201::log_utils.py::__enter__::600::lago.prefix::INFO::@
Collect artifacts: [0m [0m
2020-11-08 21:32:41,295::log_utils.py::__enter__::600::lago.prefix::INFO::
# [Thread-1] lago-basic-suite-master-host-0: [0m [0m
2020-11-08 21:32:41,296::log_utils.py::__enter__::600::lago.prefix::INFO::
# [Thread-2] lago-basic-suite-master-host-1: [0m [0m
2020-11-08 21:32:41,297::log_utils.py::__enter__::600::lago.prefix::INFO::
# [Thread-3] lago-basic-suite-master-engine: [0m [0m
2020-11-08
21:32:41,298::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0:
failed extracting files: Unable to extract paths from
lago-basic-suite-master-host-0: unreachable with SSH
2020-11-08
21:32:41,299::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0:
attempting to extract files with libguestfs
2020-11-08
21:32:41,507::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1:
failed extracting files: Unable to extract paths from
lago-basic-suite-master-host-1: unreachable with SSH
2020-11-08
21:32:41,508::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1:
attempting to extract files with libguestfs
So it seems to me like an infra issue - also lago failed to ssh there
(and also to host-0). Can anyone have a look please? Thanks.
Best regards,
Stack Trace:
assert_vm_is_alive = <function is_alive at 0x7fc7d597b488>
engine_api = <ovirtsdk4.Connection object at 0x7fc7d5979a50>
management_gw_ip = '192.168.201.1'
@order_by(_TEST_LIST)
def test_run_vms(assert_vm_is_alive, engine_api, management_gw_ip):
engine = engine_api.system_service()
vm_params = types.Vm(
initialization=types.Initialization(
user_name=VM_USER_NAME,
root_password=VM_PASSWORD
)
)
vm_params.initialization.host_name = BACKUP_VM_NAME
backup_vm_service = test_utils.get_vm_service(engine, BACKUP_VM_NAME)
backup_vm_service.start(use_cloud_init=True, vm=vm_params)
vm_params.initialization.host_name = VM2_NAME
vm2_service = test_utils.get_vm_service(engine, VM2_NAME)
vm2_service.start(use_cloud_init=True, vm=vm_params)
# CirrOS cloud-init is different, networking doesn't work since it
doesn't support the format oVirt is using
vm_params.initialization.host_name = VM0_NAME # hostname seems to work, the
others not
vm_params.initialization.dns_search = 'lago.local'
vm_params.initialization.domain = 'lago.local'
vm_params.initialization.dns_servers = management_gw_ip
vm0_service = test_utils.get_vm_service(engine, VM0_NAME)
> vm0_service.start(use_cloud_init=True, vm=vm_params)
../basic-suite-master/test-scenarios/004_basic_sanity.py:754:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:32520: in start
return self._internal_action(action, 'start', None, headers, query, wait)
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in _internal_action
return future.wait() if wait else future
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait
return self._code(response)
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback
self._check_fault(response)
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault
self._raise_error(response, body.fault)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
response = <ovirtsdk4.http.Response object at 0x7fc7d6aefad0>
detail = <ovirtsdk4.types.Fault object at 0x7fc7d6aefb50>
@staticmethod
def _raise_error(response, detail=None):
"""
Creates and raises an error containing the details of the given HTTP
response and fault.
This method is intended for internal use by other components of the
SDK. Refrain from using it directly, as backwards compatibility isn't
guaranteed.
"""
fault = detail if isinstance(detail, types.Fault) else None
msg = ''
if fault:
if fault.reason:
if msg:
msg += ' '
msg = msg + 'Fault reason is "%s".' % fault.reason
if fault.detail:
if msg:
msg += ' '
msg = msg + 'Fault detail is "%s".' % fault.detail
if response:
if response.code:
if msg:
msg += ' '
msg = msg + 'HTTP response code is %s.' % response.code
if response.message:
if msg:
msg += ' '
msg = msg + 'HTTP response message is "%s".' %
response.message
if isinstance(detail, six.string_types):
if msg:
msg += ' '
msg = msg + detail + '.'
class_ = Error
if response is not None:
if response.code in [401, 403]:
class_ = AuthError
elif response.code == 404:
class_ = NotFoundError
error = class_(msg)
error.code = response.code if response else None
error.fault = fault
> raise error
E Error: Fault reason is "Operation Failed". Fault detail is
"[Network error during communication with the Host.]". HTTP response code is
400.
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118:
Error_______________________________________________
Infra mailing list -- infra(a)ovirt.org
To unsubscribe send an email to infra-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/D5SHAOBCVBE...
--
Didi