On Sun, Nov 8, 2020 at 6:08 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 561 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #561 [Marcin Sobczyk] basic: Report start times and runnings time of tests
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_run_vms
Error Message: Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.
06:05:28 ../basic-suite-master/test-scenarios/004_basic_sanity.py::test_run_vms started at 04:05:28 FAILED (30s) 06:06:00 06:06:00 =================================== FAILURES =================================== https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... 2020-11-07 23:05:57,825-05 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-2) [06bc4dc1-f61e-4004-8623-8e91e2e2db24] Command 'org.ovirt.engine.core.bll.RunVmOnceCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Connection timeout for host 'lago-basic-suite-master-host-1', last response arrived 23158 ms ago. (Failed with error VDS_NETWORK_ERROR and code 5022) Logs on host-1 seem ok to me, but finish a bit before this, so perhaps indeed we lost communication with it (temporarily?): https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Last line is: 2020-11-07 23:05:04,434-0500 DEBUG (jsonrpc/6) [jsonrpc.JsonRpcServer] Return 'Host.getStats' in bridge with (suppressed) (__init__:360) https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Last line is: Nov 7 23:04:10 lago-basic-suite-master-host-1 systemd[1]: Started Session c177 of user root. This run (561) isn't the last one - there were a few successful runs after it, and the following (and last) failed one, 565, is due to virt-sparsify (see other thread). But this also happened to me now: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... Not copying/pasting again - it looks very similar. engine.log has same error, vdsm.log looks ok but finishes a bit earlier. https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... 2020-11-08 20:51:57,693::log_utils.py::__exit__::611::lago.prefix::INFO:: # Deploy environment: [32mSuccess [0m (in 0:14:33) 2020-11-08 20:51:57,957::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ Deploy oVirt environment: [32mSuccess [0m (in 0:14:33) 2020-11-08 21:32:41,201::log_utils.py::__enter__::600::lago.prefix::INFO::@ Collect artifacts: [0m [0m 2020-11-08 21:32:41,295::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-1] lago-basic-suite-master-host-0: [0m [0m 2020-11-08 21:32:41,296::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-2] lago-basic-suite-master-host-1: [0m [0m 2020-11-08 21:32:41,297::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-3] lago-basic-suite-master-engine: [0m [0m 2020-11-08 21:32:41,298::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0: failed extracting files: Unable to extract paths from lago-basic-suite-master-host-0: unreachable with SSH 2020-11-08 21:32:41,299::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0: attempting to extract files with libguestfs 2020-11-08 21:32:41,507::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1: failed extracting files: Unable to extract paths from lago-basic-suite-master-host-1: unreachable with SSH 2020-11-08 21:32:41,508::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1: attempting to extract files with libguestfs So it seems to me like an infra issue - also lago failed to ssh there (and also to host-0). Can anyone have a look please? Thanks. Best regards,
Stack Trace: assert_vm_is_alive = <function is_alive at 0x7fc7d597b488> engine_api = <ovirtsdk4.Connection object at 0x7fc7d5979a50> management_gw_ip = '192.168.201.1'
@order_by(_TEST_LIST) def test_run_vms(assert_vm_is_alive, engine_api, management_gw_ip): engine = engine_api.system_service()
vm_params = types.Vm( initialization=types.Initialization( user_name=VM_USER_NAME, root_password=VM_PASSWORD ) )
vm_params.initialization.host_name = BACKUP_VM_NAME backup_vm_service = test_utils.get_vm_service(engine, BACKUP_VM_NAME) backup_vm_service.start(use_cloud_init=True, vm=vm_params)
vm_params.initialization.host_name = VM2_NAME vm2_service = test_utils.get_vm_service(engine, VM2_NAME) vm2_service.start(use_cloud_init=True, vm=vm_params)
# CirrOS cloud-init is different, networking doesn't work since it doesn't support the format oVirt is using vm_params.initialization.host_name = VM0_NAME # hostname seems to work, the others not vm_params.initialization.dns_search = 'lago.local' vm_params.initialization.domain = 'lago.local' vm_params.initialization.dns_servers = management_gw_ip vm0_service = test_utils.get_vm_service(engine, VM0_NAME)
vm0_service.start(use_cloud_init=True, vm=vm_params)
../basic-suite-master/test-scenarios/004_basic_sanity.py:754: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:32520: in start return self._internal_action(action, 'start', None, headers, query, wait) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in _internal_action return future.wait() if wait else future /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback self._check_fault(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault self._raise_error(response, body.fault) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
response = <ovirtsdk4.http.Response object at 0x7fc7d6aefad0> detail = <ovirtsdk4.types.Fault object at 0x7fc7d6aefb50>
@staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault.
This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None
msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message
if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.'
class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError
error = class_(msg) error.code = response.code if response else None error.fault = fault
raise error
E Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.
/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118: Error_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/D5SHAOBCVBEPZ5...
-- Didi