- Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 1661
by jenkins＠jenkins.phx.ovirt.org 28 Aug '21

28 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1661//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 788 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '21

27 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/78… Build Number: 788 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #788 [parthdhanjal] Updating max click version [Galit Rosenthal] Fix distro, not replaced correctly ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 005_network_by_label.assign_hosts_network_label Error Message: Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409. -------------------- >> begin captured logging << -------------------- lago.utils: DEBUG: Error while running thread Thread-2 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/005_network_by_label.py", line 73, in _assign_host_network_label host_nic=nic File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 17369, in add return self._internal_add(label, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/005_network_by_label.py", line 81, in assign_hosts_network_label nt.assert_true(all(vt.join_all())) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/005_network_by_label.py", line 73, in _assign_host_network_label host_nic=nic File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 17369, in add return self._internal_add(label, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error 'Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409.\n-------------------- >> begin captured logging << --------------------\nlago.utils: DEBUG: Error while running thread Thread-2\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/005_network_by_label.py", line 73, in _assign_host_network_label\n host_nic=nic\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 17369, in add\n return self._internal_add(label, headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\nError: Fault reason is "Operation Failed". Fault detail is "[Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.]". HTTP response code is 409.\n--------------------- >> end captured logging << ---------------------'

1 1

oVirt infra daily report - unstable production jobs - 1660
by jenkins＠jenkins.phx.ovirt.org 27 Aug '21

27 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1660//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 116324, 2 (jenkins) failed "dummy-master" system tests, but isn't the failure root cause
by oVirt Jenkins 26 Aug '21

26 Aug '21

A system test invoked by the "dummy-master" change queue including change 116324,2 (jenkins) failed. However, this change seems not to be the root cause for this failure. Change 116298,4 (jenkins) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 116298,4 (jenkins) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/jenkins/+/#/c/116324/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/jenkins/+/#/c/116298/4 For failed test results see: https://jenkins.ovirt.org/job/dummy-master_change-queue-tester/680/

1 0

[CQ]: 115919,3 (otopi) failed "ovirt-4.4" system tests
by oVirt Jenkins 26 Aug '21

26 Aug '21

Change 115919,3 (otopi) is probably the reason behind recent system test failures in the "ovirt-4.4" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/c/otopi/+/#/c/115919/3 For failed test results see: https://jenkins.ovirt.org/job/ovirt-4.4_change-queue-tester/478/

1 0

[CQ]: 116286,5 (otopi) failed "ovirt-master" system tests
by oVirt Jenkins 26 Aug '21

26 Aug '21

Change 116286,5 (otopi) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/c/otopi/+/#/c/116286/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/31436/

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1446 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '21

26 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1446 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1446 [Marcin Sobczyk] backend: Add 'ip_mapping' interface [Sandro Bonazzola] otopi: drop 4.2 jobs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_live_incremental_backup_vm2 Error Message: AssertionError: False != True after 600 seconds Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7fdf191b7898> get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7fdf12e866a8> @order_by(_TEST_LIST) def test_live_incremental_backup_vm2(engine_api, get_vm_service_for_vm): vm2_service = _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.UP) vm2_backups_service = vm2_service.backups_service() backup.perform_incremental_vm_backup( > engine_api, vm2_backups_service, DISK2_NAME, "live_vm_backup") basic-suite-master/test-scenarios/test_004_basic_sanity.py:1099: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/storage_utils/backup.py:111: in perform_incremental_vm_backup correlation_id="incremental_" + correlation_id, ost_utils/ost_utils/storage_utils/backup.py:50: in perform_vm_backup allowed_exceptions=[sdk4.NotFoundError], ost_utils/ost_utils/assertions.py:96: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) ost_utils/ost_utils/assertions.py:81: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function perform_vm_backup.<locals>.<lambda> at 0x7fdf12e5c598> value = True, timeout = 600 allowed_exceptions = [<class 'ovirtsdk4.NotFoundError'>], initial_wait = 0 error_message = 'False != True after 600 seconds' def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10, error_message=None ): allowed_exceptions = allowed_exceptions or [] res = '<no-result-obtained>' with _EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: if error_message is None: error_message = '%s != %s after %s seconds' % (res, value, timeout) > raise AssertionError(error_message) E AssertionError: False != True after 600 seconds ost_utils/ost_utils/assertions.py:59: AssertionError

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 784 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '21

26 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/78… Build Number: 784 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #784 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_basic_ui_sanity.start_grid Error Message: could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local creating docker network for grid executing shell: docker network create grid 6f339f87c546555b00e8e4bd037812a59154bac509359f09e362e7464c00f2ac executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 6f339f87c546 grid bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally /usr/bin/docker-current: Error response from daemon: Get https://registry-1.docker.io/v2/selenium/hub/manifests/3.9.1-actinium: Get https://auth.docker.io/token?scope=repository%3Aselenium%2Fhub%3Apull&servi…: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ---------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 189, in start_grid raise RuntimeError("could not get ip address of selenium hub. See previous messages for probable docker failure") could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local creating docker network for grid executing shell: docker network create grid 6f339f87c546555b00e8e4bd037812a59154bac509359f09e362e7464c00f2ac executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 2ac00d18a97f bridge bridge local 6f339f87c546 grid bridge local 3b50f24d63e1 host host local 88b48acf0a2b none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally /usr/bin/docker-current: Error response from daemon: Get https://registry-1.docker.io/v2/selenium/hub/manifests/3.9.1-actinium: Get https://auth.docker.io/token?scope=repository%3Aselenium%2Fhub%3Apull&servi…: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ----------------------

1 1

oVirt infra daily report - unstable production jobs - 1659
by jenkins＠jenkins.phx.ovirt.org 26 Aug '21

26 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1659//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 116318, 2 (jenkins) failed "dummy-master" system tests, but isn't the failure root cause
by oVirt Jenkins 25 Aug '21

25 Aug '21

A system test invoked by the "dummy-master" change queue including change 116318,2 (jenkins) failed. However, this change seems not to be the root cause for this failure. Change 116298,4 (jenkins) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 116298,4 (jenkins) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/jenkins/+/#/c/116318/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/jenkins/+/#/c/116298/4 For failed test results see: https://jenkins.ovirt.org/job/dummy-master_change-queue-tester/678/

1 0