- Infra - oVirt List Archives

[CQ]: 112068,8 (ovirt-system-tests) failed "ovirt-4.3" system tests
by oVirt Jenkins 11 Nov '20

11 Nov '20

Change 112068,8 (ovirt-system-tests) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/112068/8 For failed test results see: https://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/4063/

1 0

[oVirt Jenkins] deploy-to-gated-ovirt-4.3 - Build #604 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 11 Nov '20

11 Nov '20

Build: https://jenkins.ovirt.org/job/deploy-to-gated-ovirt-4.3/604/ Build Name: #604 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 565 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Nov '20

11 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 565 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #565 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_sparsify_disk1 Error Message: AssertionError: False != True after 600 seconds Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7fb484cae950> @order_by(_TEST_LIST) def test_sparsify_disk1(engine_api): engine = engine_api.system_service() disk_service = test_utils.get_disk_service(engine, DISK1_NAME) with engine_utils.wait_for_event(engine, 1325): # USER_SPARSIFY_IMAGE_START event disk_service.sparsify() with engine_utils.wait_for_event(engine, 1326): # USER_SPARSIFY_IMAGE_FINISH_SUCCESS > pass ../basic-suite-master/test-scenarios/004_basic_sanity.py:297: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/contextlib.py:24: in __exit__ self.gen.next() ../ost_utils/ost_utils/engine_utils.py:44: in wait_for_event lambda: ../ost_utils/ost_utils/assertions.py:97: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) ../ost_utils/ost_utils/assertions.py:82: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7fb484c808c0>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 error_message = 'False != True after 600 seconds' def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10, error_message=None ): allowed_exceptions = allowed_exceptions or [] with _EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: if error_message is None: error_message = '%s != %s after %s seconds' % (res, value, timeout) > raise AssertionError(error_message) E AssertionError: False != True after 600 seconds ../ost_utils/ost_utils/assertions.py:60: AssertionError

1 6

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 546 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Nov '20

11 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/54… Build Number: 546 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #546 [Nir Soffer] nfs: Use async mount ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly - Build # 502 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '20

10 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build Number: 502 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #502 [Nir Soffer] nfs: Use async mount ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 501 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Nov '20

10 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 501 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #501 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 2

[JIRA] (OVIRT-3054) [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources)
by Alertmanager_Bot (oVirt JIRA) 10 Nov '20

10 Nov '20

Alertmanager_Bot created OVIRT-3054: --------------------------------------- Summary: [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources) Key: OVIRT-3054 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3054 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = LowDiskSpace_NodeExporter_v1_0 - device = /dev/mapper/resources_lvm-data - fstype = ext4 - instance = resources.ovirt.org:9100 - job = ovirt-resources - mountpoint = /srv/resources Annotations: - description = resources.ovirt.org:9100 of job ovirt-resources has less than 10% free disk space Source: http://prometheus-0:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100150)

1 0

oVirt infra daily report - unstable production jobs - 1371
by jenkins＠jenkins.phx.ovirt.org 10 Nov '20

10 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1371//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 499 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 499 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #499 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly - Build # 498 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build Number: 498 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #498 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

oVirt infra daily report - unstable production jobs - 1370
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1370//artifact/exported… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Failed to run project_setup.sh for: #24836 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0144.workers-phx.ovirt.org.

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Failed to run project_setup.sh for: #24834 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0141.workers-phx.ovirt.org.

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Failed to run project_setup.sh for: #24826 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0141.workers-phx.ovirt.org.

1 0

[JIRA] (OVIRT-3053) [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources)
by Shlomi Zidmi (oVirt JIRA) 09 Nov '20

09 Nov '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3053?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-3053: -------------------------------- Resolution: Fixed Status: Done (was: To Do) DIsk now has a sufficient amount of free space > [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources) > --------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3053 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3053 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="LowDiskSpace_NodeExporter_v1_0",instance="resources.ovirt.org:9100",job="ovirt-resources"} > > Labels: > - alertname = LowDiskSpace_NodeExporter_v1_0 > - device = /dev/mapper/resources_lvm-data > - fstype = ext4 > - instance = resources.ovirt.org:9100 > - job = ovirt-resources > - mountpoint = /srv/resources > Annotations: > - description = resources.ovirt.org:9100 of job ovirt-resources has less than 10% free disk space > Source: http://prometheus-0:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100149)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 561 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 561 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #561 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_run_vms Error Message: Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: assert_vm_is_alive = <function is_alive at 0x7fc7d597b488> engine_api = <ovirtsdk4.Connection object at 0x7fc7d5979a50> management_gw_ip = '192.168.201.1' @order_by(_TEST_LIST) def test_run_vms(assert_vm_is_alive, engine_api, management_gw_ip): engine = engine_api.system_service() vm_params = types.Vm( initialization=types.Initialization( user_name=VM_USER_NAME, root_password=VM_PASSWORD ) ) vm_params.initialization.host_name = BACKUP_VM_NAME backup_vm_service = test_utils.get_vm_service(engine, BACKUP_VM_NAME) backup_vm_service.start(use_cloud_init=True, vm=vm_params) vm_params.initialization.host_name = VM2_NAME vm2_service = test_utils.get_vm_service(engine, VM2_NAME) vm2_service.start(use_cloud_init=True, vm=vm_params) # CirrOS cloud-init is different, networking doesn't work since it doesn't support the format oVirt is using vm_params.initialization.host_name = VM0_NAME # hostname seems to work, the others not vm_params.initialization.dns_search = 'lago.local' vm_params.initialization.domain = 'lago.local' vm_params.initialization.dns_servers = management_gw_ip vm0_service = test_utils.get_vm_service(engine, VM0_NAME) > vm0_service.start(use_cloud_init=True, vm=vm_params) ../basic-suite-master/test-scenarios/004_basic_sanity.py:754: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:32520: in start return self._internal_action(action, 'start', None, headers, query, wait) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in _internal_action return future.wait() if wait else future /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback self._check_fault(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault self._raise_error(response, body.fault) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <ovirtsdk4.http.Response object at 0x7fc7d6aefad0> detail = <ovirtsdk4.types.Fault object at 0x7fc7d6aefb50> @staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault. This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.' class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError error = class_(msg) error.code = response.code if response else None error.fault = fault > raise error E Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118: Error

2 2

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 540 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/54… Build Number: 540 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #540 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 697, in run_vms _verify_vm_state(engine, vm_name, types.VmStatus.UP) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 102, in _verify_vm_state lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 496 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 496 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #496 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 537 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/53… Build Number: 537 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #537 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 2

[oVirt Jenkins] ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly - Build # 495 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build Number: 495 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #495 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

oVirt infra daily report - unstable production jobs - 1369
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1369//artifact/exported… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct standard-manual-runner
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Failed to run project_setup.sh for: #1594 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0141.workers-phx.ovirt.org.

1 0

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 493 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Nov '20

07 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 493 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #493 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

oVirt infra daily report - unstable production jobs - 1368
by jenkins＠jenkins.phx.ovirt.org 07 Nov '20

07 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1368//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 557 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Nov '20

07 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 557 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #557 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.100_basic_ui_sanity.test_login[firefox] Error Message: test setup failure Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.201.4' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman": if os_utils.on_centos(8) and os_utils.inside_mock(): pytest.skip("podman doesn't work in mock") hub_port = network_utils.find_free_port(4444, 4544) with podman.grid(engine_fqdn, engine_ip, hub_port=hub_port) as hub_url: yield hub_url elif backend == "docker": > with docker.grid(engine_fqdn, engine_ip) as hub_url: ../ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:75: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:118: in grid with _network(network_name): /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:61: in _network shell(["docker", "network", "create", network_name]) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'network', 'create', 'grid'], bytes_output = False kwargs = {}, process = <subprocess.Popen object at 0x7fec3c2fed90>, out = '' err = 'Error response from daemon: network with name grid already exists ' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ShellError: Command failed with rc=1. Stdout: E E Stderr: E Error response from daemon: network with name grid already exists ../ost_utils/ost_utils/shell.py:51: ShellError

1 2