- Infra - oVirt List Archives

[CQ]: 115533, 2 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 05 Jul '21

05 Jul '21

A system test invoked by the "ovirt-master" change queue including change 115533,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 115528,6 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 115528,6 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/vdsm/+/#/c/115533/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/vdsm/+/#/c/115528/6 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/30592/

1 0

[CQ]: 115528,6 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 05 Jul '21

05 Jul '21

Change 115528,6 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/c/vdsm/+/#/c/115528/6 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/30590/

1 0

[JIRA] (OVIRT-3107) [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci)
by Alertmanager_Bot (oVirt JIRA) 05 Jul '21

05 Jul '21

Alertmanager_Bot created OVIRT-3107: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci) Key: OVIRT-3107 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3107 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = shift-n11.phx.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = shift-n11.phx.ovirt.org - node_role_kubernetes_io_compute = true - region = primary - type = vm - zone = ci Annotations: - description = shift-n11.phx.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100168)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1291 - Failure!
by jenkins＠jenkins.phx.ovirt.org 05 Jul '21

05 Jul '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1291 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1291 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2 Error Message: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268> @order_by(_TEST_LIST) def test_cold_incremental_backup_vm2(engine_api, get_vm_service_for_vm): _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.DOWN) vm2_backups_service = get_vm_service_for_vm(VM2_NAME).backups_service() backup.perform_incremental_vm_backup( > engine_api, vm2_backups_service, DISK2_NAME, "cold_vm_backup") basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/storage_utils/backup.py:75: in perform_incremental_vm_backup correlation_id="incremental_" + correlation_id) ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup ), query={'correlation_id': correlation_id} /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add return self._internal_add(backup, headers, query, wait) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in _internal_add return future.wait() if wait else future /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback self._check_fault(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in _check_fault self._raise_error(response, body) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60> @staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault. This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.' class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError error = class_(msg) error.code = response.code if response else None error.fault = fault > raise error E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error

1 1

oVirt infra daily report - unstable production jobs - 1607
by jenkins＠jenkins.phx.ovirt.org 04 Jul '21

04 Jul '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1607//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] cleanup-gated-ovirt-master - Build #567 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 04 Jul '21

04 Jul '21

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-master/567/ Build Name: #567 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

oVirt infra daily report - unstable production jobs - 1606
by jenkins＠jenkins.phx.ovirt.org 03 Jul '21

03 Jul '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1606//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1287 - Failure!
by jenkins＠jenkins.phx.ovirt.org 03 Jul '21

03 Jul '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1287 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1287 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_001_initialize_engine.test_initialize_engine Error Message: failed on setup with "ost_utils.ansible.module_mappers.AnsibleExecutionError: Error running ansible: rc=2, stdout=lago-basic-suite-master-engine | FAILED! => { "changed": false, "elapsed": 120, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.2 port 22: No route to host" } lago-basic-suite-master-host-0 | FAILED! => { "changed": false, "elapsed": 120, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.3 port 22: No route to host" } lago-basic-suite-master-host-1 | FAILED! => { "changed": false, "elapsed": 120, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.4 port 22: No route to host" }" Stack Trace: ansible_vms_to_deploy = <ost_utils.ansible.module_mappers.ModuleMapper object at 0x7f2fb1959278> ansible_hosts = <ost_utils.ansible.module_mappers.ModuleMapper object at 0x7f2fb1959358> deploy_scripts = {'lago-basic-suite-master-engine': ['$OST_REPO_ROOT/common/deploy-scripts/setup_sar_stat.sh', '$OST_REPO_ROOT/common/d...t-1': ['$OST_REPO_ROOT/common/deploy-scripts/setup_sar_stat.sh', '$OST_REPO_ROOT/common/deploy-scripts/setup_host.sh']} working_dir = '/home/jenkins/workspace/ovirt-system-tests_basic-suite-master_nightly/ovirt-system-tests/deployment-basic-suite-master' request = <SubRequest 'deploy' for <Function test_initialize_engine>> run_scripts = <function run_scripts.<locals>.do_run_scripts at 0x7f2fb191a6a8> @pytest.fixture(scope="session", autouse=True) def deploy( ansible_vms_to_deploy, ansible_hosts, deploy_scripts, working_dir, request, run_scripts, ): if deployment_utils.is_deployed(working_dir): LOGGER.info("Environment already deployed") return LOGGER.info("Waiting for SSH on the VMs") > ansible_vms_to_deploy.wait_for_connection(timeout=120) ost_utils/ost_utils/pytest/fixtures/deployment.py:61: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/ansible/module_mappers.py:110: in __call__ return _run_ansible_runner(self.config_builder) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ config_builder = <ost_utils.ansible.config_builder.ConfigBuilder object at 0x7f2fb19595f8> def _run_ansible_runner(config_builder): runner = ansible_runner.Runner(config=config_builder.prepare()) LOGGER.debug(f'_run_ansible_runner: before run: {runner}') runner.run() LOGGER.debug(f'_run_ansible_runner: after run: {runner}') if runner.status != 'successful': raise AnsibleExecutionError( rc=runner.rc, > stdout=runner.stdout.read() ) E ost_utils.ansible.module_mappers.AnsibleExecutionError: Error running ansible: rc=2, stdout=lago-basic-suite-master-engine | FAILED! => { E "changed": false, E "elapsed": 120, E "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.2 port 22: No route to host" E } E lago-basic-suite-master-host-0 | FAILED! => { E "changed": false, E "elapsed": 120, E "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.3 port 22: No route to host" E } E lago-basic-suite-master-host-1 | FAILED! => { E "changed": false, E "elapsed": 120, E "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: ssh: connect to host 192.168.200.4 port 22: No route to host" E } ost_utils/ost_utils/ansible/module_mappers.py:48: AnsibleExecutionError

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 623 - Failure!
by jenkins＠jenkins.phx.ovirt.org 03 Jul '21

03 Jul '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/62… Build Number: 623 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #623 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.disk_operations Error Message: False != True after 600 seconds -------------------- >> begin captured logging << -------------------- lago.utils: DEBUG: Error while running thread Thread-63 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 932, in disk_operations vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) 'False != True after 600 seconds\n-------------------- >> begin captured logging << --------------------\nlago.utils: DEBUG: Error while running thread Thread-63\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper\n return func(get_test_prefix(), *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper\n prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs\n File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration\n testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id))\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long\n assert_equals_within_long(func, True, allowed_exceptions)\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long\n func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within\n \'%s != %s after %s seconds\' % (res, value, timeout)\nAssertionError: False != True after 600 seconds\n--------------------- >> end captured logging << ---------------------'

1 1

oVirt infra daily report - unstable production jobs - 1605
by jenkins＠jenkins.phx.ovirt.org 02 Jul '21

02 Jul '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1605//artifact/exported… Cheers, Jenkins

1 0