March 2021 - Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_el7-nightly - Build # 298 - Failure!
by jenkins＠jenkins.phx.ovirt.org 19 Mar '21

19 Mar '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build Number: 298 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #298 [Martin Perina] network: Readd network-suite-4.3 from check-patch ----------------- Failed Tests: ----------------- No tests ran.

1 2

oVirt infra daily report - unstable production jobs - 1499
by jenkins＠jenkins.phx.ovirt.org 19 Mar '21

19 Mar '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1499//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 113938, 3 (ovirt-system-tests) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 18 Mar '21

18 Mar '21

A system test invoked by the "ovirt-master" change queue including change 113938,3 (ovirt-system-tests) failed. However, this change seems not to be the root cause for this failure. Change 113144,5 (ovirt-system-tests) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 113144,5 (ovirt-system-tests) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/ovirt-system-tests/+/#/c/113938/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/ovirt-system-tests/+/#/c/113144/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/28698/

1 0

ovirt-appliance CI
by Yedidyah Bar David 18 Mar '21

18 Mar '21

Hi all, We now merged a patch [1] that changes the CI configuration of ovirt-appliance. In particular, it replaces check-patch and build-artifacts files under automation/ with new sub-stages. It worked well when I did 'ci build', before merging, but now, after merging, I ran [2], and it failed, saying: 12:07:07 WARN: Unable to find sh file automation/build-artifacts.sh or automation/build-artifacts.sh.el8, skipping sh I suppose this is due to its definition in jenkins [3], but not sure how to fix. Perhaps best replace this with an entry in standard-poll-stage-pipelines.yaml - e.g. [4] (and probably remove [3])? Something else? If [4], not sure what the implications are elsewhere. Thanks and best regards, [1] https://gerrit.ovirt.org/c/ovirt-appliance/+/113555 [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x8… [3] https://github.com/oVirt/jenkins/blob/master/jobs/confs/projects/ovirt-appl… [4] https://gerrit.ovirt.org/c/jenkins/+/113912 -- Didi

3 6

basic suite fails on test_metrics_and_log_collector
by Yedidyah Bar David 18 Mar '21

18 Mar '21

Hi all, This started a few days ago [1] and randomly happens since then: E DEBUG: Configuration: E DEBUG: command: collect E DEBUG: Traceback (most recent call last): E DEBUG: File "/usr/lib/python3.6/site-packages/ovirt_log_collector/__main__.py", line 2067, in <module> E DEBUG: '%s directory is not empty.' % (conf["local_tmp_dir"]) E DEBUG: Exception: /dev/shm/log directory is not empty.ERROR: /dev/shm/log directory is not empty.non-zero return code Michal tried to fix this by using a random directory but it still fails [2]: DEBUG: command: collect DEBUG: Traceback (most recent call last): DEBUG: File "/usr/lib/python3.6/site-packages/ovirt_log_collector/__main__.py", line 2067, in <module> DEBUG: '%s directory is not empty.' % (conf["local_tmp_dir"]) DEBUG: Exception: /dev/shm/kaN7uY directory is not empty.ERROR: /dev/shm/kaN7uY directory is not empty.non-zero return code Since I suppose that the randomness of mktemp is good enough, it must be something else. Also, the last successful run before [1] used the same OST git commit (same code), so I do not think it's something in OST's code. Any idea? I think I'll push a patch to create and use the directory right before calling ovirt-log-collector, which is probably better in other ways. Best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… -- Didi

4 11

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 968 - Failure!
by jenkins＠jenkins.phx.ovirt.org 18 Mar '21

18 Mar '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 968 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #968 [Yedidyah Bar David] automation: Add ovirt-master-snapshot ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_003_00_metrics_bootstrap.test_metrics_and_log_collector Error Message: ost_utils.ansible.module_mappers.AnsibleExecutionError: Error running ansible: rc=2, stdout=/usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) lago-basic-suite-master-engine | FAILED | rc=1 >> This command will collect system configuration and diagnostic information from this system. The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. Use the -h option to see usage. DEBUG: Configuration: DEBUG: command: collect DEBUG: Traceback (most recent call last): DEBUG: File "/usr/lib/python3.6/site-packages/ovirt_log_collector/__main__.py", line 2067, in <module> DEBUG: '%s directory is not empty.' % (conf["local_tmp_dir"]) DEBUG: Exception: /dev/shm/DOUDGa directory is not empty.ERROR: /dev/shm/DOUDGa directory is not empty.non-zero return code Stack Trace: ansible_engine = <ost_utils.ansible.module_mappers.ModuleMapper object at 0x7f4fbde6e4a8> ansible_hosts = <ost_utils.ansible.module_mappers.ModuleMapper object at 0x7f4fbde6e390> def test_metrics_and_log_collector(ansible_engine, ansible_hosts): vt = utils.VectorThread( [ functools.partial(configure_metrics, ansible_engine, ansible_hosts), functools.partial(run_log_collector, ansible_engine), ], ) vt.start_all() > vt.join_all() basic-suite-master/test-scenarios/test_003_00_metrics_bootstrap.py:96: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/utils.py:102: in join_all raise exc_info[1].with_traceback(exc_info[2]) ost_utils/ost_utils/utils.py:63: in _ret_via_queue queue.put({'return': func()}) basic-suite-master/test-scenarios/test_003_00_metrics_bootstrap.py:44: in configure_metrics ansible_engine.copy(src=configyml, dest='/etc/ovirt-engine-metrics/') ost_utils/ost_utils/ansible/module_mappers.py:139: in __call__ return _run_ansible_runner(self.config_builder) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ config_builder = <ost_utils.ansible.config_builder.ConfigBuilder object at 0x7f4fbde6e358> def _run_ansible_runner(config_builder): runner = ansible_runner.Runner(config=config_builder.prepare()) LOGGER.debug(f'_run_ansible_runner: before run: {runner}') runner.run() LOGGER.debug(f'_run_ansible_runner: after run: {runner}') if runner.status != 'successful': raise AnsibleExecutionError( rc=runner.rc, > stdout=runner.stdout.read() ) E ost_utils.ansible.module_mappers.AnsibleExecutionError: Error running ansible: rc=2, stdout=/usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! E RequestsDependencyWarning) E lago-basic-suite-master-engine | FAILED | rc=1 >> E This command will collect system configuration and diagnostic E information from this system. E The generated archive may contain data considered sensitive and its E content should be reviewed by the originating organization before E being passed to any third party. E No changes will be made to system configuration. E Use the -h option to see usage. E DEBUG: Configuration: E DEBUG: command: collect E DEBUG: Traceback (most recent call last): E DEBUG: File "/usr/lib/python3.6/site-packages/ovirt_log_collector/__main__.py", line 2067, in <module> E DEBUG: '%s directory is not empty.' % (conf["local_tmp_dir"]) E DEBUG: Exception: /dev/shm/DOUDGa directory is not empty.ERROR: /dev/shm/DOUDGa directory is not empty.non-zero return code ost_utils/ost_utils/ansible/module_mappers.py:90: AnsibleExecutionError

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_el7-nightly - Build # 295 - Failure!
by jenkins＠jenkins.phx.ovirt.org 18 Mar '21

18 Mar '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build Number: 295 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #295 [Martin Perina] network: Readd network-suite-4.3 from check-patch ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 953 - Failure!
by jenkins＠jenkins.phx.ovirt.org 18 Mar '21

18 Mar '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 953 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #953 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_preview_snapshot_with_memory Error Message: AssertionError: False != True after 600 seconds Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f817f2df668> @order_by(_TEST_LIST) def test_preview_snapshot_with_memory(engine_api): engine = engine_api.system_service() events = engine.events_service() assertions.assert_true_within_long( # wait for event 68 == USER_CREATE_SNAPSHOT_FINISHED_SUCCESS > lambda: any(e.code == 68 for e in events.list(max=6)) ) basic-suite-master/test-scenarios/test_004_basic_sanity.py:718: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/assertions.py:96: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) ost_utils/ost_utils/assertions.py:81: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function test_preview_snapshot_with_memory.<locals>.<lambda> at 0x7f817f2fc6a8> value = True, timeout = 600, allowed_exceptions = [], initial_wait = 0 error_message = 'False != True after 600 seconds' def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10, error_message=None ): allowed_exceptions = allowed_exceptions or [] res = '<no-result-obtained>' with _EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: if error_message is None: error_message = '%s != %s after %s seconds' % (res, value, timeout) > raise AssertionError(error_message) E AssertionError: False != True after 600 seconds ost_utils/ost_utils/assertions.py:59: AssertionError

1 14

oVirt infra daily report - unstable production jobs - 1498
by jenkins＠jenkins.phx.ovirt.org 18 Mar '21

18 Mar '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1498//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 113905, 5 (ovirt-system-tests) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 17 Mar '21

17 Mar '21

A system test invoked by the "ovirt-master" change queue including change 113905,5 (ovirt-system-tests) failed. However, this change seems not to be the root cause for this failure. Change 113144,5 (ovirt-system-tests) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 113144,5 (ovirt-system-tests) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/ovirt-system-tests/+/#/c/113905/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/ovirt-system-tests/+/#/c/113144/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/28692/

1 0