August 2021 - Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 1653
by jenkins＠jenkins.phx.ovirt.org 19 Aug '21

19 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1653//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 765 - Failure!
by jenkins＠jenkins.phx.ovirt.org 19 Aug '21

19 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/76… Build Number: 765 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #765 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_el7-nightly - Build # 638 - Failure!
by jenkins＠jenkins.phx.ovirt.org 19 Aug '21

19 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_el7-nightl… Build Number: 638 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #638 [parthdhanjal] Updating max click version [Sandro Bonazzola] ovirt-release: drop 4.2 and 4.3 builds ----------------- Failed Tests: ----------------- No tests ran.

2 123

oVirt infra daily report - unstable production jobs - 1652
by jenkins＠jenkins.phx.ovirt.org 18 Aug '21

18 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1652//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 116247, 1 (ovirt-node-ng-image) failed "ovirt-4.4" system tests, but isn't the failure root cause
by oVirt Jenkins 18 Aug '21

18 Aug '21

A system test invoked by the "ovirt-4.4" change queue including change 116247,1 (ovirt-node-ng-image) failed. However, this change seems not to be the root cause for this failure. Change 116009,1 (ovirt-node-ng-image) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 116009,1 (ovirt-node-ng-image) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/c/ovirt-node-ng-image/+/#/c/116247/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/c/ovirt-node-ng-image/+/#/c/116009/1 For failed test results see: https://jenkins.ovirt.org/job/ovirt-4.4_change-queue-tester/446/

1 0

[JIRA] (OVIRT-3117) dummy-master_change-queue fails on ModuleNotFoundError
by Sandro Bonazzola (oVirt JIRA) 18 Aug '21

18 Aug '21

Sandro Bonazzola created OVIRT-3117: --------------------------------------- Summary: dummy-master_change-queue fails on ModuleNotFoundError Key: OVIRT-3117 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3117 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Sandro Bonazzola Assignee: infra We have an issue in "jenkins" project job: https://jenkins-staging.ovirt.org/blue/organizations/jenkins/dummy-master_c… [2021-08-18T16:08:40.570Z] Traceback (most recent call last): [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue(a)tmp/durable-1cddc895/script.sh", line 7, in <module> [2021-08-18T16:08:40.570Z] with JenkinsChangeQueue.persist_in_artifacts() as cq: [2021-08-18T16:08:40.570Z] File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ [2021-08-18T16:08:40.570Z] return next(self.gen) [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue/jenkins/stdci_libs/jenkins_objects.py", line 315, in persist_in_artifacts [2021-08-18T16:08:40.570Z] obj = cls.load_from_artifact(artifact_file) [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue/jenkins/stdci_libs/jenkins_objects.py", line 296, in load_from_artifact [2021-08-18T16:08:40.570Z] artifact_file, cls if fallback_to_new else None [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue/jenkins/stdci_libs/jenkins_objects.py", line 270, in object_from_artifact [2021-08-18T16:08:40.570Z] return _cpickle_loads(fd.read()) [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue/jenkins/stdci_libs/jenkins_objects.py", line 223, in _cpickle_loads [2021-08-18T16:08:40.570Z] return pk.load() [2021-08-18T16:08:40.570Z] File "/root/workspace/dummy-master_change-queue/jenkins/stdci_libs/jenkins_objects.py", line 207, in _find_global [2021-08-18T16:08:40.570Z] mod = import_module(mod_name) [2021-08-18T16:08:40.570Z] File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module [2021-08-18T16:08:40.570Z] return _bootstrap._gcd_import(name[level:], package, level) [2021-08-18T16:08:40.570Z] File "<frozen importlib._bootstrap>", line 994, in _gcd_import [2021-08-18T16:08:40.570Z] File "<frozen importlib._bootstrap>", line 971, in _find_and_load [2021-08-18T16:08:40.570Z] File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked [2021-08-18T16:08:40.570Z] ModuleNotFoundError: No module named 'copy_reg' script returned exit code 1 Can you please have a look? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.* -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100174)

1 0

oVirt infra daily report - unstable production jobs - 1651
by jenkins＠jenkins.phx.ovirt.org 17 Aug '21

17 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1651//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3116) Fail fast on unsupported configuration
by Nir Soffer (oVirt JIRA) 17 Aug '21

17 Aug '21

Nir Soffer created OVIRT-3116: --------------------------------- Summary: Fail fast on unsupported configuration Key: OVIRT-3116 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3116 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra When using: host-distro: el8 The CI ignores the unknown configuration "el8" runs the job anywhere. Then the job fails after many minutes, wasting resources and developers time running on el7, where the tests cannot pass. Lets fix to fail fast on the first unknown configuration, don't run anything if the configuration is invalid. Example builds: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29292/ https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29293/ https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29295/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 759 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Aug '21

17 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/75… Build Number: 759 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #759 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1419 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Aug '21

17 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1419 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1419 [Eitan Raviv] network: set default ip_version if unknown ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] cleanup-gated-ovirt-4.3 - Build #593 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-4.3/593/ Build Name: #593 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 13

[oVirt Jenkins] cleanup-gated-dummy-master - Build #600 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Build: https://jenkins.ovirt.org/job/cleanup-gated-dummy-master/600/ Build Name: #600 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 5

oVirt infra daily report - unstable production jobs - 1650
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1650//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 755 - Failure!
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/75… Build Number: 755 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #755 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1416 - Failure!
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1416 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1416 [Eitan Raviv] network: test port isolated both directions ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_100_basic_ui_sanity.test_secure_connection_should_fail_without_root_ca[firefox] Error Message: failed on setup with "ost_utils.shell.ShellError: Command failed with rc=125. Stdout: Stderr: Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.3/libpod/_ping": dial unix ///run/podman/podman.sock: connect: connection refused" Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.200.2' selenium_artifacts_dir = '/home/jenkins/workspace/ovirt-system-tests_basic-suite-master_nightly/ovirt-system-tests/exported-artifacts/ui_tests_artifacts/' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip, selenium_artifacts_dir): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman" or backend == "podman-remote": videos_artifacts_dir = selenium_artifacts_dir with podman.grid( engine_fqdn, engine_ip, podman_cmd=backend, > videos_artifacts_dir=videos_artifacts_dir, ) as hub_url: ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:75: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:248: in grid videos_artifacts_dir, /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:195: in _grid with _pod(hub_port, podman_cmd) as pod_name: /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:70: in _pod f"{hub_port}:{hub_port}", _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['podman-remote', 'pod', 'create', '--network=slirp4netns', '-p', '4446:4446'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f8157109160>, out = '' err = 'Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.3/libpod/_ping": dial unix ///run/podman/podman.sock: connect: connection refused\n' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ost_utils.shell.ShellError: Command failed with rc=125. Stdout: E E Stderr: E Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.3/libpod/_ping": dial unix ///run/podman/podman.sock: connect: connection refused ost_utils/ost_utils/shell.py:49: ShellError

1 1

oVirt infra daily report - unstable production jobs - 1649
by jenkins＠jenkins.phx.ovirt.org 15 Aug '21

15 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1649//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1648
by jenkins＠jenkins.phx.ovirt.org 14 Aug '21

14 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1648//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1407 - Failure!
by jenkins＠jenkins.phx.ovirt.org 14 Aug '21

14 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1407 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1407 [Marcin Sobczyk] deploy: Fix wildcard pattern for HE suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_100_basic_ui_sanity.test_secure_connection_should_fail_without_root_ca[firefox] Error Message: failed on setup with "ost_utils.shell.ShellError: Command failed with rc=125. Stdout: Stderr: Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0"" Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.200.2' selenium_artifacts_dir = '/home/jenkins/workspace/ovirt-system-tests_basic-suite-master_nightly/ovirt-system-tests/exported-artifacts/ui_tests_artifacts/' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip, selenium_artifacts_dir): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman" or backend == "podman-remote": videos_artifacts_dir = selenium_artifacts_dir with podman.grid( engine_fqdn, engine_ip, podman_cmd=backend, > videos_artifacts_dir=videos_artifacts_dir, ) as hub_url: ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:75: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:248: in grid videos_artifacts_dir, /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:195: in _grid with _pod(hub_port, podman_cmd) as pod_name: /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:70: in _pod f"{hub_port}:{hub_port}", _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['podman-remote', 'pod', 'create', '--network=slirp4netns', '-p', '4444:4444'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f7b0e125208>, out = '' err = 'Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0"\n' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ost_utils.shell.ShellError: Command failed with rc=125. Stdout: E E Stderr: E Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0" ost_utils/ost_utils/shell.py:49: ShellError

1 4

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 748 - Failure!
by jenkins＠jenkins.phx.ovirt.org 13 Aug '21

13 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/74… Build Number: 748 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #748 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 1647
by jenkins＠jenkins.phx.ovirt.org 13 Aug '21

13 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1647//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host
by Nir Soffer (oVirt JIRA) 13 Aug '21

13 Aug '21

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3114?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-3114: ----------------------------------- On Thu, Aug 12, 2021 at 11:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > We see unexplained timeouts in vdsm tests in the last 2 days. > > This code that should finish in no time times out after 30 seconds: > > op = qemuimg.rebase( > vol.volumePath, > "wrong", > format=qemuimg.FORMAT.QCOW2, > backingFormat=qemuimg.FORMAT.RAW, > unsafe=True) > op.run() > > See > https://gerrit.ovirt.org/c/vdsm/+/116185 > > I'm disabling this test for now, but I need to understand what's going > on in the CI. I think the only way to understand this issue is to ssh to > a slave and debug this code manually in the same environment used > by the tests. We skip 10 tests now: https://gerrit.ovirt.org/c/vdsm/+/116189 This should be reverted when the root cause is fixed. > Unexplained timeouts in CI, need to investigate on a host > --------------------------------------------------------- > > Key: OVIRT-3114 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > We see unexplained timeouts in vdsm tests in the last 2 days. > This code that should finish in no time times out after 30 seconds: > op = qemuimg.rebase( > vol.volumePath, > "wrong", > format=qemuimg.FORMAT.QCOW2, > backingFormat=qemuimg.FORMAT.RAW, > unsafe=True) > op.run() > See > https://gerrit.ovirt.org/c/vdsm/+/116185 > I'm disabling this test for now, but I need to understand what's going > on in the CI. I think the only way to understand this issue is to ssh to > a slave and debug this code manually in the same environment used > by the tests. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0

[JIRA] (OVIRT-3115) Inconsistent Jenkins node time zones - use UTC?
by Nir Soffer (oVirt JIRA) 13 Aug '21

13 Aug '21

Nir Soffer created OVIRT-3115: --------------------------------- Summary: Inconsistent Jenkins node time zones - use UTC? Key: OVIRT-3115 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3115 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We found a broken test that fail when running on EDT timezone: https://gerrit.ovirt.org/c/vdsm/+/116187 However this test fails randomly. Looking at the run logs, we see that some nodes are using UTC: [2021-08-13T15:04:26.855Z] ## Fri Aug 13 15:04:26 UTC 2021 Running env: el8stream:centos-stream-8-x86_64 But others use EDT: [2021-08-13T16:32:26.866Z] ## Fri Aug 13 12:32:26 PM EDT 2021 Running env: el8stream:centos-stream-8-x86_64 Run logs: - https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… - https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… Having different time zones on the same system wastes time for everyone. Please use the same timezone on all systems. I think using UTC will make everyone life easier since developers and admins are located on multiple timezones and everyone know what is UTC. Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 745 - Failure!
by jenkins＠jenkins.phx.ovirt.org 12 Aug '21

12 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/74… Build Number: 745 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #745 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

oVirt infra daily report - unstable production jobs - 1646
by jenkins＠jenkins.phx.ovirt.org 12 Aug '21

12 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1646//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host
by Nir Soffer (oVirt JIRA) 12 Aug '21

12 Aug '21

Nir Soffer created OVIRT-3114: --------------------------------- Summary: Unexplained timeouts in CI, need to investigate on a host Key: OVIRT-3114 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We see unexplained timeouts in vdsm tests in the last 2 days. This code that should finish in no time times out after 30 seconds: op = qemuimg.rebase( vol.volumePath, "wrong", format=qemuimg.FORMAT.QCOW2, backingFormat=qemuimg.FORMAT.RAW, unsafe=True) op.run() See https://gerrit.ovirt.org/c/vdsm/+/116185 I'm disabling this test for now, but I need to understand what's going on in the CI. I think the only way to understand this issue is to ssh to a slave and debug this code manually in the same environment used by the tests. Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0