- Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 1649
by jenkins＠jenkins.phx.ovirt.org 16 Aug '21

16 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1649//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1648
by jenkins＠jenkins.phx.ovirt.org 15 Aug '21

15 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1648//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 1407 - Failure!
by jenkins＠jenkins.phx.ovirt.org 14 Aug '21

14 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 1407 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1407 [Marcin Sobczyk] deploy: Fix wildcard pattern for HE suite ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_100_basic_ui_sanity.test_secure_connection_should_fail_without_root_ca[firefox] Error Message: failed on setup with "ost_utils.shell.ShellError: Command failed with rc=125. Stdout: Stderr: Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0"" Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.200.2' selenium_artifacts_dir = '/home/jenkins/workspace/ovirt-system-tests_basic-suite-master_nightly/ovirt-system-tests/exported-artifacts/ui_tests_artifacts/' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip, selenium_artifacts_dir): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman" or backend == "podman-remote": videos_artifacts_dir = selenium_artifacts_dir with podman.grid( engine_fqdn, engine_ip, podman_cmd=backend, > videos_artifacts_dir=videos_artifacts_dir, ) as hub_url: ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:75: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:248: in grid videos_artifacts_dir, /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:195: in _grid with _pod(hub_port, podman_cmd) as pod_name: /usr/lib64/python3.6/contextlib.py:81: in __enter__ return next(self.gen) ost_utils/ost_utils/selenium/grid/podman.py:70: in _pod f"{hub_port}:{hub_port}", _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['podman-remote', 'pod', 'create', '--network=slirp4netns', '-p', '4444:4444'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f7b0e125208>, out = '' err = 'Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0"\n' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ost_utils.shell.ShellError: Command failed with rc=125. Stdout: E E Stderr: E Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.1.0" server "3.0.0" ost_utils/ost_utils/shell.py:49: ShellError

1 4

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 748 - Failure!
by jenkins＠jenkins.phx.ovirt.org 14 Aug '21

14 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/74… Build Number: 748 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #748 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 1647
by jenkins＠jenkins.phx.ovirt.org 14 Aug '21

14 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1647//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host
by Nir Soffer (oVirt JIRA) 14 Aug '21

14 Aug '21

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3114?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-3114: ----------------------------------- On Thu, Aug 12, 2021 at 11:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > We see unexplained timeouts in vdsm tests in the last 2 days. > > This code that should finish in no time times out after 30 seconds: > > op = qemuimg.rebase( > vol.volumePath, > "wrong", > format=qemuimg.FORMAT.QCOW2, > backingFormat=qemuimg.FORMAT.RAW, > unsafe=True) > op.run() > > See > https://gerrit.ovirt.org/c/vdsm/+/116185 > > I'm disabling this test for now, but I need to understand what's going > on in the CI. I think the only way to understand this issue is to ssh to > a slave and debug this code manually in the same environment used > by the tests. We skip 10 tests now: https://gerrit.ovirt.org/c/vdsm/+/116189 This should be reverted when the root cause is fixed. > Unexplained timeouts in CI, need to investigate on a host > --------------------------------------------------------- > > Key: OVIRT-3114 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > We see unexplained timeouts in vdsm tests in the last 2 days. > This code that should finish in no time times out after 30 seconds: > op = qemuimg.rebase( > vol.volumePath, > "wrong", > format=qemuimg.FORMAT.QCOW2, > backingFormat=qemuimg.FORMAT.RAW, > unsafe=True) > op.run() > See > https://gerrit.ovirt.org/c/vdsm/+/116185 > I'm disabling this test for now, but I need to understand what's going > on in the CI. I think the only way to understand this issue is to ssh to > a slave and debug this code manually in the same environment used > by the tests. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0

[JIRA] (OVIRT-3115) Inconsistent Jenkins node time zones - use UTC?
by Nir Soffer (oVirt JIRA) 13 Aug '21

13 Aug '21

Nir Soffer created OVIRT-3115: --------------------------------- Summary: Inconsistent Jenkins node time zones - use UTC? Key: OVIRT-3115 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3115 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We found a broken test that fail when running on EDT timezone: https://gerrit.ovirt.org/c/vdsm/+/116187 However this test fails randomly. Looking at the run logs, we see that some nodes are using UTC: [2021-08-13T15:04:26.855Z] ## Fri Aug 13 15:04:26 UTC 2021 Running env: el8stream:centos-stream-8-x86_64 But others use EDT: [2021-08-13T16:32:26.866Z] ## Fri Aug 13 12:32:26 PM EDT 2021 Running env: el8stream:centos-stream-8-x86_64 Run logs: - https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… - https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… Having different time zones on the same system wastes time for everyone. Please use the same timezone on all systems. I think using UTC will make everyone life easier since developers and admins are located on multiple timezones and everyone know what is UTC. Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 745 - Failure!
by jenkins＠jenkins.phx.ovirt.org 13 Aug '21

13 Aug '21

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/74… Build Number: 745 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #745 [parthdhanjal] Updating max click version ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 1

oVirt infra daily report - unstable production jobs - 1646
by jenkins＠jenkins.phx.ovirt.org 13 Aug '21

13 Aug '21

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1646//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host
by Nir Soffer (oVirt JIRA) 12 Aug '21

12 Aug '21

Nir Soffer created OVIRT-3114: --------------------------------- Summary: Unexplained timeouts in CI, need to investigate on a host Key: OVIRT-3114 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We see unexplained timeouts in vdsm tests in the last 2 days. This code that should finish in no time times out after 30 seconds: op = qemuimg.rebase( vol.volumePath, "wrong", format=qemuimg.FORMAT.QCOW2, backingFormat=qemuimg.FORMAT.RAW, unsafe=True) op.run() See https://gerrit.ovirt.org/c/vdsm/+/116185 I'm disabling this test for now, but I need to understand what's going on in the CI. I think the only way to understand this issue is to ssh to a slave and debug this code manually in the same environment used by the tests. Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100172)

1 0