- Infra - oVirt List Archives

[CQ]: 105932, 1 (ovirt-system-tests) failed "ovirt-4.3" system tests, but isn't the failure root cause
by oVirt Jenkins 25 Dec '19

25 Dec '19

A system test invoked by the "ovirt-4.3" change queue including change 105932,1 (ovirt-system-tests) failed. However, this change seems not to be the root cause for this failure. Change 105785,1 (ovirt-system-tests) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 105785,1 (ovirt-system-tests) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105932/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/105785/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2720/

1 0

[CQ]: 105932, 1 (ovirt-system-tests) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 25 Dec '19

25 Dec '19

A system test invoked by the "ovirt-master" change queue including change 105932,1 (ovirt-system-tests) failed. However, this change seems not to be the root cause for this failure. Change 105785,1 (ovirt-system-tests) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 105785,1 (ovirt-system-tests) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105932/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/105785/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17834/

1 0

[oVirt Jenkins] cleanup-gated-dummy-master - Build #3 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 25 Dec '19

25 Dec '19

Build: http://jenkins.ovirt.org/job/cleanup-gated-dummy-master/3/ Build Name: #3 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] deploy-to-gated-ovirt-4.3 - Build #87 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 25 Dec '19

25 Dec '19

Build: http://jenkins.ovirt.org/job/deploy-to-gated-ovirt-4.3/87/ Build Name: #87 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 1

oVirt infra daily report - unstable production jobs - 1049
by jenkins＠jenkins.phx.ovirt.org 25 Dec '19

25 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1049//artifact/exported-… Cheers, Jenkins

1 0

OST check-patch takes a very long time
by Yedidyah Bar David 24 Dec '19

24 Dec '19

Hi all, Yesterday I rebased [1], and CI still didn't finish, after more than 24 hours. Checking the console log, I already see it's going to fail, at least due to a timeout: testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_standard-check-patch/ovirt-system-tests/he-node-ng-suite-4.3/test-scenarios/008_restart_he_vm.py", line 59, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/agent/workspace/ovirt-system-tests_standard-check-patch/ovirt-system-tests/he-node-ng-suite-4.3/test-scenarios/008_restart_he_vm.py", line 191, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) A previous rebase of the same patch also failed (after 4.5 hours), and another attempt (ci please test) failed after 33 hours. Does this make sense? Seems like there is very high load there for some reason. [1] https://gerrit.ovirt.org/104827 -- Didi

1 0

Move HE suite to centos8
by Yedidyah Bar David 24 Dec '19

24 Dec '19

Hi all, Galit has been working on [1] for some time now. Willing to help with that (and getting back to a stable state with hosted-engine suites), I looked at it a few days ago, after manual deploy on centos8 with nightly master snapshot worked for me. I rebased her patches, and CI failed [2]. Out of 5 suites that it tested, 3 passed (!), and 2 failed :-( . I checked the two that failed, and see: 1. One of them failed [3] due to failing to parse JSON output from 'hosted-engine --vm-status --json'. The output seems to have been cut in the middle, after 1024 bytes: 2019-12-22 11:13:47,846::ssh.py::ssh::58::lago.ssh::DEBUG::Running 200606d0 on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-status --json 2019-12-22 11:13:48,441::ssh.py::ssh::81::lago.ssh::DEBUG::Command 200606d0 on lago-he-basic-role-remote-suite-4-3-host-0 returned with 0 2019-12-22 11:13:48,441::ssh.py::ssh::89::lago.ssh::DEBUG::Command 200606d0 on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4534 (Sun Dec 22 06:13:39 2019)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4535 (Sun Dec 22 06:13:40 2019)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "76bfd2ac", "local_conf_timestamp": 4535, "host-ts": 4534}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4532 (Sun Dec 22 06:13:38 2019)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4532 (Sun Dec 22 06:13:38 2019)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-s 2019-12-22 11:13:48,442::testlib.py::assert_equals_within::242::ovirtlago.testlib::ERROR:: * Unhandled exception in <function <lambda> at 0x7fcc58f38668> Traceback (most recent call last): If you check previous runs of this command, in the same lago log, you see that they are complete (and somewhat longer, but not much), e.g.: 2019-12-22 11:13:41,804::ssh.py::ssh::58::lago.ssh::DEBUG::Running 1c6c1226 on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-status --json 2019-12-22 11:13:43,105::ssh.py::ssh::81::lago.ssh::DEBUG::Command 1c6c1226 on lago-he-basic-role-remote-suite-4-3-host-0 returned with 0 2019-12-22 11:13:43,106::ssh.py::ssh::89::lago.ssh::DEBUG::Command 1c6c1226 on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4534 (Sun Dec 22 06:13:39 2019)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4535 (Sun Dec 22 06:13:40 2019)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "76bfd2ac", "local_conf_timestamp": 4535, "host-ts": 4534}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4532 (Sun Dec 22 06:13:38 2019)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4532 (Sun Dec 22 06:13:38 2019)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2fc395a1", "local_conf_timestamp": 4532, "host-ts": 4532}, "global_maintenance": true} So I suspect an infra issue - on the network level, ssh, etc. 2. The other seems to have failed [4] due to timeout: 2019-12-22 12:57:08,595::cmd.py::exit_handler::921::cli::DEBUG::signal 15 was caught The reason it took too long seems to be: 2019-12-22 11:08:13,233::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:57a44b4e-859e-4ff2-b897-7c8b3ade4557:Get ssh client for lago-he-node-ng-suite-4-3-host-0: 2019-12-22 11:08:13,637::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:57a44b4e-859e-4ff2-b897-7c8b3ade4557:Get ssh client for lago-he-node-ng-suite-4-3-host-0: 2019-12-22 11:08:13,942::ssh.py::ssh::58::lago.ssh::DEBUG::Running 59005d88 on lago-he-node-ng-suite-4-3-host-0: true 2019-12-22 11:08:15,605::ssh.py::ssh::81::lago.ssh::DEBUG::Command 59005d88 on lago-he-node-ng-suite-4-3-host-0 returned with 0 2019-12-22 11:08:15,606::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded for ssh to lago-he-node-ng-suite-4-3-host-0 2019-12-22 11:08:15,608::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:38e81fd0-50f0-497c-8fb0-38eedcfb3f55:Get ssh client for lago-he-node-ng-suite-4-3-host-0: 2019-12-22 11:08:16,129::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:38e81fd0-50f0-497c-8fb0-38eedcfb3f55:Get ssh client for lago-he-node-ng-suite-4-3-host-0: 2019-12-22 12:20:39,054::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4007e002-f8e2-4427-884f-af57f723aca0:Get ssh client for lago-he-node-ng-suite-4-3-storage: 2019-12-22 12:20:39,639::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4007e002-f8e2-4427-884f-af57f723aca0:Get ssh client for lago-he-node-ng-suite-4-3-storage: 2019-12-22 12:20:40,408::ssh.py::ssh::58::lago.ssh::DEBUG::Running 77b26f28 on lago-he-node-ng-suite-4-3-storage: true 2019-12-22 12:20:40,475::ssh.py::ssh::81::lago.ssh::DEBUG::Command 77b26f28 on lago-he-node-ng-suite-4-3-storage returned with 0 2019-12-22 12:20:40,476::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded for ssh to lago-he-node-ng-suite-4-3-storage Meaning, if I got it right, that it sshed to host-0, succeeded, then tried to ssh to storage, and also succeeded - but connecting took more than an hour. Again, I suspect some infra issue, perhaps same as (1.). Any clue? I am now trying it again, anyway [5]. Thanks and best regards, [1] https://gerrit.ovirt.org/104797 [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7269/ [3] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7269/… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7269/… [5] http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7273/ -- Didi

1 0

[oVirt Jenkins] cleanup-gated-ovirt-master - Build #9 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 24 Dec '19

24 Dec '19

Build: http://jenkins.ovirt.org/job/cleanup-gated-ovirt-master/9/ Build Name: #9 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

oVirt infra daily report - unstable production jobs - 1048
by jenkins＠jenkins.phx.ovirt.org 24 Dec '19

24 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1048//artifact/exported-… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2850) Fedora 29 EOL
by Ehud Yonasi (oVirt JIRA) 23 Dec '19

23 Dec '19

Ehud Yonasi created OVIRT-2850: ---------------------------------- Summary: Fedora 29 EOL Key: OVIRT-2850 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2850 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Ehud Yonasi Assignee: infra We need to prepare patches to remove it + mirrors, and let developers know to remove fc29 builds from CI. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100118)

1 0