- Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 1042
by jenkins＠jenkins.phx.ovirt.org 17 Dec '19

17 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1042//artifact/exported-… Cheers, Jenkins

1 0

[CQ]: 105184,6 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 17 Dec '19

17 Dec '19

Change 105184,6 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/105184/6 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17775/

1 0

[CQ]: 105785,1 (ovirt-system-tests) failed "ovirt-4.3" system tests
by oVirt Jenkins 17 Dec '19

17 Dec '19

Change 105785,1 (ovirt-system-tests) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/105785/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2704/

1 0

[CQ]: 105777,5 (ovirt-provider-ovn) failed "ovirt-4.3" system tests
by oVirt Jenkins 17 Dec '19

17 Dec '19

Change 105777,5 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/105777/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2701/

1 0

[CQ]: 102665,15 (jenkins) failed "dummy-master" system tests
by oVirt Jenkins 17 Dec '19

17 Dec '19

Change 102665,15 (jenkins) is probably the reason behind recent system test failures in the "dummy-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/102665/15 For failed test results see: https://jenkins-staging.ovirt.org/job/dummy-master_change-queue-tester/472/

1 0

CI system outage
by Evgheni Dereveanchin 17 Dec '19

17 Dec '19

Hello everyone, We received several reports of Jenkins jobs not running properly. The reason turned out to be the Jenkins Master which did not start jobs properly. It has been restarted and is working properly now. If you have any jobs that failed or did not start at all today - please re-trigger them or reach out to me for assistance. -- Regards, Evgheni Dereveanchin

1 0

[CQ]: 104756,8 (jenkins) failed "dummy-master" system tests
by oVirt Jenkins 17 Dec '19

17 Dec '19

Change 104756,8 (jenkins) is probably the reason behind recent system test failures in the "dummy-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/104756/8 For failed test results see: http://jenkins.ovirt.org/job/dummy-master_change-queue-tester/240/

1 0

published master engine is 13 days old
by Yedidyah Bar David 17 Dec '19

17 Dec '19

Hi all, $subject. [1] has ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch . Tried to look around, and I have a few notes/questions: 1. Last successful run of [2] is 3 days old, but apparently it wasn't published. Any idea why? 2. Failed runs of [2] are reported to infra, with emails such as: [CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause Is anyone monitoring these? Is this the only alerting that CI generates on such failures? If first is No and second is Yes, then we need someone/something to start monitoring. This was discussed a lot, but I do not see any change. Ideally, such alerts should be To'ed or Cc'ed to the author and reviewers of the patch that CI found to be guilty (which might be wrong, that's not the point). Do we plan to have something like this? Any idea when it will be ready? 3. I looked at a few recent failures of [2], specifically [3][4]. Both seem to have been killed after a timeout, while running 'engine-config'. For [3] that's clear, see [5]: 2019-12-16 17:11:44,766::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fb6611dc-55bb-4251-aeda-2578b2ec83a2:Get ssh client for lago-basic-suite-master-engine: 2019-12-16 17:11:44,931::ssh.py::ssh::58::lago.ssh::DEBUG::Running 22e2b6b6 on lago-basic-suite-master-engine: engine-config --set VdsmUseNmstate=true 2019-12-16 19:55:21,965::cmd.py::exit_handler::921::cli::DEBUG::signal 15 was caught Can't find stdout/stderr of engine-config, so it's hard to tell if it outputted anything helpful to understand why it was stuck. It's hard to tell that about [4], because it has very few artifacts collected, no idea why, notably no lago.log, but [6] does show: [36m # initialize_engine: [32mSuccess [0m (in 0:04:00) [0m [36m # engine_config: [0m [0m [0m [36m * Collect artifacts: [0m [0m [0m [36m - [Thread-34] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:04) [0m [36m * Collect artifacts: [31mERROR [0m (in 0:00:04) [0m [36m # engine_config: [31mERROR [0m (in 2:42:57) [0m /bin/bash: line 31: 5225 Killed ${_STDCI_TIMEOUT_CMD} "3h" "$script_path" < /dev/null If I run 'engine-config --set VdsmUseNmstate=true' on my 20191204120550.git04d5d05 engine, it returns quickly. Tried also adding a repo pointing at last successful run of [7], which is currently [8], and it prompts me to input a version, probably as a result of [9]. Ales/Martin, can you please have a look? Thanks. [1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/ [3] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/ [4] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/ [5] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/artifa… [6] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/artifa… [7] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/ [8] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/384/ [9] https://gerrit.ovirt.org/105440 -- Didi

3 3

oVirt infra daily report - unstable production jobs - 1041
by jenkins＠jenkins.phx.ovirt.org 16 Dec '19

16 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1041//artifact/exported-… Cheers, Jenkins

1 0

[CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 16 Dec '19

16 Dec '19

A system test invoked by the "ovirt-master" change queue including change 105472,5 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 104569,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 104569,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105472/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/104569/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/

1 0