December 2019 - Infra - oVirt List Archives

published master engine is 13 days old
by Yedidyah Bar David 17 Dec '19

17 Dec '19

Hi all, $subject. [1] has ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch . Tried to look around, and I have a few notes/questions: 1. Last successful run of [2] is 3 days old, but apparently it wasn't published. Any idea why? 2. Failed runs of [2] are reported to infra, with emails such as: [CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause Is anyone monitoring these? Is this the only alerting that CI generates on such failures? If first is No and second is Yes, then we need someone/something to start monitoring. This was discussed a lot, but I do not see any change. Ideally, such alerts should be To'ed or Cc'ed to the author and reviewers of the patch that CI found to be guilty (which might be wrong, that's not the point). Do we plan to have something like this? Any idea when it will be ready? 3. I looked at a few recent failures of [2], specifically [3][4]. Both seem to have been killed after a timeout, while running 'engine-config'. For [3] that's clear, see [5]: 2019-12-16 17:11:44,766::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fb6611dc-55bb-4251-aeda-2578b2ec83a2:Get ssh client for lago-basic-suite-master-engine: 2019-12-16 17:11:44,931::ssh.py::ssh::58::lago.ssh::DEBUG::Running 22e2b6b6 on lago-basic-suite-master-engine: engine-config --set VdsmUseNmstate=true 2019-12-16 19:55:21,965::cmd.py::exit_handler::921::cli::DEBUG::signal 15 was caught Can't find stdout/stderr of engine-config, so it's hard to tell if it outputted anything helpful to understand why it was stuck. It's hard to tell that about [4], because it has very few artifacts collected, no idea why, notably no lago.log, but [6] does show: [36m # initialize_engine: [32mSuccess [0m (in 0:04:00) [0m [36m # engine_config: [0m [0m [0m [36m * Collect artifacts: [0m [0m [0m [36m - [Thread-34] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:04) [0m [36m * Collect artifacts: [31mERROR [0m (in 0:00:04) [0m [36m # engine_config: [31mERROR [0m (in 2:42:57) [0m /bin/bash: line 31: 5225 Killed ${_STDCI_TIMEOUT_CMD} "3h" "$script_path" < /dev/null If I run 'engine-config --set VdsmUseNmstate=true' on my 20191204120550.git04d5d05 engine, it returns quickly. Tried also adding a repo pointing at last successful run of [7], which is currently [8], and it prompts me to input a version, probably as a result of [9]. Ales/Martin, can you please have a look? Thanks. [1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/ [3] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/ [4] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/ [5] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/artifa… [6] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/artifa… [7] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/ [8] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/384/ [9] https://gerrit.ovirt.org/105440 -- Didi

3 3

oVirt infra daily report - unstable production jobs - 1041
by jenkins＠jenkins.phx.ovirt.org 17 Dec '19

17 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1041//artifact/exported-… Cheers, Jenkins

1 0

[CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 16 Dec '19

16 Dec '19

A system test invoked by the "ovirt-master" change queue including change 105472,5 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 104569,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 104569,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105472/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/104569/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/

1 0

[JIRA] (OVIRT-2848) Ability to upload to artifactory.orvit.org
by Evgheni Dereveanchin (oVirt JIRA) 16 Dec '19

16 Dec '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2848?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2848: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Ability to upload to artifactory.orvit.org > ------------------------------------------ > > Key: OVIRT-2848 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2848 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Piotr Kliczewski > Assignee: Evgheni Dereveanchin > > Hi, > I would like to be able to upload to artifactory because I am experiencing > issues with maven central. > Thanks, > Piotr -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100116)

1 0

Build failed in Jenkins: system-sync_mirrors-mock-copr-el7-x86_64 #651
by jenkins＠jenkins.phx.ovirt.org 16 Dec '19

16 Dec '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-mock-copr-el7-x86_64/651/d…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-mock-copr-el7-x86_64/ws/> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 4641b50e98783e124f82197c56d7728a397e987a (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 4641b50e98783e124f82197c56d7728a397e987a Commit message: "Added NFS server container image" > git rev-list --no-walk 4641b50e98783e124f82197c56d7728a397e987a # timeout=10 [system-sync_mirrors-mock-copr-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2118053342586622187.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror mock-copr-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror mock-copr-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror mock-copr-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=mock-copr-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs mock-copr-el7 yum + local repo_name=mock-copr-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/mock-copr-el7 /var/www/html/repos/yum/mock-copr-el7/base + check_yum_sync_needed mock-copr-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=mock-copr-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/mock-copr-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync mock-copr-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=mock-copr-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=mock-copr-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/mock-copr-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: Error making cache directory: /home/jenkins/mirrors_cache/mock-copr-el7/gen error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/mock-copr-el7/gen' + reposync_out= Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_compat-4.3-suite-master - Build # 282 - Failure!
by jenkins＠jenkins.phx.ovirt.org 16 Dec '19

16 Dec '19

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/282/ Build Number: 282 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #282 [Galit Rosenthal] Move ansible master to centos8 with no reposync ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.verify_add_hosts_43 Error Message: None != True after 1200 seconds -------------------- >> begin captured logging << -------------------- 002_bootstrap: DEBUG: Host status, verify_add_hosts: lago-compat-4-3-suite-master-host-0: install_failed lago-compat-4-3-suite-master-host-1: installing --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 363, in verify_add_hosts timeout=constants.ADD_HOST_TIMEOUT File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) 'None != True after 1200 seconds\n-------------------- >> begin captured logging << --------------------\n002_bootstrap: DEBUG: Host status, verify_add_hosts:\n lago-compat-4-3-suite-master-host-0: install_failed\nlago-compat-4-3-suite-master-host-1: installing\n\n--------------------- >> end captured logging << ---------------------'

1 7

oVirt infra daily report - unstable production jobs - 1040
by jenkins＠jenkins.phx.ovirt.org 16 Dec '19

16 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1040//artifact/exported-… Cheers, Jenkins

1 0

[CQ]: 105471, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 16 Dec '19

16 Dec '19

A system test invoked by the "ovirt-master" change queue including change 105471,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 104569,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 104569,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105471/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/104569/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/

1 0

[CQ]: 104213, 16 (jenkins) failed "dummy-master" system tests, but isn't the failure root cause
by oVirt Jenkins 15 Dec '19

15 Dec '19

A system test invoked by the "dummy-master" change queue including change 104213,16 (jenkins) failed. However, this change seems not to be the root cause for this failure. Change 102662,7 (jenkins) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 102662,7 (jenkins) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/104213/16 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/102662/7 For failed test results see: https://jenkins-staging.ovirt.org/job/dummy-master_change-queue-tester/459/

1 0

Build failed in Jenkins: system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64 #157
by jenkins＠jenkins.phx.ovirt.org 15 Dec '19

15 Dec '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.6…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.6…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision b22e8cf6cc5d637ac7049094c31be5df759b1697 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f b22e8cf6cc5d637ac7049094c31be5df759b1697 Commit message: "deploy_to_container: Support flattening images" > git rev-list --no-walk b22e8cf6cc5d637ac7049094c31be5df759b1697 # timeout=10 [system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64] $ /bin/bash -xe /tmp/jenkins7275009757952587151.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-sclo-rh-release-7.6-el7 yum + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7 /var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7/base + check_yum_sync_needed centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-sclo-rh-release-7.6-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-sclo-rh-release-7.6-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-sclo-rh-release-7.6-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: centos-sclo-rh-release-7.6-el7: Check uncompressed DB failed + reposync_out= Build step 'Execute shell' marked build as failure

1 2