December 2019 - Infra - oVirt List Archives

Container-based CI backend is now available for use
by Barak Korren 26 Mar '20

26 Mar '20

A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system. I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use. *What? Were? How do I use it?* Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index…> . *Wait! I needed it to do X which it doesn't!* For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement: 1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs As you can see, we have quite a big pile of reviews to do, as always, help is very welcome... Regards, Barak. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 4

[oVirt Jenkins] ovirt-system-tests_compat-4.3-suite-master - Build # 290 - Failure!
by jenkins＠jenkins.phx.ovirt.org 30 Jan '20

30 Jan '20

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/290/ Build Number: 290 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #290 [Marcin Sobczyk] deploy: Fix vdsm coverage script ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.verify_add_hosts_43 Error Message: None != True after 1200 seconds -------------------- >> begin captured logging << -------------------- 002_bootstrap: DEBUG: Host status, verify_add_hosts: lago-compat-4-3-suite-master-host-0: install_failed lago-compat-4-3-suite-master-host-1: installing --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 363, in verify_add_hosts timeout=constants.ADD_HOST_TIMEOUT File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) 'None != True after 1200 seconds\n-------------------- >> begin captured logging << --------------------\n002_bootstrap: DEBUG: Host status, verify_add_hosts:\n lago-compat-4-3-suite-master-host-0: install_failed\nlago-compat-4-3-suite-master-host-1: installing\n\n--------------------- >> end captured logging << ---------------------'

1 43

Build failed in Jenkins: system-sync_mirrors-epel-s390x-el7-s390x #109
by jenkins＠jenkins.phx.ovirt.org 03 Jan '20

03 Jan '20

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-s390x-el7-s390x/109/d…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-s390x-el7-s390x/ws/> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 454b5bc96ad293356da6af51df1f62b03ec0e823 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 454b5bc96ad293356da6af51df1f62b03ec0e823 Commit message: "pusher: Consider dest branch when checking if push is needed" > git rev-list --no-walk 454b5bc96ad293356da6af51df1f62b03ec0e823 # timeout=10 [system-sync_mirrors-epel-s390x-el7-s390x] $ /bin/bash -xe /tmp/jenkins5659741722053903631.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-s390x-el7 s390x jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror epel-s390x-el7 s390x jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror epel-s390x-el7 s390x jenkins/data/mirrors-reposync.conf + local repo_name=epel-s390x-el7 + local repo_archs=s390x + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs epel-s390x-el7 yum + local repo_name=epel-s390x-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/epel-s390x-el7 /var/www/html/repos/yum/epel-s390x-el7/base + check_yum_sync_needed epel-s390x-el7 s390x jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=epel-s390x-el7 + local repo_archs=s390x + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/epel-s390x-el7 ++ IFS=, ++ echo s390x + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync epel-s390x-el7 s390x jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=epel-s390x-el7 ++ local repo_arch=s390x ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=epel-s390x-el7 --arch=s390x --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/epel-s390x-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: failure: repodata/bc08248d5f12773d2cacabd95b08cca382c882312507ff751455fb3b07d8ab22-primary.sqlite.bz2 from epel-s390x-el7: [Errno 256] No more mirrors to try. http://download.sinenomine.net/clefos/epel7/repodata/bc08248d5f12773d2cacab…: [Errno 14] HTTP Error 404 - Not Found + reposync_out= Build step 'Execute shell' marked build as failure

1 24

oVirt infra daily report - unstable production jobs - 1056
by jenkins＠jenkins.phx.ovirt.org 01 Jan '20

01 Jan '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1056//artifact/exported-… Cheers, Jenkins

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-7.6-el7-x86_64 #208
by jenkins＠jenkins.phx.ovirt.org 31 Dec '19

31 Dec '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-x86…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-x86…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision beba1d2f2e3531c000852f54431c5432bc76f149 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f beba1d2f2e3531c000852f54431c5432bc76f149 Commit message: "decorate.py: Add script" > git rev-list --no-walk beba1d2f2e3531c000852f54431c5432bc76f149 # timeout=10 [system-sync_mirrors-centos-updates-7.6-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5376853183813924811.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-updates-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-updates-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-updates-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-updates-7.6-el7 yum + local repo_name=centos-updates-7.6-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-updates-7.6-el7 /var/www/html/repos/yum/centos-updates-7.6-el7/base + check_yum_sync_needed centos-updates-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-updates-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-updates-7.6-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-updates-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-updates-7.6-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-updates-7.6-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-updates-7.6-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-7.6-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-7.6-el7' + reposync_out= Build step 'Execute shell' marked build as failure

1 1

[CQ]: 105993, 2 (ovirt-engine-api-model) failed "ovirt-master" system tests
by oVirt Jenkins 31 Dec '19

31 Dec '19

Change 105993,2 (ovirt-engine-api-model) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/105993/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17876/

1 0

[CQ]: 6b88172 (ovirt-engine-sdk-go) failed "ovirt-4.3" system tests
by oVirt Jenkins 31 Dec '19

31 Dec '19

Change 6b88172 (ovirt-engine-sdk-go) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-engine-sdk-go/commit/6b8817275dc97a3b99354ed… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2732/

1 0

[CQ]: 6b88172 (ovirt-engine-sdk-go) failed "ovirt-master" system tests
by oVirt Jenkins 31 Dec '19

31 Dec '19

Change 6b88172 (ovirt-engine-sdk-go) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-engine-sdk-go/commit/6b8817275dc97a3b99354ed… For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17872/

1 0

oVirt infra daily report - unstable production jobs - 1055
by jenkins＠jenkins.phx.ovirt.org 31 Dec '19

31 Dec '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/1055//artifact/exported-… Cheers, Jenkins

1 0

Build failed in Jenkins: system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64 #204
by jenkins＠jenkins.phx.ovirt.org 30 Dec '19

30 Dec '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.6…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.6…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 454b5bc96ad293356da6af51df1f62b03ec0e823 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 454b5bc96ad293356da6af51df1f62b03ec0e823 Commit message: "pusher: Consider dest branch when checking if push is needed" > git rev-list --no-walk 454b5bc96ad293356da6af51df1f62b03ec0e823 # timeout=10 [system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64] $ /bin/bash -xe /tmp/jenkins139911091726115604.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-sclo-rh-release-7.6-el7 yum + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7 /var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7/base + check_yum_sync_needed centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-sclo-rh-release-7.6-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-sclo-rh-release-7.6-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-sclo-rh-release-7.6-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-sclo-rh-release-7.6-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-sclo-rh-release-7.6-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-sclo-rh-release-7.6-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: failure: repodata/be198d32d786c933beb06e141f7f49d7b9671df70ecb09f05c39370d498a5f57-primary.sqlite.bz2 from centos-sclo-rh-release-7.6-el7: [Errno 256] No more mirrors to try. http://vault.centos.org/7.6.1810/sclo/x86_64/rh/repodata/be198d32d786c933be…: [Errno 16] error setting timestamp on file /home/jenkins/mirrors_cache/centos-sclo-rh-release-7.6-el7/be198d32d786c933beb06e141f7f49d7b9671df70ecb09f05c39370d498a5f57-primary.sqlite.bz2 from http://vault.centos.org/7.6.1810/sclo/x86_64/rh/repodata/be198d32d786c933be…, OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/centos-sclo-rh-release-7.6-el7/be198d32d786c933beb06e141f7f49d7b9671df70ecb09f05c39370d498a5f57-primary.sqlite.bz2' + reposync_out= Build step 'Execute shell' marked build as failure

1 1