- Infra - oVirt List Archives

[JIRA] (OVIRT-935) Unable to build fc25 chroots with mock
by Anton Marchukov (oVirt JIRA) 20 Dec '16

20 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-935?page=com.atlassian.jira.p… ] Anton Marchukov commented on OVIRT-935: --------------------------------------- Official information: https://github.com/rpm-software-management/mock/wiki#mock-on-el-6-and-el-7-… http://miroslav.suchy.cz/blog/archives/2015/05/20/why_mock_does_not_work_on… the later says: "This is something which you may want to do, when you are building some packages for personal usage. However you should avoid that if you are operating some build system. In such case migration to some recent Fedora is your only option." and "Change Mock settings that EL6 and EL7 users will continue to use Yum for building Fedora rawhide (and F23+). While this option seems to be good trade-off, there is big risk that you end up with different binary compared to the same process on Fedoras. For example there was issue that Rubygems requires ruby(release), which Yum resolved to normal ruby, however DNF resolved that to jruby. And that is big difference." and all the workarounds proposed as I see for the opposite - use yum enabled mocks on fedora. Upgrading Fedora seems to me standard and supported procedure and business as usual. If we do it manually we can automate it so it scales. Inserting hacks into "yum" to work with Fedora composes is not standard and not supported. Hence it is not possible to automate and it will always be manual work on its own and thus I do not think it is sustainable solution for future. Especially since we do have Fedora slaves already and have to support them anyway. > Unable to build fc25 chroots with mock > -------------------------------------- > > Key: OVIRT-935 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-935 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Nadav Goldin > Assignee: infra > > Mock fails when trying to build the chroot for fedora25, on installing '@buildsys-build' group(with dnf) stage. > Open bug: https://bugzilla.redhat.com/show_bug.cgi?id=1360781 > Temporarily workaround: add the 'libcrypt' package to '.packages.fc25' file. > Possible solution: patch mock-*fc25*.cfg file in Jenkins repository to always install 'libcrypt' first. > Until fixed this will block us from building anything with mock on fc25(hit it when trying to build fc25 RPMs for lago). > Error logs: > {code} > DEBUG util.py:502: Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/fedora-25-x86_64-2474d86945a1de9c6d14549ec2401b9c-8291/root/', '--releasever', '25', 'install', '@buildsys-build', 'git', 'python', 'python-dulwich', 'python-setuptools', 'yum', 'yum-utils', '--setopt=tsflags=nocontexts'] with env {'PS1': '<mock-chroot> \\s-\\v\\$ ', > .... > DEBUG util.py:421: Yum command has been deprecated, use dnf instead. > DEBUG util.py:421: See 'man dnf' and 'man yum2dnf' for more information. > DEBUG util.py:421: Error: libcrypt conflicts with libcrypt-nss-2.24-3.fc25.x86_64 > DEBUG util.py:421: Error: libcrypt-nss conflicts with libcrypt-2.24-3.fc25.x86_64 > DEBUG util.py:421: You could try using --skip-broken to work around the problem > DEBUG util.py:421: You could try running: rpm -Va --nofiles --nodigest > DEBUG util.py:557: Child return code was: 1 > DEBUG util.py:180: kill orphans > {code} -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

Gerrit hooks change 4.1 bugs status to MODIFIED if merged on master
by Tal Nisan 20 Dec '16

20 Dec '16

This should be change to move to MODIFIED only after the bug is merged on the 4.1 branch. Here's an example: https://gerrit.ovirt.org/#/c/68739/ Thanks.

3 2

[JIRA] (OVIRT-935) Unable to build fc25 chroots with mock
by eyal edri [Administrator] (oVirt JIRA) 20 Dec '16

20 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-935?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-935: ------------------------------------------------- This comes with a price - if we'll have more fedora slaves than centos due to this, then we'll need to to mass upgrades much more often, due to the speed of fedora releases goes EOL > Unable to build fc25 chroots with mock > -------------------------------------- > > Key: OVIRT-935 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-935 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Nadav Goldin > Assignee: infra > > Mock fails when trying to build the chroot for fedora25, on installing '@buildsys-build' group(with dnf) stage. > Open bug: https://bugzilla.redhat.com/show_bug.cgi?id=1360781 > Temporarily workaround: add the 'libcrypt' package to '.packages.fc25' file. > Possible solution: patch mock-*fc25*.cfg file in Jenkins repository to always install 'libcrypt' first. > Until fixed this will block us from building anything with mock on fc25(hit it when trying to build fc25 RPMs for lago). > Error logs: > {code} > DEBUG util.py:502: Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/fedora-25-x86_64-2474d86945a1de9c6d14549ec2401b9c-8291/root/', '--releasever', '25', 'install', '@buildsys-build', 'git', 'python', 'python-dulwich', 'python-setuptools', 'yum', 'yum-utils', '--setopt=tsflags=nocontexts'] with env {'PS1': '<mock-chroot> \\s-\\v\\$ ', > .... > DEBUG util.py:421: Yum command has been deprecated, use dnf instead. > DEBUG util.py:421: See 'man dnf' and 'man yum2dnf' for more information. > DEBUG util.py:421: Error: libcrypt conflicts with libcrypt-nss-2.24-3.fc25.x86_64 > DEBUG util.py:421: Error: libcrypt-nss conflicts with libcrypt-2.24-3.fc25.x86_64 > DEBUG util.py:421: You could try using --skip-broken to work around the problem > DEBUG util.py:421: You could try running: rpm -Va --nofiles --nodigest > DEBUG util.py:557: Child return code was: 1 > DEBUG util.py:180: kill orphans > {code} -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-935) Unable to build fc25 chroots with mock
by Anton Marchukov (oVirt JIRA) 20 Dec '16

20 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-935?page=com.atlassian.jira.p… ] Anton Marchukov commented on OVIRT-935: --------------------------------------- I support [~ederevea] in that we should not try to maintain non supported options ourselves if we have options. I would better use fedora executors with dnf for fedora mock jobs. This is part of reality that using yum with FC25 is no longer supported and will never be supported, that mock uses external package management tool to install packages and that there is no dnf in centos7. All this combined means that we cannot normally use Fedora mock on CentOS right now. And since we have Fedora slaves we can run such jobs on Fedora. > Unable to build fc25 chroots with mock > -------------------------------------- > > Key: OVIRT-935 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-935 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Nadav Goldin > Assignee: infra > > Mock fails when trying to build the chroot for fedora25, on installing '@buildsys-build' group(with dnf) stage. > Open bug: https://bugzilla.redhat.com/show_bug.cgi?id=1360781 > Temporarily workaround: add the 'libcrypt' package to '.packages.fc25' file. > Possible solution: patch mock-*fc25*.cfg file in Jenkins repository to always install 'libcrypt' first. > Until fixed this will block us from building anything with mock on fc25(hit it when trying to build fc25 RPMs for lago). > Error logs: > {code} > DEBUG util.py:502: Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/fedora-25-x86_64-2474d86945a1de9c6d14549ec2401b9c-8291/root/', '--releasever', '25', 'install', '@buildsys-build', 'git', 'python', 'python-dulwich', 'python-setuptools', 'yum', 'yum-utils', '--setopt=tsflags=nocontexts'] with env {'PS1': '<mock-chroot> \\s-\\v\\$ ', > .... > DEBUG util.py:421: Yum command has been deprecated, use dnf instead. > DEBUG util.py:421: See 'man dnf' and 'man yum2dnf' for more information. > DEBUG util.py:421: Error: libcrypt conflicts with libcrypt-nss-2.24-3.fc25.x86_64 > DEBUG util.py:421: Error: libcrypt-nss conflicts with libcrypt-2.24-3.fc25.x86_64 > DEBUG util.py:421: You could try using --skip-broken to work around the problem > DEBUG util.py:421: You could try running: rpm -Va --nofiles --nodigest > DEBUG util.py:557: Child return code was: 1 > DEBUG util.py:180: kill orphans > {code} -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-945) Add upgrade suites for 4.1
by Daniel Belenky (oVirt JIRA) 20 Dec '16

20 Dec '16

Daniel Belenky created OVIRT-945: ------------------------------------ Summary: Add upgrade suites for 4.1 Key: OVIRT-945 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-945 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Daniel Belenky Assignee: infra -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-945) Add upgrade suites for 4.1
by Daniel Belenky (oVirt JIRA) 20 Dec '16

20 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-945?page=com.atlassian.jira.p… ] Daniel Belenky reassigned OVIRT-945: ------------------------------------ Assignee: (was: infra) > Add upgrade suites for 4.1 > -------------------------- > > Key: OVIRT-945 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-945 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Daniel Belenky > -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #8 - SUCCESS!
by jenkins＠jenkins.phx.ovirt.org 20 Dec '16

20 Dec '16

------=_Part_548_537680974.1482223668400 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/8/, Build Number: 8, Build Status: SUCCESS ------=_Part_548_537680974.1482223668400--

1 0

Build failed in Jenkins: ovirt_4.1_system-tests #1
by jenkins＠jenkins.phx.ovirt.org 20 Dec '16

20 Dec '16

See <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/1/> ------------------------------------------ [...truncated 74 lines...] Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/basic_suite_master.sh-el7 /home/jenkins/workspace/test-repo_ovirt_experimental_master Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 3.2M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 22G 886G 3% / tmpfs 24G 336K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1012 ################################################################### # Slave cleanup done # ################################################################### [ovirt_4.1_system-tests] $ /bin/bash -xe /tmp/hudson3221576944294627045.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=4.1 + SUITE_TYPE=basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + OVIRT_SUITE=basic_suite_4.1 + PREFIX=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/d…> + chmod g+x <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/d…> + cd ovirt-system-tests + ../jenkins/mock_configs/mock_runner.sh --mock-confs-dir ../jenkins/mock_configs --try-proxy --execute-script automation/basic_suite_4.1.sh el7 ########################################################## ########################################################## ## Mon Dec 19 14:42:52 UTC 2016 Running env: el7:epel-7-x86_64 ########################################################## ERROR: Script automation/basic_suite_4.1.sh does not exist or is not readable. ########################################################## ## Mon Dec 19 14:42:52 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 0 seconds ## rc = 1 ########################################################## No log files found, check command output ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.1 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.1_system-tests] $ /bin/bash -xe /tmp/hudson4079159423720154558.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.1 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + OVIRT_SUITE=4.1 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/1/artifact/exported-art…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/1/artifact/exported-art…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then for log in "${logs[@]}"; do [[ "$log" = ./exported-artifacts/* ]] && continue echo "Copying ${log} to exported-artifacts" mv $log exported-artifacts/ done fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_4.1_system-tests] $ /bin/bash -x /tmp/hudson5498140227125921290.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + for log in '"${logs[@]}"' + [[ ./ovirt-system-tests/logs = ./exported-artifacts/* ]] + echo 'Copying ./ovirt-system-tests/logs to exported-artifacts' Copying ./ovirt-system-tests/logs to exported-artifacts + mv ./ovirt-system-tests/logs exported-artifacts/ + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

1 6

Build failed in Jenkins: ovirt_master_system-tests #871
by jenkins＠jenkins.phx.ovirt.org 20 Dec '16

20 Dec '16

See <http://jenkins.ovirt.org/job/ovirt_master_system-tests/871/changes> Changes: [Yaniv Kaul] Move log-collector to use /dev/shm for faster execution [Evgheni Dereveanchin] OVIRT-645 remove repoproxy for Fedora and EL6 x86_64 [Barak Korren] Make mock_cleanup.sh safer about un-mouting [Barak Korren] Fix mock_genconfig nested dict settings ------------------------------------------ [...truncated 779 lines...] + [[ -e <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 640 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=master SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson2709353388353252686.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=master + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + OVIRT_SUITE=master + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_master_system-tests/871/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_master_system-tests/871/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/871/artifact/exporte…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then for log in "${logs[@]}"; do [[ "$log" = ./exported-artifacts/* ]] && continue echo "Copying ${log} to exported-artifacts" mv $log exported-artifacts/ done fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_master_system-tests] $ /bin/bash -x /tmp/hudson7578817524005175999.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + for log in '"${logs[@]}"' + [[ ./ovirt-system-tests/logs = ./exported-artifacts/* ]] + echo 'Copying ./ovirt-system-tests/logs to exported-artifacts' Copying ./ovirt-system-tests/logs to exported-artifacts + mv ./ovirt-system-tests/logs exported-artifacts/ + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> + mock_root=epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + [[ -n epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results Archiving artifacts

1 5

Ability to run [job-name] when a tag is pushed to Gerrit
by Vojtech Szocs 19 Dec '16

19 Dec '16

Hi, this was discussed loong time ago.. how much effort would it take for oVirt CI infra to support running e.g. [build-artifacts] when a git tag is pushed to Gerrit repo? Imagine the following situation: 1, new tag is pushed -or- existing tag gets updated (force push) 2, CI infra detects the new/updated tag and runs [job-name-here] on the relevant (tagged) commit I'm asking this because in oVirt Dashboard, we're making releases from tagged commits, so we basically: - merge patch that prepares the (stable) branch for release - tag the release-prep patch & push tag to Gerrit - re-trigger [build-artifacts] for the release-prep patch Thanks, Vojtech

3 5