On Thu, Jan 12, 2017 at 10:50 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi,
Looks like the ovirt-engine-appliance package is missing again.
Looks like jenkins job became unstable. I'm looking into them. Yuval, once you're done with the 4.1 appliance can you look into the other appliance jobs failing?
Thanks in advance, Lev Veyde.
----- Original Message ----- From: jenkins@jenkins.phx.ovirt.org To: sbonazzo@redhat.com, infra@ovirt.org, lveyde@redhat.com Sent: Thursday, January 12, 2017 4:56:52 AM Subject: Build failed in Jenkins: ovirt_3.6_he-system-tests #809
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/changes>
Changes:
[Roy Golan] accept defaults of aaa-ldap plugin installation
[Sandro Bonazzola] 4.1 nightly: publish 4.1 dashboard in 4.1 repo
[Eyal Edri] fixing appliance to take from 4.1
[Daniel Belenky] Added 4.1 branch for ovirt-scheduler-proxy
[Daniel Belenky] Added build for fc24+ovirt4.0
[Daniel Belenky] Added build for fc24+ovirt4.0
[Daniel Belenky] install python-requests as a dependency
[Pavel Zhukov] Moved publishers section to job definition instead of projects
[Barak Korren] Revert "install python-requests as a dependency"
------------------------------------------ [...truncated 110.81 KB...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Thu Jan 12 02:56:45 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 275 seconds @@ rc = 127 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he- system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.3.3 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Thu Jan 12 02:56:51 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 281 seconds ## rc = 127 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_ basic_suite_3.6.sh' './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh'
# # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE=
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/ hudson2934459229741657026.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he- system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/ artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/ artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/exported-artifacts/failure_msg.txt> < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/exported-artifacts/lago_logs> < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/exported-artifacts/mock_logs> < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/ artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
shopt -s nullglob
WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}"
safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 }
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
# remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them
# Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done
if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi
[ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/ hudson3269842129067440856.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he- system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he- system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he- system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.3.3 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + [[ -n epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com