Looks like jenkins job became unstable.
I'm looking into them.
Yuval, once you're done with the 4.1 appliance can you look into the other
appliance jobs failing?
Thanks in advance,
Lev Veyde.
----- Original Message -----
From: jenkins(a)jenkins.phx.ovirt.org
To: sbonazzo(a)redhat.com, infra(a)ovirt.org, lveyde(a)redhat.com
Sent: Thursday, January 12, 2017 4:56:52 AM
Subject: Build failed in Jenkins: ovirt_3.6_he-system-tests #809
See <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/changes>
Changes:
[Roy Golan] accept defaults of aaa-ldap plugin installation
[Sandro Bonazzola] 4.1 nightly: publish 4.1 dashboard in 4.1 repo
[Eyal Edri] fixing appliance to take from 4.1
[Daniel Belenky] Added 4.1 branch for ovirt-scheduler-proxy
[Daniel Belenky] Added build for fc24+ovirt4.0
[Daniel Belenky] Added build for fc24+ovirt4.0
[Daniel Belenky] install python-requests as a dependency
[Pavel Zhukov] Moved publishers section to job definition instead of
projects
[Barak Korren] Revert "install python-requests as a dependency"
------------------------------------------
[...truncated 110.81 KB...]
Finish: shell
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@ Thu Jan 12 02:56:45 UTC 2017 automation/he_basic_suite_3.6.sh chroot
finished
@@ took 275 seconds
@@ rc = 127
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
========== Scrubbing chroot
mock \
--configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-
system-tests/ws/ovirt-system-tests"> \
--root="mocker-epel-7-x86_64.el7" \
--resultdir="./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.scrub"
\
--scrub=chroot
WARNING: Could not find required logging config file: <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/logging.ini.> Using default...
INFO: mock.py version 1.3.3 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Start: scrub ['chroot']
INFO: scrubbing chroot for mocker-epel-7-x86_64.el7
Finish: scrub ['chroot']
Finish: run
Scrub chroot took 6 seconds
============================
##########################################################
## Thu Jan 12 02:56:51 UTC 2017 Finished env: el7:epel-7-x86_64
## took 281 seconds
## rc = 127
##########################################################
find: 'logs': No such file or directory
No log files found, check command output
##!########################################################
Collecting mock logs
'./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.clean_rpmdb' ->
'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb'
'./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' ->
'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_
basic_suite_3.6.sh'
'./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.init' ->
'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init'
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=3.6
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/
hudson2934459229741657026.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=3.6
+ SUITE_TYPE=
+
WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/>
+ OVIRT_SUITE=3.6
+
TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-
system-tests/ws/ovirt-system-tests/exported-artifacts>
+ rm -rf <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/
artifact/exported-artifacts>
+ mkdir -p <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/
artifact/exported-artifacts>
+ [[ -d <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/exported-artifacts> ]]
+ mv <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/exported-artifacts/failure_msg.txt> <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/exported-artifacts/lago_logs> <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/exported-artifacts/mock_logs> <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/
artifact/exported-artifacts/>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -x
echo "shell-scripts/mock_cleanup.sh"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
shopt -s nullglob
WORKSPACE="${WORKSPACE:-$PWD}"
UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}"
UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}"
safe_umount() {
local mount="${1:?}"
local attempt
for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do
# If this is not the 1st time through the loop, Sleep a while to
let
# the problem "solve itself"
[[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY"
# Try to umount
sudo umount --lazy "$mount" && return 0
# See if the mount is already not there despite failing
findmnt --kernel --first "$mount" > /dev/null && return 0
done
echo "ERROR: Failed to umount $mount."
return 1
}
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs
sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to
umount."
fi
for mount in "${mounts[@]}"; do
safe_umount "$mount" || failed=true
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) ||
:
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
safe_umount "$mount" && continue
# If we got here, we failed $UMOUNT_RETRIES attempts so we should
make
# noise
failed=true
this_chroot_failed=true
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
# remove mock caches that are older then 2 days:
find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \
xargs -0 -tr sudo rm -rf
# We make no effort to leave around caches that may still be in use because
# packages installed in them may go out of date, so may as well recreate
them
# Drop all left over libvirt domains
for UUID in $(virsh list --all --uuid); do
virsh destroy $UUID || :
sleep 2
virsh undefine --remove-all-storage --storage vda --snapshots-metadata
$UUID || :
done
if $failed; then
echo "Cleanup script failed, propegating failure to job"
exit 1
fi
[ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/
hudson3269842129067440856.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ shopt -s nullglob
+
WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/>
+ UMOUNT_RETRIES=3
+ UMOUNT_RETRY_DELAY=1s
+ sudo chown -R jenkins <
http://jenkins.ovirt.org/job/
ovirt_3.6_he-system-tests/ws/>
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-epel-7-x86_64.el7.cfg
+ mock_root=mocker-epel-7-x86_64.el7
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-
system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-epel-7-x86_64.el7'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-
system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-
system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7
--resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/>
--orphanskill
WARNING: Could not find required logging config file: <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/logging.ini.> Using default...
INFO: mock.py version 1.3.3 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] =
'\'')[^'\'']*' <
http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/
ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg>
+ mock_root=epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8
+ [[ -n epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8
+ :
+ [[ -n '' ]]
+ find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0
+ xargs -0 -tr sudo rm -rf
++ virsh list --all --uuid
+ false
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report
files were found. Configuration error?
Archiving artifacts
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at