Build failed in Jenkins: ovirt_4.0_he-system-tests #144
Eyal Edri
eedri at redhat.com
Mon Aug 15 08:29:07 UTC 2016
can we disable the add secondary storage domain test if this is what is
failing the test?
at least until the new sanlock will be released.
GA date is 13/09 and I'm guessing it will take time to reach centos also,
so we still have at least one month of this error possibly masking real
regressions.
On Mon, Aug 15, 2016 at 11:25 AM, <jenkins at jenkins.phx.ovirt.org> wrote:
> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/144/changes>
>
> Changes:
>
> [Gal Ben Haim] Removed the hard coded path of the template repo. The
> template repo file
>
> ------------------------------------------
> [...truncated 1273 lines...]
> ## rc = 1
> ##########################################################
> ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> ##! Last 20 log enties: logs/mocker-fedora-23-x86_64.
> fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log
> ##!
> + true
> + env_cleanup
> + echo '#########################'
> #########################
> + local res=0
> + local uuid
> + echo '======== Cleaning up'
> ======== Cleaning up
> + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/deployment-he_basic_suite_4.0> ]]
> + echo '----------- Cleaning with lago'
> ----------- Cleaning with lago
> + lago --workdir <http://jenkins.ovirt.org/job/
> ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-
> he_basic_suite_4.0> destroy --yes --all-prefixes
> + echo '----------- Cleaning with lago done'
> ----------- Cleaning with lago done
> + [[ 0 != \0 ]]
> + echo '======== Cleanup done'
> ======== Cleanup done
> + exit 0
> Took 2834 seconds
> ===================================
> ##!
> ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ##!########################################################
> ##########################################################
> Build step 'Execute shell' marked build as failure
> Performing Post build task...
> Match found for :.* : True
> Logical operation result is TRUE
> Running script : #!/bin/bash -xe
> echo 'shell_scripts/system_tests.collect_logs.sh'
>
> #
> # Required jjb vars:
> # version
> #
> VERSION=4.0
> SUITE_TYPE=
>
> WORKSPACE="$PWD"
> OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
> TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
>
> rm -rf "$WORKSPACE/exported-artifacts"
> mkdir -p "$WORKSPACE/exported-artifacts"
>
> if [[ -d "$TESTS_LOGS" ]]; then
> mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
> fi
>
> [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/
> hudson2053675743404291966.sh
> + echo shell_scripts/system_tests.collect_logs.sh
> shell_scripts/system_tests.collect_logs.sh
> + VERSION=4.0
> + SUITE_TYPE=
> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
> + OVIRT_SUITE=4.0
> + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
> system-tests/ws/ovirt-system-tests/exported-artifacts>
> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/144/
> artifact/exported-artifacts>
> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/144/
> artifact/exported-artifacts>
> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/exported-artifacts> ]]
> + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/exported-artifacts/lago_logs> <
> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/exported-artifacts/test_logs> <
> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/144/
> artifact/exported-artifacts/>
> POST BUILD TASK : SUCCESS
> END OF POST BUILD TASK : 0
> Match found for :.* : True
> Logical operation result is TRUE
> Running script : #!/bin/bash -xe
> echo "shell-scripts/mock_cleanup.sh"
>
> shopt -s nullglob
>
>
> WORKSPACE="$PWD"
>
> # Make clear this is the cleanup, helps reading the jenkins logs
> cat <<EOC
> _______________________________________________________________________
> #######################################################################
> # #
> # CLEANUP #
> # #
> #######################################################################
> EOC
>
>
> # Archive the logs, we want them anyway
> logs=(
> ./*log
> ./*/logs
> )
> if [[ "$logs" ]]; then
> tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
> rm -rf "${logs[@]}"
> fi
>
> # stop any processes running inside the chroot
> failed=false
> mock_confs=("$WORKSPACE"/*/mocker*)
> # Clean current jobs mockroot if any
> for mock_conf_file in "${mock_confs[@]}"; do
> [[ "$mock_conf_file" ]] || continue
> echo "Cleaning up mock $mock_conf"
> mock_root="${mock_conf_file##*/}"
> mock_root="${mock_root%.*}"
> my_mock="/usr/bin/mock"
> my_mock+=" --configdir=${mock_conf_file%/*}"
> my_mock+=" --root=${mock_root}"
> my_mock+=" --resultdir=$WORKSPACE"
>
> #TODO: investigate why mock --clean fails to umount certain dirs
> sometimes,
> #so we can use it instead of manually doing all this.
> echo "Killing all mock orphan processes, if any."
> $my_mock \
> --orphanskill \
> || {
> echo "ERROR: Failed to kill orphans on $chroot."
> failed=true
> }
>
> mock_root="$(\
> grep \
> -Po "(?<=config_opts\['root'\] = ')[^']*" \
> "$mock_conf_file" \
> )" || :
> [[ "$mock_root" ]] || continue
> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
> if [[ "$mounts" ]]; then
> echo "Found mounted dirs inside the chroot $chroot. Trying to
> umount."
> fi
> for mount in "${mounts[@]}"; do
> sudo umount --lazy "$mount" \
> || {
> echo "ERROR: Failed to umount $mount."
> failed=true
> }
> done
> done
>
> # Clean any leftover chroot from other jobs
> for mock_root in /var/lib/mock/*; do
> this_chroot_failed=false
> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
> if [[ "$mounts" ]]; then
> echo "Found mounted dirs inside the chroot $mock_root." \
> "Trying to umount."
> fi
> for mount in "${mounts[@]}"; do
> sudo umount --lazy "$mount" \
> || {
> echo "ERROR: Failed to umount $mount."
> failed=true
> this_chroot_failed=true
> }
> done
> if ! $this_chroot_failed; then
> sudo rm -rf "$mock_root"
> fi
> done
>
> if $failed; then
> echo "Aborting."
> exit 1
> fi
>
> # remove mock system cache, we will setup proxies to do the caching and
> this
> # takes lots of space between runs
> shopt -u nullglob
> sudo rm -Rf /var/cache/mock/*
>
> # restore the permissions in the working dir, as sometimes it leaves files
> # owned by root and then the 'cleanup workspace' from jenkins job fails to
> # clean and breaks the jobs
> sudo chown -R "$USER" "$WORKSPACE"
>
> [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson895710842739065954.
> sh
> + echo shell-scripts/mock_cleanup.sh
> shell-scripts/mock_cleanup.sh
> + shopt -s nullglob
> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
> + cat
> _______________________________________________________________________
> #######################################################################
> # #
> # CLEANUP #
> # #
> #######################################################################
> + logs=(./*log ./*/logs)
> + [[ -n ./ovirt-system-tests/logs ]]
> + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
> ./ovirt-system-tests/logs/
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> init/stdout_stderr.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> install_packages/stdout_stderr.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> install_packages/state.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> install_packages/build.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> install_packages/root.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> clean_rpmdb/stdout_stderr.log
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_
> basic_suite_4.0.sh/
> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.
> he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log
> + rm -rf ./ovirt-system-tests/logs
> + failed=false
> + mock_confs=("$WORKSPACE"/*/mocker*)
> + for mock_conf_file in '"${mock_confs[@]}"'
> + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]]
> + echo 'Cleaning up mock '
> Cleaning up mock
> + mock_root=mocker-fedora-23-x86_64.fc23.cfg
> + mock_root=mocker-fedora-23-x86_64.fc23
> + my_mock=/usr/bin/mock
> + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
> system-tests/ws/ovirt-system-tests'>
> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
> + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
> system-tests/ws/'>
> + echo 'Killing all mock orphan processes, if any.'
> Killing all mock orphan processes, if any.
> + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
> system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23
> --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
> --orphanskill
> WARNING: Could not find required logging config file: <
> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/logging.ini.> Using default...
> INFO: mock.py version 1.2.18 starting (python version = 3.5.1)...
> Start: init plugins
> INFO: selinux enabled
> Finish: init plugins
> Start: run
> Finish: run
> ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <
> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
> ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg>
> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
> + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]
> + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> ++ mount
> ++ awk '{print $3}'
> ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
> + :
> + [[ -n '' ]]
> + false
> + shopt -u nullglob
> + sudo rm -Rf /var/cache/mock/fedora-23-x86_64-
> 0c362156a2fa4a935ea8b988eb73b2ad
> + sudo chown -R jenkins <http://jenkins.ovirt.org/job/
> ovirt_4.0_he-system-tests/ws/>
> POST BUILD TASK : SUCCESS
> END OF POST BUILD TASK : 1
> Recording test results
> ERROR: Step ?Publish JUnit test result report? failed: No test report
> files were found. Configuration error?
> Archiving artifacts
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
>
--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160815/e16f5693/attachment.html>
More information about the Infra
mailing list