Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Tolik Litovsky
tlitovsk at redhat.com
Tue Jul 19 08:27:17 UTC 2016
Is it only for master job ?
Or we have such repos for all branches ?
On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri at redhat.com> wrote:
> Ryan/Tolik,
> Can you build appliance only from tested engine repo [1] ? lets see how it
> affects the stability, next step will be to publish tested appliance after
> it runs Lago verification.
>
> [1]
> http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/
> (published only after ovirt-system-tests basic suite finish successfully)
>
>
> On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde at redhat.com> wrote:
>
>> Hi Eyal,
>>
>> The last failed run failed on:
>> *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a
>> few minutes depending on archive size)
>> *21:35:04* Build timed out (after 360 minutes). Marking the build as
>> failed.
>>
>> So it basically got stuck while extracting the OVF image.
>>
>> Some previous runs failed mostly on either:
>> a) broken ovirt-engine-appliance build
>> b) ovirt-engine-appliance missing from the yum repo
>>
>> We need to make sure that the process of building and publishing the
>> ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it
>> into the repo so that the build of the appliance can work, then publish it
>> to the repo as well.
>> This is extra important as the hosted-engine flow installation will
>> probably become the default one, and without synced ovirt appliance we
>> can't really test the changes in the engine.
>>
>> Thanks in advance,
>> Lev Veyde.
>>
>> ------------------------------
>> *From: *"Eyal Edri" <eedri at redhat.com>
>> *To: *jenkins at jenkins.phx.ovirt.org
>> *Cc: *"infra" <infra at ovirt.org>, "Lev Veyde" <lveyde at redhat.com>,
>> sbonazzo at redhat.com
>> *Sent: *Tuesday, July 19, 2016 8:26:22 AM
>> *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
>>
>>
>> Lev, this test is a bit flaky going from stable to failure quite often,
>> can you check what is causing it?
>> On Jul 19, 2016 12:35 AM, <jenkins at jenkins.phx.ovirt.org> wrote:
>>
>>> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
>>>
>>> Changes:
>>>
>>> [Lev Veyde] ovirt-system-tests: Add automation for
>>> he_iscsi_basic_suite_4.0
>>>
>>> [Sandro Bonazzola] vdsm: avoid fc24 out of master
>>>
>>> [Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
>>>
>>> ------------------------------------------
>>> [...truncated 620 lines...]
>>>
>>> WORKSPACE="$PWD"
>>> OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
>>> TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
>>>
>>> rm -rf "$WORKSPACE/exported-artifacts"
>>> mkdir -p "$WORKSPACE/exported-artifacts"
>>>
>>> if [[ -d "$TESTS_LOGS" ]]; then
>>> mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
>>> fi
>>>
>>> [ovirt_4.0_he-system-tests] $ /bin/bash -xe
>>> /tmp/hudson1764906258788527221.sh
>>> + echo shell_scripts/system_tests.collect_logs.sh
>>> shell_scripts/system_tests.collect_logs.sh
>>> + VERSION=4.0
>>> + SUITE_TYPE=
>>> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>>> + OVIRT_SUITE=4.0
>>> + TESTS_LOGS=<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts
>>> >
>>> + rm -rf <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-artifacts
>>> >
>>> + mkdir -p <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-artifacts
>>> >
>>> + [[ -d <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts>
>>> ]]
>>> POST BUILD TASK : SUCCESS
>>> END OF POST BUILD TASK : 0
>>> Match found for :.* : True
>>> Logical operation result is TRUE
>>> Running script : #!/bin/bash -xe
>>> echo "shell-scripts/mock_cleanup.sh"
>>>
>>> shopt -s nullglob
>>>
>>>
>>> WORKSPACE="$PWD"
>>>
>>> # Make clear this is the cleanup, helps reading the jenkins logs
>>> cat <<EOC
>>> _______________________________________________________________________
>>> #######################################################################
>>> # #
>>> # CLEANUP #
>>> # #
>>> #######################################################################
>>> EOC
>>>
>>>
>>> # Archive the logs, we want them anyway
>>> logs=(
>>> ./*log
>>> ./*/logs
>>> )
>>> if [[ "$logs" ]]; then
>>> tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
>>> rm -rf "${logs[@]}"
>>> fi
>>>
>>> # stop any processes running inside the chroot
>>> failed=false
>>> mock_confs=("$WORKSPACE"/*/mocker*)
>>> # Clean current jobs mockroot if any
>>> for mock_conf_file in "${mock_confs[@]}"; do
>>> [[ "$mock_conf_file" ]] || continue
>>> echo "Cleaning up mock $mock_conf"
>>> mock_root="${mock_conf_file##*/}"
>>> mock_root="${mock_root%.*}"
>>> my_mock="/usr/bin/mock"
>>> my_mock+=" --configdir=${mock_conf_file%/*}"
>>> my_mock+=" --root=${mock_root}"
>>> my_mock+=" --resultdir=$WORKSPACE"
>>>
>>> #TODO: investigate why mock --clean fails to umount certain dirs
>>> sometimes,
>>> #so we can use it instead of manually doing all this.
>>> echo "Killing all mock orphan processes, if any."
>>> $my_mock \
>>> --orphanskill \
>>> || {
>>> echo "ERROR: Failed to kill orphans on $chroot."
>>> failed=true
>>> }
>>>
>>> mock_root="$(\
>>> grep \
>>> -Po "(?<=config_opts\['root'\] = ')[^']*" \
>>> "$mock_conf_file" \
>>> )" || :
>>> [[ "$mock_root" ]] || continue
>>> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>>> if [[ "$mounts" ]]; then
>>> echo "Found mounted dirs inside the chroot $chroot. Trying to
>>> umount."
>>> fi
>>> for mount in "${mounts[@]}"; do
>>> sudo umount --lazy "$mount" \
>>> || {
>>> echo "ERROR: Failed to umount $mount."
>>> failed=true
>>> }
>>> done
>>> done
>>>
>>> # Clean any leftover chroot from other jobs
>>> for mock_root in /var/lib/mock/*; do
>>> this_chroot_failed=false
>>> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>>> if [[ "$mounts" ]]; then
>>> echo "Found mounted dirs inside the chroot $mock_root." \
>>> "Trying to umount."
>>> fi
>>> for mount in "${mounts[@]}"; do
>>> sudo umount --lazy "$mount" \
>>> || {
>>> echo "ERROR: Failed to umount $mount."
>>> failed=true
>>> this_chroot_failed=true
>>> }
>>> done
>>> if ! $this_chroot_failed; then
>>> sudo rm -rf "$mock_root"
>>> fi
>>> done
>>>
>>> if $failed; then
>>> echo "Aborting."
>>> exit 1
>>> fi
>>>
>>> # remove mock system cache, we will setup proxies to do the caching and
>>> this
>>> # takes lots of space between runs
>>> shopt -u nullglob
>>> sudo rm -Rf /var/cache/mock/*
>>>
>>> # restore the permissions in the working dir, as sometimes it leaves
>>> files
>>> # owned by root and then the 'cleanup workspace' from jenkins job fails
>>> to
>>> # clean and breaks the jobs
>>> sudo chown -R "$USER" "$WORKSPACE"
>>>
>>> [ovirt_4.0_he-system-tests] $ /bin/bash -xe
>>> /tmp/hudson5198775129414653216.sh
>>> + echo shell-scripts/mock_cleanup.sh
>>> shell-scripts/mock_cleanup.sh
>>> + shopt -s nullglob
>>> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>>> + cat
>>> _______________________________________________________________________
>>> #######################################################################
>>> # #
>>> # CLEANUP #
>>> # #
>>> #######################################################################
>>> + logs=(./*log ./*/logs)
>>> + [[ -n ./ovirt-system-tests/logs ]]
>>> + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
>>> ./ovirt-system-tests/logs/
>>> ./ovirt-system-tests/logs/
>>> mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
>>> <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
>>>
>>> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log
>>> + rm -rf ./ovirt-system-tests/logs
>>> + failed=false
>>> + mock_confs=("$WORKSPACE"/*/mocker*)
>>> + for mock_conf_file in '"${mock_confs[@]}"'
>>> + [[ -n <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg>
>>> ]]
>>> + echo 'Cleaning up mock '
>>> Cleaning up mock
>>> + mock_root=mocker-fedora-23-x86_64.fc23.cfg
>>> + mock_root=mocker-fedora-23-x86_64.fc23
>>> + my_mock=/usr/bin/mock
>>> + my_mock+=' --configdir=<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
>>> '>
>>> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
>>> + my_mock+=' --resultdir=<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'>
>>> + echo 'Killing all mock orphan processes, if any.'
>>> Killing all mock orphan processes, if any.
>>> + /usr/bin/mock --configdir=<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests>
>>> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>>> --orphanskill
>>> WARNING: Could not find required logging config file: <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.>
>>> Using default...
>>> INFO: mock.py version 1.2.17 starting (python version = 3.4.3)...
>>> Start: init plugins
>>> INFO: selinux enabled
>>> Finish: init plugins
>>> Start: run
>>> WARNING: Process ID 115551 still running in chroot. Killing...
>>> WARNING: Process ID 115576 still running in chroot. Killing...
>>> WARNING: Process ID 115577 still running in chroot. Killing...
>>> WARNING: Process ID 115578 still running in chroot. Killing...
>>> WARNING: Process ID 116634 still running in chroot. Killing...
>>> Finish: run
>>> ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg
>>> >
>>> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
>>> + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]
>>> + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
>>> ++ mount
>>> ++ awk '{print $3}'
>>> ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
>>> + [[ -n
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc
>>> ]]
>>> + echo 'Found mounted dirs inside the chroot . Trying to umount.'
>>> Found mounted dirs inside the chroot . Trying to umount.
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys
>>> sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root<
>>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
>>> >
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago
>>> umount:
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago:
>>> not mounted
>>> + echo 'ERROR: Failed to umount
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'
>>> ERROR: Failed to umount
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.
>>> + failed=true
>>> + for mount in '"${mounts[@]}"'
>>> + sudo umount --lazy
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems
>>> umount:
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems:
>>> mountpoint not found
>>> + echo 'ERROR: Failed to umount
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'
>>> ERROR: Failed to umount
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.
>>> + failed=true
>>> + for mock_root in '/var/lib/mock/*'
>>> + this_chroot_failed=false
>>> + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
>>> ++ mount
>>> ++ awk '{print $3}'
>>> ++ grep
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644
>>> + :
>>> + [[ -n '' ]]
>>> + false
>>> + sudo rm -rf
>>> /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644
>>> + true
>>> + echo Aborting.
>>> Aborting.
>>> + exit 1
>>> POST BUILD TASK : FAILURE
>>> END OF POST BUILD TASK : 1
>>> Recording test results
>>> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
>>> files were found. Configuration error?
>>> Archiving artifacts
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>
>>>
>>>
>>
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
--
Best regards
Tolik Litovsky
RHEV-H Team
Red Hat
Red Hat: trustworthy, transformative technology. Powered by the community.
Connect at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160719/73fba1e7/attachment.html>
More information about the Infra
mailing list