Build failed in Jenkins: ovirt_3.6_system-tests #168

Nir Soffer nsoffer at redhat.com
Tue Jun 28 17:03:21 UTC 2016


This is not a place for sending automated error message.

On Tue, Jun 28, 2016 at 7:58 PM, Eyal Edri <eedri at redhat.com> wrote:
> Why not?
> If this is a real regression?
>
> On Jun 28, 2016 7:54 PM, "Nir Soffer" <nsoffer at redhat.com> wrote:
>>
>> I think this was sent by mistake to devel at ovirt.org, please do not
>> send such message to this list.
>>
>> On Tue, Jun 28, 2016 at 7:49 PM,  <jenkins at jenkins.phx.ovirt.org> wrote:
>> > See <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/changes>
>> >
>> > Changes:
>> >
>> > [Ondra Machacek] Deploy FreeIPA to storage machine
>> >
>> > ------------------------------------------
>> > [...truncated 554 lines...]
>> > ##      took 2194 seconds
>> > ##      rc = 1
>> > ##########################################################
>> > ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
>> > ##! Last 20 log enties:
>> > logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
>> > ##!
>> > @ Collect artifacts:
>> >   # ovirt-role metadata entry will be soon deprecated, instead you
>> > should use the vm-provider entry in the domain definiton and set it no one
>> > of: ovirt-node, ovirt-engine, ovirt-host
>> >   # [Thread-1] lago_basic_suite_3_6_engine:
>> >   # [Thread-2] lago_basic_suite_3_6_host1:
>> >   # [Thread-3] lago_basic_suite_3_6_host0:
>> >   # [Thread-4] lago_basic_suite_3_6_storage:
>> >   # [Thread-3] lago_basic_suite_3_6_host0: Success (in 0:00:16)
>> >   # [Thread-2] lago_basic_suite_3_6_host1: Success (in 0:00:16)
>> >   # [Thread-1] lago_basic_suite_3_6_engine: Success (in 0:00:16)
>> >   # [Thread-4] lago_basic_suite_3_6_storage: Success (in 0:00:16)
>> > @ Collect artifacts: Success (in 0:00:16)
>> >
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests>
>> > @@@@ ERROR: Failed running
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/basic_suite_3.6/test-scenarios/002_bootstrap.py>
>> > #########################
>> > ======== Cleaning up
>> > ----------- Cleaning with lago
>> > ----------- Cleaning with lago done
>> > ======== Cleanup done
>> > Took 2033 seconds
>> > ===================================
>> > ##!
>> > ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> > ##!########################################################
>> > ##########################################################
>> > Build step 'Execute shell' marked build as failure
>> > Performing Post build task...
>> > Match found for :.* : True
>> > Logical operation result is TRUE
>> > Running script  : #!/bin/bash -xe
>> > echo 'shell_scripts/system_tests.collect_logs.sh'
>> >
>> > #
>> > # Required jjb vars:
>> > #    version
>> > #
>> > VERSION=3.6
>> > SUITE_TYPE=
>> >
>> > WORKSPACE="$PWD"
>> > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
>> > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
>> >
>> > rm -rf "$WORKSPACE/exported-artifacts"
>> > mkdir -p "$WORKSPACE/exported-artifacts"
>> >
>> > if [[ -d "$TESTS_LOGS" ]]; then
>> >     mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
>> > fi
>> >
>> > [ovirt_3.6_system-tests] $ /bin/bash -xe
>> > /tmp/hudson3155110657872379488.sh
>> > + echo shell_scripts/system_tests.collect_logs.sh
>> > shell_scripts/system_tests.collect_logs.sh
>> > + VERSION=3.6
>> > + SUITE_TYPE=
>> > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
>> > + OVIRT_SUITE=3.6
>> > +
>> > TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts>
>> > + rm -rf
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported-artifacts>
>> > + mkdir -p
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported-artifacts>
>> > + [[ -d
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts>
>> > ]]
>> > + mv
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs>
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts/nosetests-001_initialize_engine.py.xml>
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts/nosetests-002_bootstrap.py.xml>
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/exported-artifacts/test_logs>
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported-artifacts/>
>> > POST BUILD TASK : SUCCESS
>> > END OF POST BUILD TASK : 0
>> > Match found for :.* : True
>> > Logical operation result is TRUE
>> > Running script  : #!/bin/bash -xe
>> > echo "shell-scripts/mock_cleanup.sh"
>> >
>> > shopt -s nullglob
>> >
>> >
>> > WORKSPACE="$PWD"
>> >
>> > # Make clear this is the cleanup, helps reading the jenkins logs
>> > cat <<EOC
>> > _______________________________________________________________________
>> > #######################################################################
>> > #                                                                     #
>> > #                               CLEANUP                               #
>> > #                                                                     #
>> > #######################################################################
>> > EOC
>> >
>> >
>> > # Archive the logs, we want them anyway
>> > logs=(
>> >     ./*log
>> >     ./*/logs
>> > )
>> > if [[ "$logs" ]]; then
>> >     tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
>> >     rm -rf "${logs[@]}"
>> > fi
>> >
>> > # stop any processes running inside the chroot
>> > failed=false
>> > mock_confs=("$WORKSPACE"/*/mocker*)
>> > # Clean current jobs mockroot if any
>> > for mock_conf_file in "${mock_confs[@]}"; do
>> >     [[ "$mock_conf_file" ]] || continue
>> >     echo "Cleaning up mock $mock_conf"
>> >     mock_root="${mock_conf_file##*/}"
>> >     mock_root="${mock_root%.*}"
>> >     my_mock="/usr/bin/mock"
>> >     my_mock+=" --configdir=${mock_conf_file%/*}"
>> >     my_mock+=" --root=${mock_root}"
>> >     my_mock+=" --resultdir=$WORKSPACE"
>> >
>> >     #TODO: investigate why mock --clean fails to umount certain dirs
>> > sometimes,
>> >     #so we can use it instead of manually doing all this.
>> >     echo "Killing all mock orphan processes, if any."
>> >     $my_mock \
>> >         --orphanskill \
>> >     || {
>> >         echo "ERROR:  Failed to kill orphans on $chroot."
>> >         failed=true
>> >     }
>> >
>> >     mock_root="$(\
>> >         grep \
>> >             -Po "(?<=config_opts\['root'\] = ')[^']*" \
>> >             "$mock_conf_file" \
>> >     )" || :
>> >     [[ "$mock_root" ]] || continue
>> >     mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>> >     if [[ "$mounts" ]]; then
>> >         echo "Found mounted dirs inside the chroot $chroot. Trying to
>> > umount."
>> >     fi
>> >     for mount in "${mounts[@]}"; do
>> >         sudo umount --lazy "$mount" \
>> >         || {
>> >             echo "ERROR:  Failed to umount $mount."
>> >             failed=true
>> >         }
>> >     done
>> > done
>> >
>> > # Clean any leftover chroot from other jobs
>> > for mock_root in /var/lib/mock/*; do
>> >     this_chroot_failed=false
>> >     mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>> >     if [[ "$mounts" ]]; then
>> >         echo "Found mounted dirs inside the chroot $mock_root." \
>> >              "Trying to umount."
>> >     fi
>> >     for mount in "${mounts[@]}"; do
>> >         sudo umount --lazy "$mount" \
>> >         || {
>> >             echo "ERROR:  Failed to umount $mount."
>> >             failed=true
>> >             this_chroot_failed=true
>> >         }
>> >     done
>> >     if ! $this_chroot_failed; then
>> >         sudo rm -rf "$mock_root"
>> >     fi
>> > done
>> >
>> > if $failed; then
>> >     echo "Aborting."
>> >     exit 1
>> > fi
>> >
>> > # remove mock system cache, we will setup proxies to do the caching and
>> > this
>> > # takes lots of space between runs
>> > shopt -u nullglob
>> > sudo rm -Rf /var/cache/mock/*
>> >
>> > # restore the permissions in the working dir, as sometimes it leaves
>> > files
>> > # owned by root and then the 'cleanup workspace' from jenkins job fails
>> > to
>> > # clean and breaks the jobs
>> > sudo chown -R "$USER" "$WORKSPACE"
>> >
>> > [ovirt_3.6_system-tests] $ /bin/bash -xe
>> > /tmp/hudson1974597178250713835.sh
>> > + echo shell-scripts/mock_cleanup.sh
>> > shell-scripts/mock_cleanup.sh
>> > + shopt -s nullglob
>> > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
>> > + cat
>> > _______________________________________________________________________
>> > #######################################################################
>> > #                                                                     #
>> > #                               CLEANUP                               #
>> > #                                                                     #
>> > #######################################################################
>> > + logs=(./*log ./*/logs)
>> > + [[ -n ./ovirt-system-tests/logs ]]
>> > + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
>> > ./ovirt-system-tests/logs/
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
>> >
>> > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
>> > + rm -rf ./ovirt-system-tests/logs
>> > + failed=false
>> > + mock_confs=("$WORKSPACE"/*/mocker*)
>> > + for mock_conf_file in '"${mock_confs[@]}"'
>> > + [[ -n
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg>
>> > ]]
>> > + echo 'Cleaning up mock '
>> > Cleaning up mock
>> > + mock_root=mocker-fedora-23-x86_64.fc23.cfg
>> > + mock_root=mocker-fedora-23-x86_64.fc23
>> > + my_mock=/usr/bin/mock
>> > + my_mock+='
>> > --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests'>
>> > + my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
>> > + my_mock+='
>> > --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/'>
>> > + echo 'Killing all mock orphan processes, if any.'
>> > Killing all mock orphan processes, if any.
>> > + /usr/bin/mock
>> > --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests>
>> > --root=mocker-fedora-23-x86_64.fc23
>> > --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
>> > --orphanskill
>> > WARNING: Could not find required logging config file:
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/logging.ini.>
>> > Using default...
>> > INFO: mock.py version 1.2.17 starting (python version = 3.4.3)...
>> > Start: init plugins
>> > INFO: selinux enabled
>> > Finish: init plugins
>> > Start: run
>> > Finish: run
>> > ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg>
>> > + mock_root=fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
>> > + [[ -n fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f ]]
>> > + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
>> > ++ mount
>> > ++ grep fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
>> > ++ awk '{print $3}'
>> > + :
>> > + [[ -n '' ]]
>> > + false
>> > + shopt -u nullglob
>> > + sudo rm -Rf
>> > /var/cache/mock/fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
>> > + sudo chown -R jenkins
>> > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
>> > POST BUILD TASK : SUCCESS
>> > END OF POST BUILD TASK : 1
>> > Recording test results
>> > Archiving artifacts
>> > _______________________________________________
>> > Infra mailing list
>> > Infra at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/infra
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>



More information about the Infra mailing list