Re: Build failed in Jenkins: ovirt_3.6_system-tests #168

Why not? If this is a real regression? On Jun 28, 2016 7:54 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: > > I think this was sent by mistake to devel(a)ovirt.org, please do not > send such message to this list. > > On Tue, Jun 28, 2016 at 7:49 PM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > See <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/changes> > > > > Changes: > > > > [Ondra Machacek] Deploy FreeIPA to storage machine > > > > ------------------------------------------ > > [...truncated 554 lines...] > > ## took 2194 seconds > > ## rc = 1 > > ########################################################## > > ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > > ##! Last 20 log enties: > > logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log > > ##! > > @ Collect artifacts: > > # ovirt-role metadata entry will be soon deprecated, instead you > > should use the vm-provider entry in the domain definiton and set it no one > > of: ovirt-node, ovirt-engine, ovirt-host > > # [Thread-1] lago_basic_suite_3_6_engine: > > # [Thread-2] lago_basic_suite_3_6_host1: > > # [Thread-3] lago_basic_suite_3_6_host0: > > # [Thread-4] lago_basic_suite_3_6_storage: > > # [Thread-3] lago_basic_suite_3_6_host0: Success (in 0:00:16) > > # [Thread-2] lago_basic_suite_3_6_host1: Success (in 0:00:16) > > # [Thread-1] lago_basic_suite_3_6_engine: Success (in 0:00:16) > > # [Thread-4] lago_basic_suite_3_6_storage: Success (in 0:00:16) > > @ Collect artifacts: Success (in 0:00:16) > > > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > @@@@ ERROR: Failed running > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > ######################### > > ======== Cleaning up > > ----------- Cleaning with lago > > ----------- Cleaning with lago done > > ======== Cleanup done > > Took 2033 seconds > > =================================== > > ##! > > ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > ##!######################################################## > > ########################################################## > > Build step 'Execute shell' marked build as failure > > Performing Post build task... > > Match found for :.* : True > > Logical operation result is TRUE > > Running script : #!/bin/bash -xe > > echo 'shell_scripts/system_tests.collect_logs.sh' > > > > # > > # Required jjb vars: > > # version > > # > > VERSION=3.6 > > SUITE_TYPE= > > > > WORKSPACE="$PWD" > > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" > > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" > > > > rm -rf "$WORKSPACE/exported-artifacts" > > mkdir -p "$WORKSPACE/exported-artifacts" > > > > if [[ -d "$TESTS_LOGS" ]]; then > > mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" > > fi > > > > [ovirt_3.6_system-tests] $ /bin/bash -xe > > /tmp/hudson3155110657872379488.sh > > + echo shell_scripts/system_tests.collect_logs.sh > > shell_scripts/system_tests.collect_logs.sh > > + VERSION=3.6 > > + SUITE_TYPE= > > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> > > + OVIRT_SUITE=3.6 > > + > > TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovi... > > + rm -rf > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported... > > + mkdir -p > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported... > > + [[ -d > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > ]] > > + mv > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/168/artifact/exported... > > POST BUILD TASK : SUCCESS > > END OF POST BUILD TASK : 0 > > Match found for :.* : True > > Logical operation result is TRUE > > Running script : #!/bin/bash -xe > > echo "shell-scripts/mock_cleanup.sh" > > > > shopt -s nullglob > > > > > > WORKSPACE="$PWD" > > > > # Make clear this is the cleanup, helps reading the jenkins logs > > cat <<EOC > > _______________________________________________________________________ > > ####################################################################### > > # # > > # CLEANUP # > > # # > > ####################################################################### > > EOC > > > > > > # Archive the logs, we want them anyway > > logs=( > > ./*log > > ./*/logs > > ) > > if [[ "$logs" ]]; then > > tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > > rm -rf "${logs[@]}" > > fi > > > > # stop any processes running inside the chroot > > failed=false > > mock_confs=("$WORKSPACE"/*/mocker*) > > # Clean current jobs mockroot if any > > for mock_conf_file in "${mock_confs[@]}"; do > > [[ "$mock_conf_file" ]] || continue > > echo "Cleaning up mock $mock_conf" > > mock_root="${mock_conf_file##*/}" > > mock_root="${mock_root%.*}" > > my_mock="/usr/bin/mock" > > my_mock+=" --configdir=${mock_conf_file%/*}" > > my_mock+=" --root=${mock_root}" > > my_mock+=" --resultdir=$WORKSPACE" > > > > #TODO: investigate why mock --clean fails to umount certain dirs > > sometimes, > > #so we can use it instead of manually doing all this. > > echo "Killing all mock orphan processes, if any." > > $my_mock \ > > --orphanskill \ > > || { > > echo "ERROR: Failed to kill orphans on $chroot." > > failed=true > > } > > > > mock_root="$(\ > > grep \ > > -Po "(?<=config_opts\['root'\] = ')[^']*" \ > > "$mock_conf_file" \ > > )" || : > > [[ "$mock_root" ]] || continue > > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > > if [[ "$mounts" ]]; then > > echo "Found mounted dirs inside the chroot $chroot. Trying to > > umount." > > fi > > for mount in "${mounts[@]}"; do > > sudo umount --lazy "$mount" \ > > || { > > echo "ERROR: Failed to umount $mount." > > failed=true > > } > > done > > done > > > > # Clean any leftover chroot from other jobs > > for mock_root in /var/lib/mock/*; do > > this_chroot_failed=false > > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > > if [[ "$mounts" ]]; then > > echo "Found mounted dirs inside the chroot $mock_root." \ > > "Trying to umount." > > fi > > for mount in "${mounts[@]}"; do > > sudo umount --lazy "$mount" \ > > || { > > echo "ERROR: Failed to umount $mount." > > failed=true > > this_chroot_failed=true > > } > > done > > if ! $this_chroot_failed; then > > sudo rm -rf "$mock_root" > > fi > > done > > > > if $failed; then > > echo "Aborting." > > exit 1 > > fi > > > > # remove mock system cache, we will setup proxies to do the caching and > > this > > # takes lots of space between runs > > shopt -u nullglob > > sudo rm -Rf /var/cache/mock/* > > > > # restore the permissions in the working dir, as sometimes it leaves > > files > > # owned by root and then the 'cleanup workspace' from jenkins job fails > > to > > # clean and breaks the jobs > > sudo chown -R "$USER" "$WORKSPACE" > > > > [ovirt_3.6_system-tests] $ /bin/bash -xe > > /tmp/hudson1974597178250713835.sh > > + echo shell-scripts/mock_cleanup.sh > > shell-scripts/mock_cleanup.sh > > + shopt -s nullglob > > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> > > + cat > > _______________________________________________________________________ > > ####################################################################### > > # # > > # CLEANUP # > > # # > > ####################################################################### > > + logs=(./*log ./*/logs) > > + [[ -n ./ovirt-system-tests/logs ]] > > + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs > > ./ovirt-system-tests/logs/ > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/ > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ > > > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log > > + rm -rf ./ovirt-system-tests/logs > > + failed=false > > + mock_confs=("$WORKSPACE"/*/mocker*) > > + for mock_conf_file in '"${mock_confs[@]}"' > > + [[ -n > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > ]] > > + echo 'Cleaning up mock ' > > Cleaning up mock > > + mock_root=mocker-fedora-23-x86_64.fc23.cfg > > + mock_root=mocker-fedora-23-x86_64.fc23 > > + my_mock=/usr/bin/mock > > + my_mock+=' > > --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests'> > > + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' > > + my_mock+=' > > --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/'> > > + echo 'Killing all mock orphan processes, if any.' > > Killing all mock orphan processes, if any. > > + /usr/bin/mock > > --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests> > > --root=mocker-fedora-23-x86_64.fc23 > > --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> > > --orphanskill > > WARNING: Could not find required logging config file: > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > Using default... > > INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... > > Start: init plugins > > INFO: selinux enabled > > Finish: init plugins > > Start: run > > Finish: run > > ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests... > > + mock_root=fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f > > + [[ -n fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f ]] > > + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > > ++ mount > > ++ grep fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f > > ++ awk '{print $3}' > > + : > > + [[ -n '' ]] > > + false > > + shopt -u nullglob > > + sudo rm -Rf > > /var/cache/mock/fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f > > + sudo chown -R jenkins > > <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> > > POST BUILD TASK : SUCCESS > > END OF POST BUILD TASK : 1 > > Recording test results > > Archiving artifacts > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > >