Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32

Hi Eyal, The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed. So it basically got stuck while extracting the OVF image. Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine. Thanks in advance, Lev Veyde. ------------------------------ *From: *"Eyal Edri" <eedri(a)redhat.com&gt; *To: *jenkins(a)jenkins.phx.ovirt.org *Cc: *"infra" <infra(a)ovirt.org&gt;, "Lev Veyde" <lveyde(a)redhat.com&gt;, sbonazzo(a)redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32 Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes> > > Changes: > > [Lev Veyde] ovirt-system-tests: Add automation for > he_iscsi_basic_suite_4.0 > > [Sandro Bonazzola] vdsm: avoid fc24 out of master > > [Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing > > ------------------------------------------ > [...truncated 620 lines...] > > WORKSPACE="$PWD" > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" > > rm -rf "$WORKSPACE/exported-artifacts" > mkdir -p "$WORKSPACE/exported-artifacts" > > if [[ -d "$TESTS_LOGS" ]]; then > mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" > fi > > [ovirt_4.0_he-system-tests] $ /bin/bash -xe > /tmp/hudson1764906258788527221.sh > + echo shell_scripts/system_tests.collect_logs.sh > shell_scripts/system_tests.collect_logs.sh > + VERSION=4.0 > + SUITE_TYPE= > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> > + OVIRT_SUITE=4.0 > + TESTS_LOGS=< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > > > + rm -rf < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/export... > > > + mkdir -p < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/export... > > > + [[ -d < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > ]] > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 0 > Match found for :.* : True > Logical operation result is TRUE > Running script : #!/bin/bash -xe > echo "shell-scripts/mock_cleanup.sh" > > shopt -s nullglob > > > WORKSPACE="$PWD" > > # Make clear this is the cleanup, helps reading the jenkins logs > cat <<EOC > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > EOC > > > # Archive the logs, we want them anyway > logs=( > ./*log > ./*/logs > ) > if [[ "$logs" ]]; then > tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > rm -rf "${logs[@]}" > fi > > # stop any processes running inside the chroot > failed=false > mock_confs=("$WORKSPACE"/*/mocker*) > # Clean current jobs mockroot if any > for mock_conf_file in "${mock_confs[@]}"; do > [[ "$mock_conf_file" ]] || continue > echo "Cleaning up mock $mock_conf" > mock_root="${mock_conf_file##*/}" > mock_root="${mock_root%.*}" > my_mock="/usr/bin/mock" > my_mock+=" --configdir=${mock_conf_file%/*}" > my_mock+=" --root=${mock_root}" > my_mock+=" --resultdir=$WORKSPACE" > > #TODO: investigate why mock --clean fails to umount certain dirs > sometimes, > #so we can use it instead of manually doing all this. > echo "Killing all mock orphan processes, if any." > $my_mock \ > --orphanskill \ > || { > echo "ERROR: Failed to kill orphans on $chroot." > failed=true > } > > mock_root="$(\ > grep \ > -Po "(?<=config_opts\['root'\] = ')[^']*" \ > "$mock_conf_file" \ > )" || : > [[ "$mock_root" ]] || continue > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $chroot. Trying to > umount." > fi > for mount in "${mounts[@]}"; do > sudo umount --lazy "$mount" \ > || { > echo "ERROR: Failed to umount $mount." > failed=true > } > done > done > > # Clean any leftover chroot from other jobs > for mock_root in /var/lib/mock/*; do > this_chroot_failed=false > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $mock_root." \ > "Trying to umount." > fi > for mount in "${mounts[@]}"; do > sudo umount --lazy "$mount" \ > || { > echo "ERROR: Failed to umount $mount." > failed=true > this_chroot_failed=true > } > done > if ! $this_chroot_failed; then > sudo rm -rf "$mock_root" > fi > done > > if $failed; then > echo "Aborting." > exit 1 > fi > > # remove mock system cache, we will setup proxies to do the caching and > this > # takes lots of space between runs > shopt -u nullglob > sudo rm -Rf /var/cache/mock/* > > # restore the permissions in the working dir, as sometimes it leaves files > # owned by root and then the 'cleanup workspace' from jenkins job fails to > # clean and breaks the jobs > sudo chown -R "$USER" "$WORKSPACE" > > [ovirt_4.0_he-system-tests] $ /bin/bash -xe > /tmp/hudson5198775129414653216.sh > + echo shell-scripts/mock_cleanup.sh > shell-scripts/mock_cleanup.sh > + shopt -s nullglob > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> > + cat > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > + logs=(./*log ./*/logs) > + [[ -n ./ovirt-system-tests/logs ]] > + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs > ./ovirt-system-tests/logs/ > ./ovirt-system-tests/logs/ > mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/ > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ > <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-... > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log > > ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log > + rm -rf ./ovirt-system-tests/logs > + failed=false > + mock_confs=("$WORKSPACE"/*/mocker*) > + for mock_conf_file in '"${mock_confs[@]}"' > + [[ -n < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > ]] > + echo 'Cleaning up mock ' > Cleaning up mock > + mock_root=mocker-fedora-23-x86_64.fc23.cfg > + mock_root=mocker-fedora-23-x86_64.fc23 > + my_mock=/usr/bin/mock > + my_mock+=' --configdir=< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests > '> > + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' > + my_mock+=' --resultdir=< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> > + echo 'Killing all mock orphan processes, if any.' > Killing all mock orphan processes, if any. > + /usr/bin/mock --configdir=< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > --root=mocker-fedora-23-x86_64.fc23 --resultdir=< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill > WARNING: Could not find required logging config file: < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > Using default... > INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... > Start: init plugins > INFO: selinux enabled > Finish: init plugins > Start: run > WARNING: Process ID 115551 still running in chroot. Killing... > WARNING: Process ID 115576 still running in chroot. Killing... > WARNING: Process ID 115577 still running in chroot. Killing... > WARNING: Process ID 115578 still running in chroot. Killing... > WARNING: Process ID 116634 still running in chroot. Killing... > Finish: run > ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te... > > > + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad > + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] > + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > ++ mount > ++ awk '{print $3}' > ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad > + [[ -n > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc > ]] > + echo 'Found mounted dirs inside the chroot . Trying to umount.' > Found mounted dirs inside the chroot . Trying to umount. > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys > sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests > > > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago > umount: > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: > not mounted > + echo 'ERROR: Failed to umount > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' > ERROR: Failed to umount > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. > + failed=true > + for mount in '"${mounts[@]}"' > + sudo umount --lazy > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems > umount: > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: > mountpoint not found > + echo 'ERROR: Failed to umount > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' > ERROR: Failed to umount > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. > + failed=true > + for mock_root in '/var/lib/mock/*' > + this_chroot_failed=false > + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > ++ mount > ++ awk '{print $3}' > ++ grep > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 > + : > + [[ -n '' ]] > + false > + sudo rm -rf > /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 > + true > + echo Aborting. > Aborting. > + exit 1 > POST BUILD TASK : FAILURE > END OF POST BUILD TASK : 1 > Recording test results > ERROR: Step ‘Publish JUnit test result report’ failed: No test report > files were found. Configuration error? > Archiving artifacts > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > >