Jenkins build became unstable: ovirt_4.0_he-system-tests #28
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/29/changes> Changes: [Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0 ------------------------------------------ [...truncated 1165 lines...] ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ##! + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 Took 2534 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5121418818499878205.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/29/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/29/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/test_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/29/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson987503546056593632.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.18 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ++ awk '{print $3}' + : + [[ -n '' ]] + false + shopt -u nullglob + sudo rm -Rf /var/cache/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/30/changes> Changes: [Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0 ------------------------------------------ [...truncated 1169 lines...] ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ##! + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 Took 3051 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson7193842952153957552.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/30/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/30/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/test_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/30/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1655441941351082741.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.18 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + : + [[ -n '' ]] + false + shopt -u nullglob + sudo rm -Rf /var/cache/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/31/changes> Changes: [Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0 [Sharon Naftaly] Fix wrong jenkins repo conf in coverity job ------------------------------------------ [...truncated 1159 lines...] ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ##! + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_4.0> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 Took 2948 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson6017133605164751950.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/31/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/31/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts/test_logs> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/31/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson6968928070349882997.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.18 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + : + [[ -n '' ]] + false + shopt -u nullglob + sudo rm -Rf /var/cache/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes> Changes: [Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0 [Sandro Bonazzola] vdsm: avoid fc24 out of master [Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing ------------------------------------------ [...truncated 620 lines...] WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
------=_Part_9639957_1092846933.1468912250886
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi Eyal,=20
The last failed run failed on:=20
15:50:02 [ INFO ] Extracting disk image from OVF archive (could take a few =
minutes depending on archive size)=20
21:35:04 Build timed out (after 360 minutes). Marking the build as failed.=
=20
So it basically got stuck while extracting the OVF image.=20
Some previous runs failed mostly on either:=20
a) broken ovirt-engine-appliance build=20
b) ovirt-engine-appliance missing from the yum repo=20
We need to make sure that the process of building and publishing the ovirt-=
engine-appliance works flawlessly e.g. build ovirt-engine, publish it into =
the repo so that the build of the appliance can work, then publish it to th=
e repo as well.=20
This is extra important as the hosted-engine flow installation will probabl=
y become the default one, and without synced ovirt appliance we can't reall=
y test the changes in the engine.=20
Thanks in advance,=20
Lev Veyde.=20
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com>=20
To: jenkins@jenkins.phx.ovirt.org=20
Cc: "infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@re=
dhat.com=20
Sent: Tuesday, July 19, 2016 8:26:22 AM=20
Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32=20
Lev, this test is a bit flaky going from stable to failure quite often, can=
you check what is causing it?=20
On Jul 19, 2016 12:35 AM, < jenkins@jenkins.phx.ovirt.org > wrote:=20
See < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes >=
=20
Changes:=20
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0=
=20
[Sandro Bonazzola] vdsm: avoid fc24 out of master=20
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing=20
------------------------------------------=20
[...truncated 620 lines...]=20
WORKSPACE=3D"$PWD"=20
OVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"=20
TESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"=20
rm -rf "$WORKSPACE/exported-artifacts"=20
mkdir -p "$WORKSPACE/exported-artifacts"=20
if [[ -d "$TESTS_LOGS" ]]; then=20
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"=20
fi=20
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.=
sh=20
+ echo shell_scripts/ system_tests.collect_logs.sh=20
shell_scripts/ system_tests.collect_logs.sh=20
+ VERSION=3D4.0=20
+ SUITE_TYPE=3D=20
+ WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ =
>=20
+ OVIRT_SUITE=3D4.0=20
+ TESTS_LOGS=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/=
ovirt-system-tests/exported-artifacts >=20
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifa=
ct/exported-artifacts >=20
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/arti=
fact/exported-artifacts >=20
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s=
ystem-tests/exported-artifacts > ]]=20
POST BUILD TASK : SUCCESS=20
END OF POST BUILD TASK : 0=20
Match found for :.* : True=20
Logical operation result is TRUE=20
Running script : #!/bin/bash -xe=20
echo "shell-scripts/mock_cleanup.sh"=20
shopt -s nullglob=20
WORKSPACE=3D"$PWD"=20
# Make clear this is the cleanup, helps reading the jenkins logs=20
cat <<EOC=20
_______________________________________________________________________=20
#######################################################################=20
# #=20
# CLEANUP #=20
# #=20
#######################################################################=20
EOC=20
# Archive the logs, we want them anyway=20
logs=3D(=20
./*log=20
./*/logs=20
)=20
if [[ "$logs" ]]; then=20
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"=20
rm -rf "${logs[@]}"=20
fi=20
# stop any processes running inside the chroot=20
failed=3Dfalse=20
mock_confs=3D("$WORKSPACE"/*/mocker*)=20
# Clean current jobs mockroot if any=20
for mock_conf_file in "${mock_confs[@]}"; do=20
[[ "$mock_conf_file" ]] || continue=20
echo "Cleaning up mock $mock_conf"=20
mock_root=3D"${mock_conf_file##*/}"=20
mock_root=3D"${mock_root%.*}"=20
my_mock=3D"/usr/bin/mock"=20
my_mock+=3D" --configdir=3D${mock_conf_file%/*}"=20
my_mock+=3D" --root=3D${mock_root}"=20
my_mock+=3D" --resultdir=3D$WORKSPACE"=20
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,=
=20
#so we can use it instead of manually doing all this.=20
echo "Killing all mock orphan processes, if any."=20
$my_mock \=20
--orphanskill \=20
|| {=20
echo "ERROR: Failed to kill orphans on $chroot."=20
failed=3Dtrue=20
}=20
mock_root=3D"$(\=20
grep \=20
-Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*" \=20
"$mock_conf_file" \=20
)" || :=20
[[ "$mock_root" ]] || continue=20
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20
if [[ "$mounts" ]]; then=20
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."=20
fi=20
for mount in "${mounts[@]}"; do=20
sudo umount --lazy "$mount" \=20
|| {=20
echo "ERROR: Failed to umount $mount."=20
failed=3Dtrue=20
}=20
done=20
done=20
# Clean any leftover chroot from other jobs=20
for mock_root in /var/lib/mock/*; do=20
this_chroot_failed=3Dfalse=20
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20
if [[ "$mounts" ]]; then=20
echo "Found mounted dirs inside the chroot $mock_root." \=20
"Trying to umount."=20
fi=20
for mount in "${mounts[@]}"; do=20
sudo umount --lazy "$mount" \=20
|| {=20
echo "ERROR: Failed to umount $mount."=20
failed=3Dtrue=20
this_chroot_failed=3Dtrue=20
}=20
done=20
if ! $this_chroot_failed; then=20
sudo rm -rf "$mock_root"=20
fi=20
done=20
if $failed; then=20
echo "Aborting."=20
exit 1=20
fi=20
# remove mock system cache, we will setup proxies to do the caching and thi=
s=20
# takes lots of space between runs=20
shopt -u nullglob=20
sudo rm -Rf /var/cache/mock/*=20
# restore the permissions in the working dir, as sometimes it leaves files=
=20
# owned by root and then the 'cleanup workspace' from jenkins job fails to=
=20
# clean and breaks the jobs=20
sudo chown -R "$USER" "$WORKSPACE"=20
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.=
sh=20
+ echo shell-scripts/mock_cleanup.sh=20
shell-scripts/mock_cleanup.sh=20
+ shopt -s nullglob=20
+ WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ =
>=20
+ cat=20
_______________________________________________________________________=20
#######################################################################=20
# #=20
# CLEANUP #=20
# #=20
#######################################################################=20
+ logs=3D(./*log ./*/logs)=20
+ [[ -n ./ovirt-system-tests/logs ]]=20
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs=20
./ovirt-system-tests/logs/=20
./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.=
sh/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l=
og=20
+ rm -rf ./ovirt-system-tests/logs=20
+ failed=3Dfalse=20
+ mock_confs=3D("$WORKSPACE"/*/mocker*)=20
+ for mock_conf_file in '"${mock_confs[@]}"'=20
+ [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s=
ystem-tests/mocker-fedora-23-x86_64.fc23.cfg > ]]=20
+ echo 'Cleaning up mock '=20
Cleaning up mock=20
+ mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg=20
+ mock_root=3Dmocker-fedora-23-x86_64.fc23=20
+ my_mock=3D/usr/bin/mock=20
+ my_mock+=3D' --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy=
stem-tests/ws/ovirt-system-tests '>=20
+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'=20
+ my_mock+=3D' --resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy=
stem-tests/ws/ '>=20
+ echo 'Killing all mock orphan processes, if any.'=20
Killing all mock orphan processes, if any.=20
+ /usr/bin/mock --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-s=
ystem-tests/ws/ovirt-system-tests > --root=3Dmocker-fedora-23-x86_64.fc23 -=
-resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >=
--orphanskill=20
WARNING: Could not find required logging config file: < http://jenkins.ovir=
t.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini .> Us=
ing default...=20
INFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...=20
Start: init plugins=20
INFO: selinux enabled=20
Finish: init plugins=20
Start: run=20
WARNING: Process ID 115551 still running in chroot. Killing...=20
WARNING: Process ID 115576 still running in chroot. Killing...=20
WARNING: Process ID 115577 still running in chroot. Killing...=20
WARNING: Process ID 115578 still running in chroot. Killing...=20
WARNING: Process ID 116634 still running in chroot. Killing...=20
Finish: run=20
++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' < http://=
jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocke=
r-fedora-23-x86_64.fc23.cfg >=20
+ mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20
+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]=20
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20
++ mount=20
++ awk '{print $3}'=20
++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20
+ [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc ]]=20
+ echo 'Found mounted dirs inside the chroot . Trying to umount.'=20
Found mounted dirs inside the chroot . Trying to umount.=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/sys=20
sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/shm=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/pts=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/cache/yum=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-test=
s/ws/ovirt-system-tests >=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/run/libvirt=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/lib/lago=20
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/var/lib/lago: not mounted=20
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa=
4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'=20
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8=
b988eb73b2ad-91644/root/var/lib/lago.=20
+ failed=3Dtrue=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc/filesystems=20
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc/filesystems: mountpoint not found=20
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa=
4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'=20
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8=
b988eb73b2ad-91644/root/proc/filesystems.=20
+ failed=3Dtrue=20
+ for mock_root in '/var/lib/mock/*'=20
+ this_chroot_failed=3Dfalse=20
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20
++ mount=20
++ awk '{print $3}'=20
++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44=20
+ :=20
+ [[ -n '' ]]=20
+ false=20
+ sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2=
ad-91644=20
+ true=20
+ echo Aborting.=20
Aborting.=20
+ exit 1=20
POST BUILD TASK : FAILURE=20
END OF POST BUILD TASK : 1=20
Recording test results=20
ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t=
est report files were found. Configuration error?=20
Archiving artifacts=20
_______________________________________________=20
Infra mailing list=20
Infra@ovirt.org=20
http://lists.ovirt.org/mailman/listinfo/infra=20
------=_Part_9639957_1092846933.1468912250886
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: times new roman, new york, times, se=
rif; font-size: 12pt; color: #000000"><div>Hi Eyal,<br></div><div><br></div=
><div>The last failed run failed on: </div><div><span class=3D"timestamp"><=
b>15:50:02</b> </span>[ INFO ] Extracting disk image from OVF archive (coul=
d take a few minutes depending on archive size)</div><div><span class=3D"ti=
mestamp"><b>21:35:04</b> </span>Build timed out (after 360 minutes). Markin=
g the build as failed.<br></div><div><br></div><div>So it basically got stu=
ck while extracting the OVF image.<br></div><div><br></div><div>Some previo=
us runs failed mostly on either:<br></div><div>a) broken ovirt-engine-appli=
ance build<br></div><div>b) ovirt-engine-appliance missing from the yum rep=
o<br></div><div><br></div><div>We need to make sure that the process of bui=
lding and publishing the ovirt-engine-appliance works flawlessly e.g. build=
ovirt-engine, publish it into the repo so that the build of the appliance =
can work, then publish it to the repo as well.<br></div><div>This is extra =
important as the hosted-engine flow installation will probably become the d=
efault one, and without synced ovirt appliance we can't really test the cha=
nges in the engine.<br></div><div><br></div><div><span name=3D"x"></span>Th=
anks in advance,<br>Lev Veyde.<span name=3D"x"></span><br></div><div><br></=
div><hr id=3D"zwchr"><div style=3D"color:#000;font-weight:normal;font-style=
:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-si=
ze:12pt;"><b>From: </b>"Eyal Edri" <eedri@redhat.com><br><b>To: </b>j=
enkins@jenkins.phx.ovirt.org<br><b>Cc: </b>"infra" <infra@ovirt.org>,=
"Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com<br><b>Sent: </b=
>Tuesday, July 19, 2016 8:26:22 AM<br><b>Subject: </b>Re: Build failed in J=
enkins: ovirt_4.0_he-system-tests #32<br><div><br></div><p dir=3D"ltr">Lev,=
this test is a bit flaky going from stable to failure quite often,&n=
bsp; can you check what is causing it? </p>
<div class=3D"gmail_quote">On Jul 19, 2016 12:35 AM, <<a href=3D"mailto=
:jenkins@jenkins.phx.ovirt.org" target=3D"_blank">jenkins@jenkins.phx.ovirt=
.org</a>> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">See <<a href=3D"ht=
tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes" rel=3D"nor=
eferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system=
-tests/32/changes</a>><br>
<br>
Changes:<br>
<br>
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0=
<br>
<br>
[Sandro Bonazzola] vdsm: avoid fc24 out of master<br>
<br>
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing<br>
<br>
------------------------------------------<br>
[...truncated 620 lines...]<br>
<br>
WORKSPACE=3D"$PWD"<br>
OVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"<br>
TESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"<br>
<br>
rm -rf "$WORKSPACE/exported-artifacts"<br>
mkdir -p "$WORKSPACE/exported-artifacts"<br>
<br>
if [[ -d "$TESTS_LOGS" ]]; then<br>
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"<br>
fi<br>
<br>
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.=
sh<br>
+ echo shell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D=
"noreferrer" target=3D"_blank">system_tests.collect_logs.sh</a><br>
shell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D"norefe=
rrer" target=3D"_blank">system_tests.collect_logs.sh</a><br>
+ VERSION=3D4.0<br>
+ SUITE_TYPE=3D<br>
+ WORKSPACE=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst=
em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org=
/job/ovirt_4.0_he-system-tests/ws/</a>><br>
+ OVIRT_SUITE=3D4.0<br>
+ TESTS_LOGS=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-sys=
tem-tests/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" targ=
et=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovi=
rt-system-tests/exported-artifacts</a>><br>
+ rm -rf <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-te=
sts/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">ht=
tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-a=
rtifacts</a>><br>
+ mkdir -p <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-=
tests/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">=
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported=
-artifacts</a>><br>
+ [[ -d <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes=
ts/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" target=3D"_=
blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-syst=
em-tests/exported-artifacts</a>> ]]<br>
POST BUILD TASK : SUCCESS<br>
END OF POST BUILD TASK : 0<br>
Match found for :.* : True<br>
Logical operation result is TRUE<br>
Running script : #!/bin/bash -xe<br>
echo "shell-scripts/mock_cleanup.sh"<br>
<br>
shopt -s nullglob<br>
<br>
<br>
WORKSPACE=3D"$PWD"<br>
<br>
# Make clear this is the cleanup, helps reading the jenkins logs<br>
cat <<EOC<br>
_______________________________________________________________________<br>
#######################################################################<br>
# &nbs=
p; &n=
bsp; =
#<br>
# &nbs=
p; CLEANUP &nb=
sp; &=
nbsp;#<br>
# &nbs=
p; &n=
bsp; =
#<br>
#######################################################################<br>
EOC<br>
<br>
<br>
# Archive the logs, we want them anyway<br>
logs=3D(<br>
./*log<br>
./*/logs<br>
)<br>
if [[ "$logs" ]]; then<br>
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"<br>
rm -rf "${logs[@]}"<br>
fi<br>
<br>
# stop any processes running inside the chroot<br>
failed=3Dfalse<br>
mock_confs=3D("$WORKSPACE"/*/mocker*)<br>
# Clean current jobs mockroot if any<br>
for mock_conf_file in "${mock_confs[@]}"; do<br>
[[ "$mock_conf_file" ]] || continue<br>
echo "Cleaning up mock $mock_conf"<br>
mock_root=3D"${mock_conf_file##*/}"<br>
mock_root=3D"${mock_root%.*}"<br>
my_mock=3D"/usr/bin/mock"<br>
my_mock+=3D" --configdir=3D${mock_conf_file%/*}"<br>
my_mock+=3D" --root=3D${mock_root}"<br>
my_mock+=3D" --resultdir=3D$WORKSPACE"<br>
<br>
#TODO: investigate why mock --clean fails to umount certain d=
irs sometimes,<br>
#so we can use it instead of manually doing all this.<br>
echo "Killing all mock orphan processes, if any."<br>
$my_mock \<br>
--orphanskill \<br>
|| {<br>
echo "ERROR: Failed to kill orphans on $c=
hroot."<br>
failed=3Dtrue<br>
}<br>
<br>
mock_root=3D"$(\<br>
grep \<br>
-Po "(?<=3Dconfig_opts\['root'=
\] =3D ')[^']*" \<br>
"$mock_conf_file" \<br>
)" || :<br>
[[ "$mock_root" ]] || continue<br>
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||=
:<br>
if [[ "$mounts" ]]; then<br>
echo "Found mounted dirs inside the chroot $chr=
oot. Trying to umount."<br>
fi<br>
for mount in "${mounts[@]}"; do<br>
sudo umount --lazy "$mount" \<br>
|| {<br>
echo "ERROR: Failed to umou=
nt $mount."<br>
failed=3Dtrue<br>
}<br>
done<br>
done<br>
<br>
# Clean any leftover chroot from other jobs<br>
for mock_root in /var/lib/mock/*; do<br>
this_chroot_failed=3Dfalse<br>
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||=
:<br>
if [[ "$mounts" ]]; then<br>
echo "Found mounted dirs inside the chroot $moc=
k_root." \<br>
"Trying to umount."<br>
fi<br>
for mount in "${mounts[@]}"; do<br>
sudo umount --lazy "$mount" \<br>
|| {<br>
echo "ERROR: Failed to umou=
nt $mount."<br>
failed=3Dtrue<br>
this_chroot_failed=3Dtrue<br>
}<br>
done<br>
if ! $this_chroot_failed; then<br>
sudo rm -rf "$mock_root"<br>
fi<br>
done<br>
<br>
if $failed; then<br>
echo "Aborting."<br>
exit 1<br>
fi<br>
<br>
# remove mock system cache, we will setup proxies to do the caching and thi=
s<br>
# takes lots of space between runs<br>
shopt -u nullglob<br>
sudo rm -Rf /var/cache/mock/*<br>
<br>
# restore the permissions in the working dir, as sometimes it leaves files<=
br>
# owned by root and then the 'cleanup workspace' from jenkins job fails to<=
br>
# clean and breaks the jobs<br>
sudo chown -R "$USER" "$WORKSPACE"<br>
<br>
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.=
sh<br>
+ echo shell-scripts/mock_cleanup.sh<br>
shell-scripts/mock_cleanup.sh<br>
+ shopt -s nullglob<br>
+ WORKSPACE=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst=
em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org=
/job/ovirt_4.0_he-system-tests/ws/</a>><br>
+ cat<br>
_______________________________________________________________________<br>
#######################################################################<br>
# &nbs=
p; &n=
bsp; =
#<br>
# &nbs=
p; CLEANUP &nb=
sp; &=
nbsp;#<br>
# &nbs=
p; &n=
bsp; =
#<br>
#######################################################################<br>
+ logs=3D(./*log ./*/logs)<br>
+ [[ -n ./ovirt-system-tests/logs ]]<br>
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs<br>
./ovirt-system-tests/logs/<br>
./ovirt-system-tests/logs/<a href=3D"http://mocker-fedora-23-x86_64.fc23.he=
_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/" rel=3D"norefe=
rrer" target=3D"_blank">mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/=
<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/<br=
>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/</a><br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l=
og<br>
+ rm -rf ./ovirt-system-tests/logs<br>
+ failed=3Dfalse<br>
+ mock_confs=3D("$WORKSPACE"/*/mocker*)<br>
+ for mock_conf_file in '"${mock_confs[@]}"'<br>
+ [[ -n <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes=
ts/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferre=
r" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests=
/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg</a>> ]]<br>
+ echo 'Cleaning up mock '<br>
Cleaning up mock<br>
+ mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg<br>
+ mock_root=3Dmocker-fedora-23-x86_64.fc23<br>
+ my_mock=3D/usr/bin/mock<br>
+ my_mock+=3D' --configdir=3D<<a href=3D"http://jenkins.ovirt.org/job/ov=
irt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D=
"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-sy=
stem-tests</a>'><br>
+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'<br>
+ my_mock+=3D' --resultdir=3D<<a href=3D"http://jenkins.ovirt.org/job/ov=
irt_4.0_he-system-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://je=
nkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/</a>'><br>
+ echo 'Killing all mock orphan processes, if any.'<br>
Killing all mock orphan processes, if any.<br>
+ /usr/bin/mock --configdir=3D<<a href=3D"http://jenkins.ovirt.org/job/o=
virt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=
=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt=
-system-tests</a>> --root=3Dmocker-fedora-23-x86_64.fc23 --resultdir=3D&=
lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/" r=
el=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_=
he-system-tests/ws/</a>> --orphanskill<br>
WARNING: Could not find required logging config file: <<a href=3D"http:/=
/jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logg=
ing.ini" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/=
ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini</a>.> Using =
default...<br>
INFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...<br>
Start: init plugins<br>
INFO: selinux enabled<br>
Finish: init plugins<br>
Start: run<br>
WARNING: Process ID 115551 still running in chroot. Killing...<br>
WARNING: Process ID 115576 still running in chroot. Killing...<br>
WARNING: Process ID 115577 still running in chroot. Killing...<br>
WARNING: Process ID 115578 still running in chroot. Killing...<br>
WARNING: Process ID 116634 still running in chroot. Killing...<br>
Finish: run<br>
++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' <<a=
href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-sy=
stem-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferrer" target=3D"_=
blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-syst=
em-tests/mocker-fedora-23-x86_64.fc23.cfg</a>><br>
+ mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br>
+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]<br>
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br>
++ mount<br>
++ awk '{print $3}'<br>
++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br>
+ [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc ]]<br>
+ echo 'Found mounted dirs inside the chroot . Trying to umount.'<br>
Found mounted dirs inside the chroot . Trying to umount.<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/sys<br>
sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/shm<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/pts<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/cache/yum<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_h=
e-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D"_blank">=
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test=
s</a>><br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/run/libvirt<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/lib/lago<br>
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/var/lib/lago: not mounted<br>
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621=
56a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'<br>
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a=
935ea8b988eb73b2ad-91644/root/var/lib/lago.<br>
+ failed=3Dtrue<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc/filesystems<br>
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc/filesystems: mountpoint not found<br>
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621=
56a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'<br>
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a=
935ea8b988eb73b2ad-91644/root/proc/filesystems.<br>
+ failed=3Dtrue<br>
+ for mock_root in '/var/lib/mock/*'<br>
+ this_chroot_failed=3Dfalse<br>
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br>
++ mount<br>
++ awk '{print $3}'<br>
++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44<br>
+ :<br>
+ [[ -n '' ]]<br>
+ false<br>
+ sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2=
ad-91644<br>
+ true<br>
+ echo Aborting.<br>
Aborting.<br>
+ exit 1<br>
POST BUILD TASK : FAILURE<br>
END OF POST BUILD TASK : 1<br>
Recording test results<br>
ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t=
est report files were found. Configuration error?<br>
Archiving artifacts<br>
_______________________________________________<br>
Infra mailing list<br>
<a href=3D"mailto:Infra@ovirt.org" target=3D"_blank">Infra@ovirt.org</a><br=
>
<a href=3D"http://lists.ovirt.org/mailman/listinfo/infra" rel=3D"noreferrer=
" target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/infra</a><br>
<br>
<br>
</blockquote></div>
</div><div><br></div></div></body></html>
------=_Part_9639957_1092846933.1468912250886--
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification. [1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully) On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Jul 19, 2016 at 9:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
does the above repo take rpms from same jobs used by nightly publisher?
Is it only for master job ? Or we have such repos for all branches ? On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Best regards Tolik Litovsky RHEV-H Team Red Hat Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com
we have it for all versions: http://resources.ovirt.org/repos/ovirt/experimental/ On Tue, Jul 19, 2016 at 11:27 AM, Tolik Litovsky <tlitovsk@redhat.com> wrote:
Is it only for master job ? Or we have such repos for all branches ?
On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
+ OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
+ cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Best regards Tolik Litovsky RHEV-H Team Red Hat
Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Jul 19, 2016 at 10:30 AM, Eyal Edri <eedri@redhat.com> wrote:
we have it for all versions: http://resources.ovirt.org/repos/ovirt/experimental/
I guess http://resources.ovirt.org/repos/ovirt/experimental/3.6.7/ can be dropped.
On Tue, Jul 19, 2016 at 11:27 AM, Tolik Litovsky <tlitovsk@redhat.com> wrote:
Is it only for master job ? Or we have such repos for all branches ?
On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Best regards Tolik Litovsky RHEV-H Team Red Hat
Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On Tue, Jul 19, 2016 at 10:30 AM, Eyal Edri <eedri@redhat.com> wrote:
we have it for all versions: http://resources.ovirt.org/repos/ovirt/experimental/
So: http://resources.ovirt.org/repos/ovirt/experimental/3.6/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/4.0/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested should be equivalend to the snapshot + snapshot-static repos right?
On Tue, Jul 19, 2016 at 11:27 AM, Tolik Litovsky <tlitovsk@redhat.com> wrote:
Is it only for master job ? Or we have such repos for all branches ?
On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-...
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests...
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests
+ for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Best regards Tolik Litovsky RHEV-H Team Red Hat
Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
Yes. On Thu, Jul 21, 2016 at 9:51 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Tue, Jul 19, 2016 at 10:30 AM, Eyal Edri <eedri@redhat.com> wrote:
we have it for all versions: http://resources.ovirt.org/repos/ovirt/experimental/
So:
http://resources.ovirt.org/repos/ovirt/experimental/3.6/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/4.0/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested
should be equivalend to the snapshot + snapshot-static repos right?
On Tue, Jul 19, 2016 at 11:27 AM, Tolik Litovsky <tlitovsk@redhat.com> wrote:
Is it only for master job ? Or we have such repos for all branches ?
On Tue, Jul 19, 2016 at 10:13 AM, Eyal Edri <eedri@redhat.com> wrote:
Ryan/Tolik, Can you build appliance only from tested engine repo [1] ? lets see how it affects the stability, next step will be to publish tested appliance after it runs Lago verification.
[1] http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/ (published only after ovirt-system-tests basic suite finish successfully)
On Tue, Jul 19, 2016 at 10:10 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
The last failed run failed on: *15:50:02* [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) *21:35:04* Build timed out (after 360 minutes). Marking the build as failed.
So it basically got stuck while extracting the OVF image.
Some previous runs failed mostly on either: a) broken ovirt-engine-appliance build b) ovirt-engine-appliance missing from the yum repo
We need to make sure that the process of building and publishing the ovirt-engine-appliance works flawlessly e.g. build ovirt-engine, publish it into the repo so that the build of the appliance can work, then publish it to the repo as well. This is extra important as the hosted-engine flow installation will probably become the default one, and without synced ovirt appliance we can't really test the changes in the engine.
Thanks in advance, Lev Veyde.
------------------------------ *From: *"Eyal Edri" <eedri@redhat.com> *To: *jenkins@jenkins.phx.ovirt.org *Cc: *"infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com *Sent: *Tuesday, July 19, 2016 8:26:22 AM *Subject: *Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32
Lev, this test is a bit flaky going from stable to failure quite often, can you check what is causing it? On Jul 19, 2016 12:35 AM, <jenkins@jenkins.phx.ovirt.org> wrote:
See < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes>
Changes:
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0
[Sandro Bonazzola] vdsm: avoid fc24 out of master
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing
------------------------------------------ [...truncated 620 lines...]
WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests... > + rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-... > + mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-... > + [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC
_______________________________________________________________________
####################################################################### # # # CLEANUP # # #
####################################################################### EOC
# Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi
# stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true }
mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done
# Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done
if $failed; then echo "Aborting." exit 1 fi
# remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat
_______________________________________________________________________
####################################################################### # # # CLEANUP # # #
####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ <http://mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/> ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests '> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.17 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run WARNING: Process ID 115551 still running in chroot. Killing... WARNING: Process ID 115576 still running in chroot. Killing... WARNING: Process ID 115577 still running in chroot. Killing... WARNING: Process ID 115578 still running in chroot. Killing... WARNING: Process ID 116634 still running in chroot. Killing... Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests... > + mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc ]] + echo 'Found mounted dirs inside the chroot . Trying to umount.' Found mounted dirs inside the chroot . Trying to umount. + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/sys sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/shm + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/dev/pts + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/cache/yum + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests > + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/run/libvirt + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago: not mounted + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago. + failed=true + for mount in '"${mounts[@]}"' + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems: mountpoint not found + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.' ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems. + failed=true + for mock_root in '/var/lib/mock/*' + this_chroot_failed=false + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + : + [[ -n '' ]] + false + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-91644 + true + echo Aborting. Aborting. + exit 1 POST BUILD TASK : FAILURE END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Best regards Tolik Litovsky RHEV-H Team Red Hat
Red Hat: trustworthy, transformative technology. Powered by the community. Connect at redhat.com
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Thu, Jul 21, 2016 at 9:21 AM, Eyal Edri <eedri@redhat.com> wrote:
Yes.
On Thu, Jul 21, 2016 at 9:51 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Tue, Jul 19, 2016 at 10:30 AM, Eyal Edri <eedri@redhat.com> wrote:
we have it for all versions: http://resources.ovirt.org/repos/ovirt/experimental/
So:
http://resources.ovirt.org/repos/ovirt/experimental/3.6/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/4.0/latest.tested http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested
should be equivalend to the snapshot + snapshot-static repos right?
Ok, I'll try to find the time for comparing the repos and see what's missing and why.
participants (5)
-
Eyal Edri -
jenkins＠jenkins.phx.ovirt.org -
Lev Veyde -
Sandro Bonazzola -
Tolik Litovsky