Build failed in Jenkins: ovirt_3.6_he-system-tests #790
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/changes> Changes: [Yaniv Kaul] Fixes and changes to storage tests [Dan Kenigsberg] vdsm: exclude f23 from 4.1 branch [Yaniv Bronhaim] Update Jenkins slaves for vdsm to f24 and f25 and remove excludes [Eyal Edri] update vdsm jobs to use 4.1 jobs [ngoldin] Add fc25 lago jobs [Juan Hernandez] Build and check API model from branch 4.1 ------------------------------------------ [...truncated 608 lines...] + local ID=3 + lvcreate -L20G -n lun3_bdev vg1_storage + targetcli /backstores/block create name=lun3_bdev dev=/dev/vg1_storage/lun3_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun3_bdev + for I in '$(seq $NUM_LUNS)' + create_lun 4 + local ID=4 + lvcreate -L20G -n lun4_bdev vg1_storage + targetcli /backstores/block create name=lun4_bdev dev=/dev/vg1_storage/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set auth userid=username password=password + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set attribute demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 default_cmdsn_depth=64 + targetcli saveconfig + systemctl enable target Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. + systemctl start target + sed -i 's/#node.session.auth.authmethod = CHAP/node.session.auth.authmethod = CHAP/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.username = username/node.session.auth.username = username/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.password = password/node.session.auth.password = password/g' /etc/iscsi/iscsid.conf + iscsiadm -m discovery -t sendtargets -p 127.0.0.1 + iscsiadm -m node -L all + rescan-scsi-bus.sh which: no multipath in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) + lsscsi -i + grep 36 + awk '{print $NF}' + sort + iscsiadm -m node -U all + iscsiadm -m node -o delete + systemctl stop iscsi.service + systemctl disable iscsi.service Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service. + install_deps_389ds + yum install -y --downloaddir=/dev/shm 389-ds-base + setup_389ds + DOMAIN=lago.local + PASSWORD=12345678 ++ sed s/_/-/g ++ hostname + HOSTNAME=lago-he-basic-suite-3-6-storage.lago.local ++ /sbin/ip -4 -o addr show dev eth0 ++ awk '{split($4,a,"."); print a[1] "." a[2] "." a[3] "." a[4]}' ++ awk -F/ '{print $1}' + ADDR=192.168.200.2 + cat + sed -i s/@HOSTNAME@/lago-he-basic-suite-3-6-storage.lago.local/g answer_file.inf + sed -i s/@PASSWORD@/12345678/g answer_file.inf + sed -i s/@DOMAIN@/lago.local/g answer_file.inf + cat + /usr/sbin/setup-ds.pl --silent --file=answer_file.inf Warning: using root as the server user id. You are strongly encouraged to use a non-root user. * [Thread-2] Deploy VM lago-he-basic-suite-3-6-storage: ERROR (in 0:02:27) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage * [Thread-4] Deploy VM lago-he-basic-suite-3-6-host0: ERROR (in 0:08:50) * [Thread-5] Deploy VM lago-he-basic-suite-3-6-host1: ERROR (in 0:08:51) # Deploy environment: ERROR (in 0:08:51) @ Deploy oVirt environment: ERROR (in 0:08:51) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 281, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 68, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage + res=1 + exit 1 + cleanup + rm -rf exported-artifacts + mkdir -p exported-artifacts + [[ -d deployment-he-basic-suite-3.6/current/logs ]] + mv deployment-he-basic-suite-3.6/current/logs exported-artifacts/lago_logs + find deployment-he-basic-suite-3.6 -iname 'nose*.xml' -exec mv '{}' exported-artifacts/ ';' + [[ -d test_logs ]] + [[ -e failure_msg.txt ]] + mv failure_msg.txt exported-artifacts/ + ./run_suite.sh --cleanup he-basic-suite-3.6 + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() ++ getopt -o ho:e:n:b:cs:r: --long help,output:,engine:,node:,boot-iso:,cleanup --long extra-rpm-source,reposync-config: -n run_suite.sh -- --cleanup he-basic-suite-3.6 + options=' --cleanup -- '\''he-basic-suite-3.6'\''' + [[ 0 != \0 ]] + eval set -- ' --cleanup -- '\''he-basic-suite-3.6'\''' ++ set -- --cleanup -- he-basic-suite-3.6 + true + case $1 in + DO_CLEANUP=true + shift + true + case $1 in + shift + break + [[ -z he-basic-suite-3.6 ]] ++ realpath he-basic-suite-3.6 + export SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + '[' -z '' ']' + export PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 607 seconds =================================== logout Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Tue Dec 27 13:49:16 UTC 2016 automation/he_basic_suite_3.6.sh chroot finished @@ took 617 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="logs/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 5 seconds ============================ ########################################################## ## Tue Dec 27 13:49:21 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 622 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh/he_basic_suite_3.6.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 607 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson6318118267408142511.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/791/changes> Changes: [Sandro Bonazzola] 3.6 repo: drop double *-debuginfo [Juan Hernandez] Build and check API metamodel from branch 1.1 ------------------------------------------ [...truncated 608 lines...] + local ID=3 + lvcreate -L20G -n lun3_bdev vg1_storage + targetcli /backstores/block create name=lun3_bdev dev=/dev/vg1_storage/lun3_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun3_bdev + for I in '$(seq $NUM_LUNS)' + create_lun 4 + local ID=4 + lvcreate -L20G -n lun4_bdev vg1_storage + targetcli /backstores/block create name=lun4_bdev dev=/dev/vg1_storage/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set auth userid=username password=password + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set attribute demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 default_cmdsn_depth=64 + targetcli saveconfig + systemctl enable target Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. + systemctl start target + sed -i 's/#node.session.auth.authmethod = CHAP/node.session.auth.authmethod = CHAP/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.username = username/node.session.auth.username = username/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.password = password/node.session.auth.password = password/g' /etc/iscsi/iscsid.conf + iscsiadm -m discovery -t sendtargets -p 127.0.0.1 + iscsiadm -m node -L all + rescan-scsi-bus.sh which: no multipath in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) + lsscsi -i + grep 36 + sort + awk '{print $NF}' + iscsiadm -m node -U all + iscsiadm -m node -o delete + systemctl stop iscsi.service + systemctl disable iscsi.service Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service. + install_deps_389ds + yum install -y --downloaddir=/dev/shm 389-ds-base + setup_389ds + DOMAIN=lago.local + PASSWORD=12345678 ++ hostname ++ sed s/_/-/g + HOSTNAME=lago-he-basic-suite-3-6-storage.lago.local ++ /sbin/ip -4 -o addr show dev eth0 ++ awk '{split($4,a,"."); print a[1] "." a[2] "." a[3] "." a[4]}' ++ awk -F/ '{print $1}' + ADDR=192.168.200.2 + cat + sed -i s/@HOSTNAME@/lago-he-basic-suite-3-6-storage.lago.local/g answer_file.inf + sed -i s/@PASSWORD@/12345678/g answer_file.inf + sed -i s/@DOMAIN@/lago.local/g answer_file.inf + cat + /usr/sbin/setup-ds.pl --silent --file=answer_file.inf Warning: using root as the server user id. You are strongly encouraged to use a non-root user. * [Thread-2] Deploy VM lago-he-basic-suite-3-6-storage: ERROR (in 0:02:15) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage * [Thread-5] Deploy VM lago-he-basic-suite-3-6-host1: ERROR (in 0:08:40) * [Thread-4] Deploy VM lago-he-basic-suite-3-6-host0: ERROR (in 0:09:44) # Deploy environment: ERROR (in 0:09:44) @ Deploy oVirt environment: ERROR (in 0:09:44) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 281, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 68, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage + res=1 + exit 1 + cleanup + rm -rf exported-artifacts + mkdir -p exported-artifacts + [[ -d deployment-he-basic-suite-3.6/current/logs ]] + mv deployment-he-basic-suite-3.6/current/logs exported-artifacts/lago_logs + find deployment-he-basic-suite-3.6 -iname 'nose*.xml' -exec mv '{}' exported-artifacts/ ';' + [[ -d test_logs ]] + [[ -e failure_msg.txt ]] + mv failure_msg.txt exported-artifacts/ + ./run_suite.sh --cleanup he-basic-suite-3.6 + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() ++ getopt -o ho:e:n:b:cs:r: --long help,output:,engine:,node:,boot-iso:,cleanup --long extra-rpm-source,reposync-config: -n run_suite.sh -- --cleanup he-basic-suite-3.6 + options=' --cleanup -- '\''he-basic-suite-3.6'\''' + [[ 0 != \0 ]] + eval set -- ' --cleanup -- '\''he-basic-suite-3.6'\''' ++ set -- --cleanup -- he-basic-suite-3.6 + true + case $1 in + DO_CLEANUP=true + shift + true + case $1 in + shift + break + [[ -z he-basic-suite-3.6 ]] ++ realpath he-basic-suite-3.6 + export SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + '[' -z '' ']' + export PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 666 seconds =================================== logout Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Wed Dec 28 03:03:26 UTC 2016 automation/he_basic_suite_3.6.sh chroot finished @@ took 676 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="logs/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Wed Dec 28 03:03:32 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 682 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh/he_basic_suite_3.6.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 666 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson873998699656879733.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/791/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/791/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/791/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/792/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables [Gil Shinar] Changed build discarder conf to deploy experimental [Gil Shinar] Removing vdsm ppc64 build for now ------------------------------------------ [...truncated 608 lines...] + local ID=3 + lvcreate -L20G -n lun3_bdev vg1_storage + targetcli /backstores/block create name=lun3_bdev dev=/dev/vg1_storage/lun3_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun3_bdev + for I in '$(seq $NUM_LUNS)' + create_lun 4 + local ID=4 + lvcreate -L20G -n lun4_bdev vg1_storage + targetcli /backstores/block create name=lun4_bdev dev=/dev/vg1_storage/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set auth userid=username password=password + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set attribute demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 default_cmdsn_depth=64 + targetcli saveconfig + systemctl enable target Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. + systemctl start target + sed -i 's/#node.session.auth.authmethod = CHAP/node.session.auth.authmethod = CHAP/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.username = username/node.session.auth.username = username/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.password = password/node.session.auth.password = password/g' /etc/iscsi/iscsid.conf + iscsiadm -m discovery -t sendtargets -p 127.0.0.1 + iscsiadm -m node -L all + rescan-scsi-bus.sh which: no multipath in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) + lsscsi -i + grep 36 + awk '{print $NF}' + sort + iscsiadm -m node -U all + iscsiadm -m node -o delete + systemctl stop iscsi.service + systemctl disable iscsi.service Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service. + install_deps_389ds + yum install -y --downloaddir=/dev/shm 389-ds-base + setup_389ds + DOMAIN=lago.local + PASSWORD=12345678 ++ hostname ++ sed s/_/-/g + HOSTNAME=lago-he-basic-suite-3-6-storage.lago.local ++ /sbin/ip -4 -o addr show dev eth0 ++ awk '{split($4,a,"."); print a[1] "." a[2] "." a[3] "." a[4]}' ++ awk -F/ '{print $1}' + ADDR=192.168.200.2 + cat + sed -i s/@HOSTNAME@/lago-he-basic-suite-3-6-storage.lago.local/g answer_file.inf + sed -i s/@PASSWORD@/12345678/g answer_file.inf + sed -i s/@DOMAIN@/lago.local/g answer_file.inf + cat + /usr/sbin/setup-ds.pl --silent --file=answer_file.inf Warning: using root as the server user id. You are strongly encouraged to use a non-root user. * [Thread-2] Deploy VM lago-he-basic-suite-3-6-storage: ERROR (in 0:02:25) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage * [Thread-5] Deploy VM lago-he-basic-suite-3-6-host1: ERROR (in 0:08:29) * [Thread-4] Deploy VM lago-he-basic-suite-3-6-host0: ERROR (in 0:08:29) # Deploy environment: ERROR (in 0:08:29) @ Deploy oVirt environment: ERROR (in 0:08:30) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 281, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 68, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6/default/scripts/_home_jenkins_workspace_ovirt_3.6_he-system-tests_ovirt-system-tests_he-basic-suite-3.6_.._common_deploy-scripts_setup_storage_unified_el7.sh> failed with status 1 on lago-he-basic-suite-3-6-storage + res=1 + exit 1 + cleanup + rm -rf exported-artifacts + mkdir -p exported-artifacts + [[ -d deployment-he-basic-suite-3.6/current/logs ]] + mv deployment-he-basic-suite-3.6/current/logs exported-artifacts/lago_logs + find deployment-he-basic-suite-3.6 -iname 'nose*.xml' -exec mv '{}' exported-artifacts/ ';' + [[ -d test_logs ]] + [[ -e failure_msg.txt ]] + mv failure_msg.txt exported-artifacts/ + ./run_suite.sh --cleanup he-basic-suite-3.6 + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() ++ getopt -o ho:e:n:b:cs:r: --long help,output:,engine:,node:,boot-iso:,cleanup --long extra-rpm-source,reposync-config: -n run_suite.sh -- --cleanup he-basic-suite-3.6 + options=' --cleanup -- '\''he-basic-suite-3.6'\''' + [[ 0 != \0 ]] + eval set -- ' --cleanup -- '\''he-basic-suite-3.6'\''' ++ set -- --cleanup -- he-basic-suite-3.6 + true + case $1 in + DO_CLEANUP=true + shift + true + case $1 in + shift + break + [[ -z he-basic-suite-3.6 ]] ++ realpath he-basic-suite-3.6 + export SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he-basic-suite-3.6> + '[' -z '' ']' + export PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 593 seconds =================================== logout Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Thu Dec 29 03:02:13 UTC 2016 automation/he_basic_suite_3.6.sh chroot finished @@ took 604 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="logs/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Thu Dec 29 03:02:19 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 610 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh/he_basic_suite_3.6.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he-basic-suite-3.6> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 593 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson8978171956504135217.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/792/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/792/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/792/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/793/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables [Barak Korren] Fix standard-ci artifact collection [Barak Korren] Made mock runner create logs in a safe place [Eyal Edri] enable integration for getbadgets on ovirt projects [Eyal Edri] make manual trigger generic [Eyal Edri] add new 'custom repos' parameter to standard CI [Eyal Edri] Update system-tests.sh to accept also custom repos ------------------------------------------ [...truncated 41 lines...] Pruning obsolete local branches Fetching upstream changes from git://gerrit.ovirt.org/jenkins.git
git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6ef858608152457d02216ecb693417f54a0b1240 (origin/master) git config core.sparsecheckout # timeout=10 git checkout -f 6ef858608152457d02216ecb693417f54a0b1240 git rev-list 0e7fcd0091f0f76a88553920776296bec9f28156 # timeout=10 git branch -a # timeout=10 git rev-parse remotes/origin/master^{commit} # timeout=10 [ovirt_3.6_he-system-tests] $ /bin/bash -e /tmp/hudson5227744277625227450.sh shell-scripts/cleanup_slave.sh ################################################################### # Cleaning up slave # ################################################################### Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 3.9M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 22G 887G 3% / tmpfs 24G 344K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 4.0K 4.8G 1% /run/user/1012
Cleaning up postgres databases /tmp ~/workspace/ovirt_3.6_he-system-tests sudo: psql: command not found ~/workspace/ovirt_3.6_he-system-tests done Cleaning up journal logs (if any) Redirecting to /bin/systemctl restart systemd-journald.service Cleaning up /var/tmp done Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/ovirt_4.1_system-tests Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 3.9M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 22G 887G 3% / tmpfs 24G 344K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 4.0K 4.8G 1% /run/user/1012 ################################################################### # Slave cleanup done # ################################################################### [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson8341506101271135224.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=3.6 + SUITE_TYPE=he_basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + GIT_SUBDIR=ovirt-system-tests + OVIRT_SUITE=he_basic_suite_3.6 + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_3.6> + OVIRT_SUITE_DIR=ovirt-system-tests/he_basic-suite-3.6 + '[' -z '$' ']' + echo '' /tmp/hudson8341506101271135224.sh: line 19: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he_basic-suite-3.6/extra_sources>: No such file or directory Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson1924912245005789345.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/793/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/793/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson4159813628617463718.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/794/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables ------------------------------------------ [...truncated 41 lines...] Pruning obsolete local branches Fetching upstream changes from git://gerrit.ovirt.org/jenkins.git
git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6ef858608152457d02216ecb693417f54a0b1240 (origin/master) git config core.sparsecheckout # timeout=10 git checkout -f 6ef858608152457d02216ecb693417f54a0b1240 git rev-list 6ef858608152457d02216ecb693417f54a0b1240 # timeout=10 git branch -a # timeout=10 git rev-parse remotes/origin/master^{commit} # timeout=10 [ovirt_3.6_he-system-tests] $ /bin/bash -e /tmp/hudson1426184164411203900.sh shell-scripts/cleanup_slave.sh ################################################################### # Cleaning up slave # ################################################################### Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.0M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 17G 892G 2% / tmpfs 24G 344K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1012
Cleaning up postgres databases /tmp ~/workspace/ovirt_3.6_he-system-tests sudo: psql: command not found ~/workspace/ovirt_3.6_he-system-tests done Cleaning up journal logs (if any) Redirecting to /bin/systemctl restart systemd-journald.service Cleaning up /var/tmp done Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/ovirt_4.1_system-tests Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.0M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 16G 892G 2% / tmpfs 24G 344K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1012 ################################################################### # Slave cleanup done # ################################################################### [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson2493590504192633312.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=3.6 + SUITE_TYPE=he_basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + GIT_SUBDIR=ovirt-system-tests + OVIRT_SUITE=he_basic_suite_3.6 + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_3.6> + OVIRT_SUITE_DIR=ovirt-system-tests/he_basic-suite-3.6 + '[' -z '$' ']' + echo '' /tmp/hudson2493590504192633312.sh: line 19: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he_basic-suite-3.6/extra_sources>: No such file or directory Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson2637622193939301217.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/794/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/794/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson453840677439823350.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/795/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables ------------------------------------------ [...truncated 41 lines...] Pruning obsolete local branches Fetching upstream changes from git://gerrit.ovirt.org/jenkins.git
git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6ef858608152457d02216ecb693417f54a0b1240 (origin/master) git config core.sparsecheckout # timeout=10 git checkout -f 6ef858608152457d02216ecb693417f54a0b1240 git rev-list 6ef858608152457d02216ecb693417f54a0b1240 # timeout=10 git branch -a # timeout=10 git rev-parse remotes/origin/master^{commit} # timeout=10 [ovirt_3.6_he-system-tests] $ /bin/bash -e /tmp/hudson338964030771733638.sh shell-scripts/cleanup_slave.sh ################################################################### # Cleaning up slave # ################################################################### Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.0M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 18G 890G 2% / tmpfs 24G 348K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1012
Cleaning up postgres databases /tmp ~/workspace/ovirt_3.6_he-system-tests sudo: psql: command not found ~/workspace/ovirt_3.6_he-system-tests done Cleaning up journal logs (if any) Redirecting to /bin/systemctl restart systemd-journald.service Cleaning up /var/tmp done Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/ovirt_4.1_system-tests Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.0M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 908G 18G 890G 2% / tmpfs 24G 348K 24G 1% /tmp /dev/sda1 253M 121M 132M 48% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1012 ################################################################### # Slave cleanup done # ################################################################### [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson4763745502520437137.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=3.6 + SUITE_TYPE=he_basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + GIT_SUBDIR=ovirt-system-tests + OVIRT_SUITE=he_basic_suite_3.6 + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/deployment-he_basic_suite_3.6> + OVIRT_SUITE_DIR=ovirt-system-tests/he_basic-suite-3.6 + '[' -z '$' ']' + echo '' /tmp/hudson4763745502520437137.sh: line 19: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/he_basic-suite-3.6/extra_sources>: No such file or directory Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson7790272581281300712.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/795/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/795/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson1231816163382908833.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/796/changes> Changes: [Sandro Bonazzola] ovirt-3.6-hosted-engine: fix yum deps ------------------------------------------ [...truncated 767 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Mon Jan 2 20:50:36 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 706 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.2hPacl90/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Mon Jan 2 20:50:42 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 712 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.2hPacl90/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.2hPacl90/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.2hPacl90/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson1833596254076257934.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/796/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/796/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/796/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson885586299604292555.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + [[ -n epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/797/changes> Changes: [Daniel Belenky] upgrade-suite: upgrade from 4.0 to experimental ------------------------------------------ [...truncated 758 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Tue Jan 3 03:03:50 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 643 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.BMAyO333/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Tue Jan 3 03:03:56 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 649 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.BMAyO333/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.BMAyO333/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.BMAyO333/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson4285542815310253230.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/797/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/797/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/797/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson3734801319491179213.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + [[ -n epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/798/changes> Changes: [Sandro Bonazzola] ovirt-3.6-hosted-engine: fix yum deps [Eyal Edri] adding dashboard 4.1 to build from master [Gil Shinar] Fix some more issues with SDK [Gil Shinar] Some missing 4.1 in experimental [Pavel Zhukov] Add publishers to deploy-to-XXX jobs [Eyal Edri] add missing 4.1 build jobs [Eyal Edri] adding more missing projects to 4.1 ------------------------------------------ [...truncated 768 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Tue Jan 3 13:43:53 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 801 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.Skwntvia/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 19 seconds ============================ ########################################################## ## Tue Jan 3 13:44:12 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 820 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.Skwntvia/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.Skwntvia/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.Skwntvia/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson1716018258153034861.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/798/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/798/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/798/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson5402030096541529665.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + [[ -n epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/799/changes> Changes: [Your Name] network utils: introducing a network utility module [Gil Shinar] Add 4.1 to vmconsole [Martin Perina] Updates aaa-ldap repository locations to experimental repos [Eyal Edri] remove fc23 and add fc25 builds for ovirt-engine [Eyal Edri] Refactoring vdsm yaml [Barak Korren] Added mirror management jobs [Barak Korren] Added support for creating indexes of latest repos [Barak Korren] Ready mirrors for production use ------------------------------------------ [...truncated 632 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Wed Jan 4 03:00:13 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 422 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.g2G6HUyA/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Wed Jan 4 03:00:19 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 428 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.g2G6HUyA/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.g2G6HUyA/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.g2G6HUyA/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson7406337097271783431.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/799/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/799/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/799/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson1803466406423297537.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + [[ -n epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/800/changes> Changes: [ngoldin] Run in /dev/shm if more than 15GB are available [Barak Korren] Re-factored system-tests builders ------------------------------------------ [...truncated 685 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Wed Jan 4 17:57:49 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 349 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.X3RvoZJF/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 5 seconds ============================ ########################################################## ## Wed Jan 4 17:57:54 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 354 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.X3RvoZJF/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.X3RvoZJF/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.X3RvoZJF/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson5277197084354723682.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/800/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/800/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/800/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson5220888527301341669.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + [[ -n epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/801/changes> Changes: [ngoldin] Use 'lago collect' instead of 'lago ovirt collect' ------------------------------------------ [...truncated 653 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Thu Jan 5 03:00:51 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 463 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.5CbEScHB/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Thu Jan 5 03:00:57 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 469 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.5CbEScHB/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.5CbEScHB/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.5CbEScHB/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson2451110223976424542.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/801/artifact/exported-artifacts> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/801/artifact/exported-artifacts> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/failure_msg.txt> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/lago_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/exported-artifacts/mock_logs> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/801/artifact/exported-artifacts/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson5230433777584195965.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/logging.ini.> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> + mock_root=epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + [[ -n epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-aa7d25407b9c15b3736f25fc4fcf3710 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts
participants (1)
-
jenkins＠jenkins.phx.ovirt.org