December 2016 - Infra - oVirt List Archives

Re: false CI -1
by Martin Mucha 01 Feb '17

01 Feb '17

Hi, this seems as false ci -1 http://jenkins.ovirt.org/job/ovirt-engine_master_find-bugs_created/9180/con… M.

3 5

[oVirt Jenkins] system-backup_jenkins_old_ovirt_org - Build # 145 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jan '17

21 Jan '17

------=_Part_434_600438096.1481932952050 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/ Build: http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/145/ Build Number: 145 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #145 [Eyal Edri] move ovirt-engine build artifacts to build on-change [Sandro Bonazzola] Revert "qemu: use 7.3 beta repo" [Sandro Bonazzola] publishers: added 4.1 nightly publisher [Sandro Bonazzola] repoclosure: fedora: add virt-preview repository [Vinzenz Feenstra] ovirt-guest-agent: drop fc23 [Gil Shinar] Fix guest agent in publisher jobs [Sandro Bonazzola] ovirt-host-deploy: move 4.1 to stable branch [Martin Perina] aaa-ldap: Create stable branch for oVirt 4.0 [Sandro Bonazzola] ovirt-log-collector: move 4.1 to stable branch [Eyal Edri] add more history to standard jobs [Barak Korren] Fix issue with mock_genconfig [Evgheni Dereveanchin] increase max concurrent builds to 30 [Evgheni Dereveanchin] OVIRT-645 remove repoproxy for Fedora and EL6 x86_64 [Barak Korren] Make mock_cleanup.sh safer about un-mouting [Barak Korren] Fix mock_genconfig nested dict settings [Sandro Bonazzola] repo_closure_check: fix el7 repos for metrics [Sandro Bonazzola] repos-check-closure: add 4.1 repos [Sandro Bonazzola] ovirt-live: add 4.1 branch jobs ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_434_600438096.1481932952050--

1 3

Build failed in Jenkins: ovirt_3.6_he-system-tests #790
by jenkins＠jenkins.phx.ovirt.org 05 Jan '17

05 Jan '17

See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/changes> Changes: [Yaniv Kaul] Fixes and changes to storage tests [Dan Kenigsberg] vdsm: exclude f23 from 4.1 branch [Yaniv Bronhaim] Update Jenkins slaves for vdsm to f24 and f25 and remove excludes [Eyal Edri] update vdsm jobs to use 4.1 jobs [ngoldin] Add fc25 lago jobs [Juan Hernandez] Build and check API model from branch 4.1 ------------------------------------------ [...truncated 608 lines...] + local ID=3 + lvcreate -L20G -n lun3_bdev vg1_storage + targetcli /backstores/block create name=lun3_bdev dev=/dev/vg1_storage/lun3_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun3_bdev + for I in '$(seq $NUM_LUNS)' + create_lun 4 + local ID=4 + lvcreate -L20G -n lun4_bdev vg1_storage + targetcli /backstores/block create name=lun4_bdev dev=/dev/vg1_storage/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun4_bdev + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set auth userid=username password=password + targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set attribute demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 default_cmdsn_depth=64 + targetcli saveconfig + systemctl enable target Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. + systemctl start target + sed -i 's/#node.session.auth.authmethod = CHAP/node.session.auth.authmethod = CHAP/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.username = username/node.session.auth.username = username/g' /etc/iscsi/iscsid.conf + sed -i 's/#node.session.auth.password = password/node.session.auth.password = password/g' /etc/iscsi/iscsid.conf + iscsiadm -m discovery -t sendtargets -p 127.0.0.1 + iscsiadm -m node -L all + rescan-scsi-bus.sh which: no multipath in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) + lsscsi -i + grep 36 + awk '{print $NF}' + sort + iscsiadm -m node -U all + iscsiadm -m node -o delete + systemctl stop iscsi.service + systemctl disable iscsi.service Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service. + install_deps_389ds + yum install -y --downloaddir=/dev/shm 389-ds-base + setup_389ds + DOMAIN=lago.local + PASSWORD=12345678 ++ sed s/_/-/g ++ hostname + HOSTNAME=lago-he-basic-suite-3-6-storage.lago.local ++ /sbin/ip -4 -o addr show dev eth0 ++ awk '{split($4,a,"."); print a[1] "." a[2] "." a[3] "." a[4]}' ++ awk -F/ '{print $1}' + ADDR=192.168.200.2 + cat + sed -i s/@HOSTNAME@/lago-he-basic-suite-3-6-storage.lago.local/g answer_file.inf + sed -i s/@PASSWORD@/12345678/g answer_file.inf + sed -i s/@DOMAIN@/lago.local/g answer_file.inf + cat + /usr/sbin/setup-ds.pl --silent --file=answer_file.inf Warning: using root as the server user id. You are strongly encouraged to use a non-root user. * [Thread-2] Deploy VM lago-he-basic-suite-3-6-storage: ERROR (in 0:02:27) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> failed with status 1 on lago-he-basic-suite-3-6-storage * [Thread-4] Deploy VM lago-he-basic-suite-3-6-host0: ERROR (in 0:08:50) * [Thread-5] Deploy VM lago-he-basic-suite-3-6-host1: ERROR (in 0:08:51) # Deploy environment: ERROR (in 0:08:51) @ Deploy oVirt environment: ERROR (in 0:08:51) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 281, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 68, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name(), RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> failed with status 1 on lago-he-basic-suite-3-6-storage + res=1 + exit 1 + cleanup + rm -rf exported-artifacts + mkdir -p exported-artifacts + [[ -d deployment-he-basic-suite-3.6/current/logs ]] + mv deployment-he-basic-suite-3.6/current/logs exported-artifacts/lago_logs + find deployment-he-basic-suite-3.6 -iname 'nose*.xml' -exec mv '{}' exported-artifacts/ ';' + [[ -d test_logs ]] + [[ -e failure_msg.txt ]] + mv failure_msg.txt exported-artifacts/ + ./run_suite.sh --cleanup he-basic-suite-3.6 + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() ++ getopt -o ho:e:n:b:cs:r: --long help,output:,engine:,node:,boot-iso:,cleanup --long extra-rpm-source,reposync-config: -n run_suite.sh -- --cleanup he-basic-suite-3.6 + options=' --cleanup -- '\''he-basic-suite-3.6'\''' + [[ 0 != \0 ]] + eval set -- ' --cleanup -- '\''he-basic-suite-3.6'\''' ++ set -- --cleanup -- he-basic-suite-3.6 + true + case $1 in + DO_CLEANUP=true + shift + true + case $1 in + shift + break + [[ -z he-basic-suite-3.6 ]] ++ realpath he-basic-suite-3.6 + export SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + '[' -z '' ']' + export PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 607 seconds =================================== logout Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Tue Dec 27 13:49:16 UTC 2016 automation/he_basic_suite_3.6.sh chroot finished @@ took 617 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="logs/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 5 seconds ============================ ########################################################## ## Tue Dec 27 13:49:21 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 622 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh/he_basic_suite_3.6.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 607 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson6318118267408142511.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/exporte…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

1 11

how does findbugs work?
by Michal Skrivanek 04 Jan '17

04 Jan '17

Hi, I was trying to see why the recent NPE[1] was not found by findbugs. Typically identifying potential NPE is the most easy thing…but it was not reported. So I triggered the build again on the wrong patch [2] and it doesn’t show anything useful[3] Does it work at all? There seem to be no artifacts to review that Thanks, michal [1] https://gerrit.ovirt.org/#/c/68731 [2] https://gerrit.ovirt.org/#/c/68400/5 [3] http://jenkins.ovirt.org/job/ovirt-engine_master_find-bugs_created/11732/

2 3

Re: vdsm ppc64 build fails
by Eyal Edri 03 Jan '17

03 Jan '17

Adding Danken, We currently don't have ppc64le builds for vdsm 4.1 due to this. On Sun, Dec 25, 2016 at 4:54 PM, Karanbir Singh <kbsingh(a)karan.org> wrote: > Altarch contet is released to http://mirror.centos.org/altarch > > Regards > > > On 25 December 2016 14:26:25 GMT+00:00, Eyal Edri <eedri(a)redhat.com> > wrote: >> >> Can you help with links to the alterarch so we can use them in the >> meantime? >> >> On Sun, Dec 25, 2016 at 3:50 PM, Karanbir Singh <kbsingh(a)karan.org> >> wrote: >> >>> We pushed ppc64le altarch 2 days back. I dont believe the SIG content os >>> there as yet. Just getting the arch's content lined up into cbs has proven >>> to be a challenge. >>> >>> Regards >>> >>> >>> On 25 December 2016 13:21:27 GMT+00:00, Sandro Bonazzola < >>> sbonazzo(a)redhat.com> wrote: >>>> >>>> Ppc64le is on altarch not on centos subdirectory. About the virt sig >>>> packages adding Karanbir, afaik both ppc64le and aarch64 are not yet pushed >>>> to mirrors. >>>> >>>> Il 25/Dic/2016 01:40 PM, "Eyal Edri" <eedri(a)redhat.com> ha scritto: >>>> >>>>> missing ppc64le repo on centos: >>>>> >>>>> DEBUG util.py:421: http://mirror.centos.org/cento >>>>> s/7/virt/ppc64le/ovirt-4.0/repodata/repomd.xml: [Errno 14] HTTP Error >>>>> 404 - Not Found >>>>> >>>>> If there is an alternate repo, please update it on >>>>> automation/.packages files in vdsm repo. >>>>> >>>>> -- >>>>> Eyal Edri >>>>> Associate Manager >>>>> RHV DevOps >>>>> EMEA ENG Virtualization R&D >>>>> Red Hat Israel >>>>> >>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >>>> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 <+972%209-769-2018> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 2

Build failed in Jenkins: ovirt_master_system-tests_manual #4
by jenkins＠jenkins.phx.ovirt.org 01 Jan '17

01 Jan '17

See <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/4/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables [Eyal Edri] enable integration for getbadgets on ovirt projects [Eyal Edri] make manual trigger generic [Eyal Edri] add new 'custom repos' parameter to standard CI [Eyal Edri] Update system-tests.sh to accept also custom repos ------------------------------------------ [...truncated 856 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Fri Dec 30 14:19:52 UTC 2016 automation/basic_suite_master.sh chroot finished @@ took 1522 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.fb3QTHLu/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 5 seconds ============================ ########################################################## ## Fri Dec 30 14:19:57 UTC 2016 Finished env: el7:epel-7-x86_64 ## took 1527 seconds ## rc = 1 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.fb3QTHLu/mocker-epel-7-x86_64.el7.basic_suite_master.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.basic_suite_master.sh' './mock_logs.fb3QTHLu/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.fb3QTHLu/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=master SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_master_system-tests_manual] $ /bin/bash -xe /tmp/hudson8914126124449348695.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=master + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/> + OVIRT_SUITE=master + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/4/artifact/ex…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/4/artifact/ex…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/4/artifact/ex…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_master_system-tests_manual] $ /bin/bash -x /tmp/hudson2325850048195419590.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/ws/ovirt-syst…> + mock_root=epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + [[ -n epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf sudo rm -rf /var/cache/mock/fedora-25-x86_64 /var/cache/mock/fedora-24-x86_64-774e2e8696ce3533639f115c2ae5f080 ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results Archiving artifacts

1 2

Build failed in Jenkins: ovirt_4.0_he-system-tests #612
by jenkins＠jenkins.phx.ovirt.org 01 Jan '17

01 Jan '17

See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/612/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables [Barak Korren] Fix standard-ci artifact collection [Barak Korren] Made mock runner create logs in a safe place [Eyal Edri] enable integration for getbadgets on ovirt projects [Eyal Edri] make manual trigger generic [Eyal Edri] add new 'custom repos' parameter to standard CI [Eyal Edri] Update system-tests.sh to accept also custom repos ------------------------------------------ [...truncated 44 lines...] > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6ef858608152457d02216ecb693417f54a0b1240 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 6ef858608152457d02216ecb693417f54a0b1240 > git rev-list 0e7fcd0091f0f76a88553920776296bec9f28156 # timeout=10 > git branch -a # timeout=10 > git rev-parse remotes/origin/master^{commit} # timeout=10 [ovirt_4.0_he-system-tests] $ /bin/bash -e /tmp/hudson6140029738256576947.sh shell-scripts/cleanup_slave.sh ################################################################### # Cleaning up slave # ################################################################### Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.8M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 894G 17G 832G 2% / tmpfs 24G 336K 24G 1% /tmp /dev/sda1 240M 93M 132M 42% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1016 tmpfs 4.8G 0 4.8G 0% /run/user/1013 --------------------------------------------------------------- Cleaning up postgres databases /tmp ~/workspace/ovirt_4.0_he-system-tests sudo: psql: command not found ~/workspace/ovirt_4.0_he-system-tests done Cleaning up journal logs (if any) Redirecting to /bin/systemctl restart systemd-journald.service Cleaning up /var/tmp done Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/ovirt-system-tests_master_check-patch-fc24-x86_64 Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.8M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 894G 17G 832G 2% / tmpfs 24G 336K 24G 1% /tmp /dev/sda1 240M 93M 132M 42% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1016 tmpfs 4.8G 0 4.8G 0% /run/user/1013 ################################################################### # Slave cleanup done # ################################################################### [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson8347200169630191906.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=4.0 + SUITE_TYPE=he_basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + GIT_SUBDIR=ovirt-system-tests + OVIRT_SUITE=he_basic_suite_4.0 + PREFIX=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + OVIRT_SUITE_DIR=ovirt-system-tests/he_basic-suite-4.0 + '[' -z '$' ']' + echo '' /tmp/hudson8347200169630191906.sh: line 19: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…>: No such file or directory Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1149753924956821861.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/612/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/612/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/hudson3954265573632757137.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf sudo rm -rf /var/cache/mock/epel-7-x86_64-ed452d1ea0f6dadf50c66d3b3f6b3a6e ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts

1 3

oVirt infra daily report - unstable production jobs - 186
by jenkins＠jenkins.phx.ovirt.org 01 Jan '17

01 Jan '17

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/186//artifact/exported-a… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 185
by jenkins＠jenkins.phx.ovirt.org 31 Dec '16

31 Dec '16

------=_Part_34_1004499502.1483138803528 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/185//artifact/exported-a… Cheers, Jenkins ------=_Part_34_1004499502.1483138803528 Content-Type: text/html; charset=us-ascii; name=upstream_report.html Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=upstream_report.html Content-ID: <upstream_report.html> <!DOCTYPE html><head><style type="text/css"> table.gridtable { border-collapse: collapse; table-layout:fixed; width:1600px; font-family: monospace; font-size:13px; } .head { font-size:20px; font-family: arial; } .sub { font-size:18px; background-color:#e5e5e5; font-family: arial; } pre { font-family: monospace; display: inline; white-space: pre-wrap; white-space: -moz-pre-wrap !important; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } </style> </head> <body> <table class="gridtable" border=2> <tr><th colspan=2 class=head> RHEVM CI Jenkins Daily Report - 30/12/2016 </th></tr><tr><th colspan=2 class=sub> <font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font> </th></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_build-…">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-fc24-x86…">ovirt-system-tests_master_check-patch-fc24-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/">ovirt_3.6_he-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_3.6_image-ng-system-tests/">ovirt_3.6_image-ng-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/">ovirt_4.0_he-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.0_image-ng-system-tests/">ovirt_4.0_image-ng-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests_manual/">ovirt_master_system-tests_manual</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/">system-backup_jenkins_old_ovirt_org</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  <br><br>Job disabled - The old Jenkins is probably not up any more </td></tr> ------=_Part_34_1004499502.1483138803528--

1 0

Build failed in Jenkins: ovirt_master_system-tests #889
by jenkins＠jenkins.phx.ovirt.org 30 Dec '16

30 Dec '16

See <http://jenkins.ovirt.org/job/ovirt_master_system-tests/889/changes> Changes: [Your Name] 002_bootstrap: removing unused local variables [Barak Korren] Fix standard-ci artifact collection [Barak Korren] Made mock runner create logs in a safe place ------------------------------------------ [...truncated 43 lines...] > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision ba9a1a004f0a28a8368c55dff90f41d42b6b3813 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f ba9a1a004f0a28a8368c55dff90f41d42b6b3813 > git rev-list 0e7fcd0091f0f76a88553920776296bec9f28156 # timeout=10 > git branch -a # timeout=10 > git rev-parse remotes/origin/master^{commit} # timeout=10 [ovirt_master_system-tests] $ /bin/bash -e /tmp/hudson4849152370772860847.sh shell-scripts/cleanup_slave.sh ################################################################### # Cleaning up slave # ################################################################### Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.5M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 894G 17G 832G 2% / tmpfs 24G 244K 24G 1% /tmp /dev/sda1 240M 93M 131M 42% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1015 tmpfs 4.8G 0 4.8G 0% /run/user/1013 --------------------------------------------------------------- Cleaning up postgres databases /tmp ~/workspace/ovirt_master_system-tests sudo: psql: command not found ~/workspace/ovirt_master_system-tests done Cleaning up journal logs (if any) Redirecting to /bin/systemctl restart systemd-journald.service Cleaning up /var/tmp done Emptying some common logs /var/log/wtmp Done /home/jenkins/workspace/ovirt-system-tests_master_check-patch-fc24-x86_64 Making sure there are no device mappings... Removing the used loop devices... Redirecting to /bin/systemctl restart libvirtd.service --------------------------------------------------------------- Filesystem Size Used Avail Use% Mounted on devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 4.5M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sda3 894G 17G 832G 2% / tmpfs 24G 244K 24G 1% /tmp /dev/sda1 240M 93M 131M 42% /boot tmpfs 4.8G 0 4.8G 0% /run/user/1015 tmpfs 4.8G 0 4.8G 0% /run/user/1013 ################################################################### # Slave cleanup done # ################################################################### [ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson6882223931526162872.sh + echo shell-scripts/system_tests.sh shell-scripts/system_tests.sh + VERSION=master + SUITE_TYPE=basic + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + GIT_SUBDIR=ovirt-system-tests + OVIRT_SUITE=basic_suite_master + PREFIX='<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/$/deployment-basi…'> + OVIRT_SUITE_DIR='$/basic-suite-master' + '[' -z '$' ']' + echo '' /tmp/hudson6882223931526162872.sh: line 19: <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/$/basic-suite-mas…>: No such file or directory Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=master SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson6037528795769975680.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=master + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + OVIRT_SUITE=master + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_master_system-tests/889/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_master_system-tests/889/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-test…> ]] POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_master_system-tests] $ /bin/bash -x /tmp/hudson2312842374362757078.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts

1 1