January 2017 - Infra - oVirt List Archives

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4809 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 18 Jan '17

18 Jan '17

------=_Part_205_640910850.1484700229779 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4809/, Build Number: 4809, Build Status: FAILURE ------=_Part_205_640910850.1484700229779--

1 1

Fwd: [lago-devel] Lago v0.32 released
by Nadav Goldin 18 Jan '17

18 Jan '17

Hi, as luckily this bug was discovered early, and mock cache refreshes once in 2 days, the only failure I could spot due to that is here[1], so to be sure Lago v0.32 won't get installed on the slaves, I cleaned up manually the mock cache on all ovirt-srv*, so on next run they should pull v0.33. For the record, if you see this error: .... 21:29:13 File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 103, in merge 21:29:13 ).format(sources, output_dir) 21:29:13 RepositoryMergeError: Failed merging repoman sources: [' "conf:/home/jenkins/workspace /test-repo_ovirt_experimental_4.1/ovirt-system-tests/extra_sources":only-missing', .... ] make sure lago v0.33 is installed(the version is displayed at the beginning of each experimental job run). The bug is in the "..extra_source":only-missing" parameter. Thanks gal for doing another quick release and my apologies for missing this, Nadav. [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/310/ ---------- Forwarded message ---------- From: Nadav Goldin <ngoldin(a)redhat.com> Date: Wed, Jan 18, 2017 at 12:18 AM Subject: Re: [lago-devel] Lago v0.32 released To: Gal Ben Haim <gbenhaim(a)redhat.com> Cc: lago-devel(a)ovirt.org Hi, After releasing, gbenhaim discovered a bug in 'lago ovirt reposetup', specifically in the combination of passing custom repoman filters and loading repoman sources from files.So the patch[1] was reverted, and the version was bumped to v0.33. When upgrading, you should upgrade directly to v0.33. Custom repoman filters will have to wait for the next version, however, passing a custom repoman configuration file was not affected, and can be done by running: lago ovirt reposetup --repoman-config ./path/to/repoman.config Thanks, Nadav. [1] https://github.com/lago-project/lago/commit/208bfc34051e6f6d6783e353c22868f… On Tue, Jan 17, 2017 at 8:24 PM, Gal Ben Haim <gbenhaim(a)redhat.com> wrote: > Adding some more info: > > Docs: http://lago.readthedocs.io/en/0.32/ > RPM Repository: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ > GitHub: https://github.com/lago-project/lago/ > For OST docs: http://ovirt-system-tests.readthedocs.io/en/latest/ > > Gal Ben Haim > gbenhaim(a)redhat.com > RHEV DevOps > > ----- Original Message ----- > From: "Gal Ben Haim" <gbenhaim(a)redhat.com> > To: lago-devel(a)ovirt.org > Sent: Tuesday, January 17, 2017 8:00:14 PM > Subject: Lago v0.32 released > > Hi All, > > Lago version v0.32 is out. > > the main changes are: > > 1. Allow to configure the cpu model in the init file by adding > "cpu_model: MODEL" to the vm definition (for example "cpu_model: SandyBridge"). > > 2. Allow to specify "repoman" config to "lago ovirt reposetup" with "--repoman-config". > > The full change-log: > > * Tue Jan 17 2017 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.32.0 > a7a99aee: Merge pull request #431 from nvgoldin/revert_ovirt_cpu > 66704a0c: Revert "ovirtlago: drop get_ovirt_cpu_family" > > * Mon Jan 16 2017 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.31.4 > 6dd96515: Merge pull request #418 from lago-project/adding_qemu_kvm_ev_dep > a7d71ad2: Force usage of qemu-kvm-rhev on centos/rhel > > * Mon Jan 16 2017 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.31.3 > f78e351b: Merge pull request #425 from nvgoldin/custom_repoman_conf > 41553b76: ovirtlago: allow defining custom repoman config file > 76bf0bae: ovirtlago: don't download source RPMs by default > 208bfc34: ovirtlago: allow passing custom repoman filters > b816daf8: ovirtlago: remove deprecated per-project repo directories > > * Mon Jan 16 2017 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.31.2 > 7f25ef33: Merge pull request #427 from nvgoldin/he_cpu > f31b74c4: ovirtlago: drop get_ovirt_cpu_family > 548a32a6: Allow configuring cpu_model per libvirt VM > > * Mon Jan 16 2017 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.31.1 > 5e647d32: Merge pull request #424 from lveyde/patch-1 > 8629d41f: Return SandyBridge CPU family > > Gal Ben Haim > gbenhaim(a)redhat.com > RHEV DevOps > > _______________________________________________ > lago-devel mailing list > lago-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/lago-devel

2 1

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4807 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 18 Jan '17

18 Jan '17

------=_Part_203_77381758.1484698003059 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4807/, Build Number: 4807, Build Status: FAILURE ------=_Part_203_77381758.1484698003059--

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4805 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 18 Jan '17

18 Jan '17

------=_Part_197_1542439534.1484689510819 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4805/, Build Number: 4805, Build Status: FAILURE ------=_Part_197_1542439534.1484689510819--

1 1

oVirt infra daily report - unstable production jobs - 203
by jenkins＠jenkins.phx.ovirt.org 18 Jan '17

18 Jan '17

------=_Part_199_1205824335.1484694008070 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/203//artifact/exported-a… Cheers, Jenkins ------=_Part_199_1205824335.1484694008070 Content-Type: text/html; charset=us-ascii; name=upstream_report.html Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=upstream_report.html Content-ID: <upstream_report.html> <!DOCTYPE html><head><style type="text/css"> table.gridtable { border-collapse: collapse; table-layout:fixed; width:1600px; font-family: monospace; font-size:13px; } .head { font-size:20px; font-family: arial; } .sub { font-size:18px; background-color:#e5e5e5; font-family: arial; } pre { font-family: monospace; display: inline; white-space: pre-wrap; white-space: -moz-pre-wrap !important; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } </style> </head> <body> <table class="gridtable" border=2> <tr><th colspan=2 class=head> RHEVM CI Jenkins Daily Report - 17/01/2017 </th></tr><tr><th colspan=2 class=sub> <font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font> </th></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_build-…">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_3.6_image-ng-system-tests/">ovirt_3.6_image-ng-system-tests</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.1_system-tests_archived_for_deletion/">ovirt_4.1_system-tests_archived_for_deletion</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.1_system-tests_manual/">ovirt_4.1_system-tests_manual</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt_4.1_system-tests_pre/">ovirt_4.1_system-tests_pre</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_4.1_check-closure_fc24_merged/">repos_4.1_check-closure_fc24_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_master_check-closure_el7_merged/">repos_master_check-closure_el7_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/">system-backup_jenkins_old_ovirt_org</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/">test-repo_ovirt_experimental_4.1</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/">test-repo_ovirt_experimental_master</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> ------=_Part_199_1205824335.1484694008070--

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #308 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 17 Jan '17

17 Jan '17

------=_Part_191_1963885725.1484686820241 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/308/, Build Number: 308, Build Status: FAILURE ------=_Part_191_1963885725.1484686820241--

1 1

[JIRA] (OVIRT-1032) Jenkins repo chek-patch are limited to running on el7 slaves
by Barak Korren (oVirt JIRA) 17 Jan '17

17 Jan '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1032?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-1032: ----------------------------------- Assignee: Barak Korren (was: infra) > Jenkins repo chek-patch are limited to running on el7 slaves > ------------------------------------------------------------ > > Key: OVIRT-1032 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1032 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Jenkins > Reporter: Barak Korren > Assignee: Barak Korren > > There is no reason for it to be like this, better make them run on any slaves. -- This message was sent by Atlassian JIRA (v1000.695.1#100025)

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_4.1 - Build #306 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 17 Jan '17

17 Jan '17

------=_Part_186_1322884875.1484678963709 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/306/, Build Number: 306, Build Status: FAILURE ------=_Part_186_1322884875.1484678963709--

1 1

Build failed in Jenkins: ovirt_3.6_he-system-tests #809
by jenkins＠jenkins.phx.ovirt.org 17 Jan '17

17 Jan '17

See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/changes> Changes: [Roy Golan] accept defaults of aaa-ldap plugin installation [Sandro Bonazzola] 4.1 nightly: publish 4.1 dashboard in 4.1 repo [Eyal Edri] fixing appliance to take from 4.1 [Daniel Belenky] Added 4.1 branch for ovirt-scheduler-proxy [Daniel Belenky] Added build for fc24+ovirt4.0 [Daniel Belenky] Added build for fc24+ovirt4.0 [Daniel Belenky] install python-requests as a dependency [Pavel Zhukov] Moved publishers section to job definition instead of projects [Barak Korren] Revert "install python-requests as a dependency" ------------------------------------------ [...truncated 110.81 KB...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Thu Jan 12 02:56:45 UTC 2017 automation/he_basic_suite_3.6.sh chroot finished @@ took 275 seconds @@ rc = 127 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.3.3 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Thu Jan 12 02:56:51 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 281 seconds ## rc = 127 ########################################################## find: 'logs': No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.clean_rpmdb' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb' './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh' './mock_logs.S2Vzskpt/mocker-epel-7-x86_64.el7.init' -> 'exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init' ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson2934459229741657026.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/809/artifact/exporte…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_3.6_he-system-tests] $ /bin/bash -x /tmp/hudson3269842129067440856.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.3.3 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-test…> + mock_root=epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + [[ -n epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-e30b297ddc7e818ebfbed823dcce8eb8 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

3 10

Build failed in Jenkins: ovirt_4.0_he-system-tests #627
by jenkins＠jenkins.phx.ovirt.org 17 Jan '17

17 Jan '17

See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/changes> Changes: [Lev Veyde] Mask NetworkManager service [Eyal Edri] fix imgbased job names in jjb [Daniel Belenky] fixing jjb version for cockpit-ovirt [Gil Shinar] Add some more 4.1 to experimental [Juan Hernandez] Don't build RPMs for the JBoss modules Maven plugin [pkliczewski] jsonrpc 4.1 branch ------------------------------------------ [...truncated 749 lines...] Finish: shell @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@ Tue Jan 10 02:42:07 UTC 2017 automation/he_basic_suite_4.0.sh chroot finished @@ took 360 seconds @@ rc = 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ========== Scrubbing chroot mock \ --configdir="<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests"> \ --root="mocker-epel-7-x86_64.el7" \ --resultdir="./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.scrub" \ --scrub=chroot WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Start: scrub ['chroot'] INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 Finish: scrub ['chroot'] Finish: run Scrub chroot took 6 seconds ============================ ########################################################## ## Tue Jan 10 02:42:13 UTC 2017 Finished env: el7:epel-7-x86_64 ## took 366 seconds ## rc = 1 ########################################################## find: ‘logs’: No such file or directory No log files found, check command output ##!######################################################## Collecting mock logs ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.clean_rpmdb’ -> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb’ ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’ -> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’ ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.init’ -> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init’ ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson302101162661598371.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/artifact/exporte…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/hudson1888216492513466503.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + mock_root=epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b + [[ -n epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts

6 23