Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32

------=_Part_9639957_1092846933.1468912250886 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Eyal,=20 The last failed run failed on:=20 15:50:02 [ INFO ] Extracting disk image from OVF archive (could take a few = minutes depending on archive size)=20 21:35:04 Build timed out (after 360 minutes). Marking the build as failed.= =20 So it basically got stuck while extracting the OVF image.=20 Some previous runs failed mostly on either:=20 a) broken ovirt-engine-appliance build=20 b) ovirt-engine-appliance missing from the yum repo=20 We need to make sure that the process of building and publishing the ovirt-= engine-appliance works flawlessly e.g. build ovirt-engine, publish it into = the repo so that the build of the appliance can work, then publish it to th= e repo as well.=20 This is extra important as the hosted-engine flow installation will probabl= y become the default one, and without synced ovirt appliance we can't reall= y test the changes in the engine.=20 Thanks in advance,=20 Lev Veyde.=20 ----- Original Message ----- From: "Eyal Edri" <eedri(a)redhat.comTo: jenkins(a)jenkins.phx.ovirt.org=20 Cc: "infra" <infra(a)ovirt.org&gt;, "Lev Veyde" <lveyde(a)redhat.com&gt;, sbonazzo@re= dhat.com=20 Sent: Tuesday, July 19, 2016 8:26:22 AM=20 Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32=20 Lev, this test is a bit flaky going from stable to failure quite often, can= you check what is causing it?=20 On Jul 19, 2016 12:35 AM, < jenkins(a)jenkins.phx.ovirt.org > wrote:=20 See < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes >= =20 Changes:=20 [Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0= =20 [Sandro Bonazzola] vdsm: avoid fc24 out of master=20 [Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing=20 ------------------------------------------=20 [...truncated 620 lines...]=20 WORKSPACE=3D"$PWD"=20 OVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"=20 TESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"=20 rm -rf "$WORKSPACE/exported-artifacts"=20 mkdir -p "$WORKSPACE/exported-artifacts"=20 if [[ -d "$TESTS_LOGS" ]]; then=20 mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"=20 fi=20 [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.= sh=20 + echo shell_scripts/ system_tests.collect_logs.sh=20 shell_scripts/ system_tests.collect_logs.sh=20 + VERSION=3D4.0=20 + SUITE_TYPE=3D=20 + WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ = + OVIRT_SUITE=3D4.0=20 + TESTS_LOGS=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/= ovirt-system-tests/exported-artifacts + rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifa= ct/exported-artifacts + mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/arti= fact/exported-artifacts + [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s= ystem-tests/exported-artifacts > ]]=20 POST BUILD TASK : SUCCESS=20 END OF POST BUILD TASK : 0=20 Match found for :.* : True=20 Logical operation result is TRUE=20 Running script : #!/bin/bash -xe=20 echo "shell-scripts/mock_cleanup.sh"=20 shopt -s nullglob=20 WORKSPACE=3D"$PWD"=20 # Make clear this is the cleanup, helps reading the jenkins logs=20 cat <<EOC=20 _______________________________________________________________________=20 #######################################################################=20 # #=20 # CLEANUP #=20 # #=20 #######################################################################=20 EOC=20 # Archive the logs, we want them anyway=20 logs=3D(=20 ./*log=20 ./*/logs=20 )=20 if [[ "$logs" ]]; then=20 tar cvzf exported-artifacts/logs.tgz "${logs[@]}"=20 rm -rf "${logs[@]}"=20 fi=20 # stop any processes running inside the chroot=20 failed=3Dfalse=20 mock_confs=3D("$WORKSPACE"/*/mocker*)=20 # Clean current jobs mockroot if any=20 for mock_conf_file in "${mock_confs[@]}"; do=20 [[ "$mock_conf_file" ]] || continue=20 echo "Cleaning up mock $mock_conf"=20 mock_root=3D"${mock_conf_file##*/}"=20 mock_root=3D"${mock_root%.*}"=20 my_mock=3D"/usr/bin/mock"=20 my_mock+=3D" --configdir=3D${mock_conf_file%/*}"=20 my_mock+=3D" --root=3D${mock_root}"=20 my_mock+=3D" --resultdir=3D$WORKSPACE"=20 #TODO: investigate why mock --clean fails to umount certain dirs sometimes,= =20 #so we can use it instead of manually doing all this.=20 echo "Killing all mock orphan processes, if any."=20 $my_mock \=20 --orphanskill \=20 || {=20 echo "ERROR: Failed to kill orphans on $chroot."=20 failed=3Dtrue=20 }=20 mock_root=3D"$(\=20 grep \=20 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*" \=20 "$mock_conf_file" \=20 )" || :=20 [[ "$mock_root" ]] || continue=20 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20 if [[ "$mounts" ]]; then=20 echo "Found mounted dirs inside the chroot $chroot. Trying to umount."=20 fi=20 for mount in "${mounts[@]}"; do=20 sudo umount --lazy "$mount" \=20 || {=20 echo "ERROR: Failed to umount $mount."=20 failed=3Dtrue=20 }=20 done=20 done=20 # Clean any leftover chroot from other jobs=20 for mock_root in /var/lib/mock/*; do=20 this_chroot_failed=3Dfalse=20 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20 if [[ "$mounts" ]]; then=20 echo "Found mounted dirs inside the chroot $mock_root." \=20 "Trying to umount."=20 fi=20 for mount in "${mounts[@]}"; do=20 sudo umount --lazy "$mount" \=20 || {=20 echo "ERROR: Failed to umount $mount."=20 failed=3Dtrue=20 this_chroot_failed=3Dtrue=20 }=20 done=20 if ! $this_chroot_failed; then=20 sudo rm -rf "$mock_root"=20 fi=20 done=20 if $failed; then=20 echo "Aborting."=20 exit 1=20 fi=20 # remove mock system cache, we will setup proxies to do the caching and thi= s=20 # takes lots of space between runs=20 shopt -u nullglob=20 sudo rm -Rf /var/cache/mock/*=20 # restore the permissions in the working dir, as sometimes it leaves files= =20 # owned by root and then the 'cleanup workspace' from jenkins job fails to= =20 # clean and breaks the jobs=20 sudo chown -R "$USER" "$WORKSPACE"=20 [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.= sh=20 + echo shell-scripts/mock_cleanup.sh=20 shell-scripts/mock_cleanup.sh=20 + shopt -s nullglob=20 + WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ = + cat=20 _______________________________________________________________________=20 #######################################################################=20 # #=20 # CLEANUP #=20 # #=20 #######################################################################=20 + logs=3D(./*log ./*/logs)=20 + [[ -n ./ovirt-system-tests/logs ]]=20 + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs=20 ./ovirt-system-tests/logs/=20 ./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.= sh/=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s= h/he_basic_suite_4.0.sh.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s= tderr.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo= t.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui= ld.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta= te.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std= out_stderr.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log=20 ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l= og=20 + rm -rf ./ovirt-system-tests/logs=20 + failed=3Dfalse=20 + mock_confs=3D("$WORKSPACE"/*/mocker*)=20 + for mock_conf_file in '"${mock_confs[@]}"'=20 + [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s= ystem-tests/mocker-fedora-23-x86_64.fc23.cfg > ]]=20 + echo 'Cleaning up mock '=20 Cleaning up mock=20 + mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg=20 + mock_root=3Dmocker-fedora-23-x86_64.fc23=20 + my_mock=3D/usr/bin/mock=20 + my_mock+=3D' --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy= stem-tests/ws/ovirt-system-tests '+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'=20 + my_mock+=3D' --resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy= stem-tests/ws/ '+ echo 'Killing all mock orphan processes, if any.'=20 Killing all mock orphan processes, if any.=20 + /usr/bin/mock --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-s= ystem-tests/ws/ovirt-system-tests > --root=3Dmocker-fedora-23-x86_64.fc23 -= -resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >= --orphanskill=20 WARNING: Could not find required logging config file: < http://jenkins.ovir= t.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini .> Us= ing default...=20 INFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...=20 Start: init plugins=20 INFO: selinux enabled=20 Finish: init plugins=20 Start: run=20 WARNING: Process ID 115551 still running in chroot. Killing...=20 WARNING: Process ID 115576 still running in chroot. Killing...=20 WARNING: Process ID 115577 still running in chroot. Killing...=20 WARNING: Process ID 115578 still running in chroot. Killing...=20 WARNING: Process ID 116634 still running in chroot. Killing...=20 Finish: run=20 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' < http://= jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocke= r-fedora-23-x86_64.fc23.cfg + mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20 + [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]=20 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20 ++ mount=20 ++ awk '{print $3}'=20 ++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20 + [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/proc ]]=20 + echo 'Found mounted dirs inside the chroot . Trying to umount.'=20 Found mounted dirs inside the chroot . Trying to umount.=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/proc=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/sys=20 sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/dev/shm=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/dev/pts=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/var/cache/yum=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-test= s/ws/ovirt-system-tests + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/run/libvirt=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/var/lib/lago=20 umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/var/lib/lago: not mounted=20 + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa= 4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'=20 ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8= b988eb73b2ad-91644/root/var/lib/lago.=20 + failed=3Dtrue=20 + for mount in '"${mounts[@]}"'=20 + sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/proc/filesystems=20 umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/proc/filesystems: mountpoint not found=20 + echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa= 4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'=20 ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8= b988eb73b2ad-91644/root/proc/filesystems.=20 + failed=3Dtrue=20 + for mock_root in '/var/lib/mock/*'=20 + this_chroot_failed=3Dfalse=20 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20 ++ mount=20 ++ awk '{print $3}'=20 ++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44=20 + :=20 + [[ -n '' ]]=20 + false=20 + sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2= ad-91644=20 + true=20 + echo Aborting.=20 Aborting.=20 + exit 1=20 POST BUILD TASK : FAILURE=20 END OF POST BUILD TASK : 1=20 Recording test results=20 ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t= est report files were found. Configuration error?=20 Archiving artifacts=20 _______________________________________________=20 Infra mailing list=20 Infra(a)ovirt.org=20 http://lists.ovirt.org/mailman/listinfo/infra=20 ------=_Part_9639957_1092846933.1468912250886 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: times new roman, new york, times, se= rif; font-size: 12pt; color: #000000"><div>Hi Eyal,<br></div><div><br></div= b>15:50:02</b> </span>[ INFO ] Extracting disk image from OVF archive (coul= d take a few minutes depending on archive size)</div><div><span class=3D"ti= mestamp"><b>21:35:04</b> </span>Build timed out (after 360 minutes). Markin= g the build as failed.<br></div><div><br></div><div>So it basically got stu= ck while extracting the OVF image.<br></div><div><br></div><div>Some previo= us runs failed mostly on either:<br></div><div>a) broken ovirt-engine-appli= ance build<br></div><div>b) ovirt-engine-appliance missing from the yum rep= o<br></div><div><br></div><div>We need to make sure that the process of bui= lding and publishing the ovirt-engine-appliance works flawlessly e.g. build= ovirt-engine, publish it into the repo so that the build of the appliance = can work, then publish it to the repo as well.<br></div><div>This is extra = important as the hosted-engine flow installation will probably become the d= efault one, and without synced ovirt appliance we can't really test the cha= nges in the engine.<br></div><div><br></div><div><span name=3D"x"></span>Th= anks in advance,<br>Lev Veyde.<span name=3D"x"></span><br></div><div><br></= div><hr id=3D"zwchr"><div style=3D"color:#000;font-weight:normal;font-style= :normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-si= ze:12pt;"><b>From: </b>"Eyal Edri" &amp;lt;eedri(a)redhat.com&amp;gt;&lt;br&gt;&lt;b&gt;To: </b>j= enkins(a)jenkins.phx.ovirt.org&lt;br&gt;&lt;b&gt;Cc: </b>"infra" &amp;lt;infra(a)ovirt.org&amp;gt;,= "Lev Veyde" &amp;lt;lveyde(a)redhat.com&amp;gt;, sbonazzo(a)redhat.com&lt;br&gt;&lt;b&gt;Sent: </b= enkins: ovirt_4.0_he-system-tests #32<br><div><br></div><p dir=3D"ltr">Lev,= &nbsp; this test is a bit flaky going from stable to failure quite often,&n= bsp; can you check what is causing it? </p<div class=3D"gmail_quote">On Jul 19, 2016 12:35 AM, &lt;<a href=3D"mailto= :jenkins@jenkins.phx.ovirt.org" target=3D&quot;_blank&quot;&gt;jenkins(a)jenkins.phx.ovirt= .org</a>&gt; wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">See &lt;<a href=3D"ht= tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes" rel=3D"nor= eferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-... -tests/32/changes</a>&gt;<br<brChanges:<br<br[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0= <br<br[Sandro Bonazzola] vdsm: avoid fc24 out of master<br<br[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing<br<br------------------------------------------<br[...truncated 620 lines...]<br<brWORKSPACE=3D"$PWD"<brOVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"<brTESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"<br<brrm -rf "$WORKSPACE/exported-artifacts"<brmkdir -p "$WORKSPACE/exported-artifacts"<br<brif [[ -d "$TESTS_LOGS" ]]; then<br&nbsp; &nbsp; mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"<brfi<br<br[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.= sh<br+ echo shell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D= "noreferrer" target=3D"_blank">system_tests.collect_logs.sh</a><brshell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D"norefe= rrer" target=3D"_blank">system_tests.collect_logs.sh</a><br+ VERSION=3D4.0<br+ SUITE_TYPE=3D<br+ WORKSPACE=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst= em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org= /job/ovirt_4.0_he-system-tests/ws/</a>&gt;<br+ OVIRT_SUITE=3D4.0<br+ TESTS_LOGS=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-sys= tem-tests/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" targ= et=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-syst... rt-system-tests/exported-artifacts</a>&gt;<br+ rm -rf &lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-te= sts/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">ht= tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-a= rtifacts</a>&gt;<br+ mkdir -p &lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-= tests/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">= http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported= -artifacts</a>&gt;<br+ [[ -d &lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes= ts/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" target=3D"_= blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/... em-tests/exported-artifacts</a>&gt; ]]<brPOST BUILD TASK : SUCCESS<brEND OF POST BUILD TASK : 0<brMatch found for :.* : True<brLogical operation result is TRUE<brRunning script&nbsp; : #!/bin/bash -xe<brecho "shell-scripts/mock_cleanup.sh"<br<brshopt -s nullglob<br<br<brWORKSPACE=3D"$PWD"<br<br# Make clear this is the cleanup, helps reading the jenkins logs<brcat &lt;&lt;EOC<br_______________________________________________________________________<br#######################################################################<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &n= bsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; = &nbsp; &nbsp; &nbsp;#<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;CLEANUP&nbsp; &nbsp; &nbsp; &nbsp; &nb= sp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &= nbsp;#<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &n= bsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; = &nbsp; &nbsp; &nbsp;#<br#######################################################################<brEOC<br<br<br# Archive the logs, we want them anyway<brlogs=3D(<br&nbsp; &nbsp; ./*log<br&nbsp; &nbsp; ./*/logs<br)<brif [[ "$logs" ]]; then<br&nbsp; &nbsp; tar cvzf exported-artifacts/logs.tgz "${logs[@]}"<br&nbsp; &nbsp; rm -rf "${logs[@]}"<brfi<br<br# stop any processes running inside the chroot<brfailed=3Dfalse<brmock_confs=3D("$WORKSPACE"/*/mocker*)<br# Clean current jobs mockroot if any<brfor mock_conf_file in "${mock_confs[@]}"; do<br&nbsp; &nbsp; [[ "$mock_conf_file" ]] || continue<br&nbsp; &nbsp; echo "Cleaning up mock $mock_conf"<br&nbsp; &nbsp; mock_root=3D"${mock_conf_file##*/}"<br&nbsp; &nbsp; mock_root=3D"${mock_root%.*}"<br&nbsp; &nbsp; my_mock=3D"/usr/bin/mock"<br&nbsp; &nbsp; my_mock+=3D" --configdir=3D${mock_conf_file%/*}"<br&nbsp; &nbsp; my_mock+=3D" --root=3D${mock_root}"<br&nbsp; &nbsp; my_mock+=3D" --resultdir=3D$WORKSPACE"<br<br&nbsp; &nbsp; #TODO: investigate why mock --clean fails to umount certain d= irs sometimes,<br&nbsp; &nbsp; #so we can use it instead of manually doing all this.<br&nbsp; &nbsp; echo "Killing all mock orphan processes, if any."<br&nbsp; &nbsp; $my_mock \<br&nbsp; &nbsp; &nbsp; &nbsp; --orphanskill \<br&nbsp; &nbsp; || {<br&nbsp; &nbsp; &nbsp; &nbsp; echo "ERROR:&nbsp; Failed to kill orphans on $c= hroot."<br&nbsp; &nbsp; &nbsp; &nbsp; failed=3Dtrue<br&nbsp; &nbsp; }<br<br&nbsp; &nbsp; mock_root=3D"$(\<br&nbsp; &nbsp; &nbsp; &nbsp; grep \<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -Po "(?&lt;=3Dconfig_opts\['root'= \] =3D ')[^']*" \<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "$mock_conf_file" \<br&nbsp; &nbsp; )" || :<br&nbsp; &nbsp; [[ "$mock_root" ]] || continue<br&nbsp; &nbsp; mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||= :<br&nbsp; &nbsp; if [[ "$mounts" ]]; then<br&nbsp; &nbsp; &nbsp; &nbsp; echo "Found mounted dirs inside the chroot $chr= oot. Trying to umount."<br&nbsp; &nbsp; fi<br&nbsp; &nbsp; for mount in "${mounts[@]}"; do<br&nbsp; &nbsp; &nbsp; &nbsp; sudo umount --lazy "$mount" \<br&nbsp; &nbsp; &nbsp; &nbsp; || {<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; echo "ERROR:&nbsp; Failed to umou= nt $mount."<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; failed=3Dtrue<br&nbsp; &nbsp; &nbsp; &nbsp; }<br&nbsp; &nbsp; done<brdone<br<br# Clean any leftover chroot from other jobs<brfor mock_root in /var/lib/mock/*; do<br&nbsp; &nbsp; this_chroot_failed=3Dfalse<br&nbsp; &nbsp; mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||= :<br&nbsp; &nbsp; if [[ "$mounts" ]]; then<br&nbsp; &nbsp; &nbsp; &nbsp; echo "Found mounted dirs inside the chroot $moc= k_root." \<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Trying to umount."<br&nbsp; &nbsp; fi<br&nbsp; &nbsp; for mount in "${mounts[@]}"; do<br&nbsp; &nbsp; &nbsp; &nbsp; sudo umount --lazy "$mount" \<br&nbsp; &nbsp; &nbsp; &nbsp; || {<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; echo "ERROR:&nbsp; Failed to umou= nt $mount."<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; failed=3Dtrue<br&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; this_chroot_failed=3Dtrue<br&nbsp; &nbsp; &nbsp; &nbsp; }<br&nbsp; &nbsp; done<br&nbsp; &nbsp; if ! $this_chroot_failed; then<br&nbsp; &nbsp; &nbsp; &nbsp; sudo rm -rf "$mock_root"<br&nbsp; &nbsp; fi<brdone<br<brif $failed; then<br&nbsp; &nbsp; echo "Aborting."<br&nbsp; &nbsp; exit 1<brfi<br<br# remove mock system cache, we will setup proxies to do the caching and thi= s<br# takes lots of space between runs<brshopt -u nullglob<brsudo rm -Rf /var/cache/mock/*<br<br# restore the permissions in the working dir, as sometimes it leaves files<= br# owned by root and then the 'cleanup workspace' from jenkins job fails to<= br# clean and breaks the jobs<brsudo chown -R "$USER" "$WORKSPACE"<br<br[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.= sh<br+ echo shell-scripts/mock_cleanup.sh<brshell-scripts/mock_cleanup.sh<br+ shopt -s nullglob<br+ WORKSPACE=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst= em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org= /job/ovirt_4.0_he-system-tests/ws/</a>&gt;<br+ cat<br_______________________________________________________________________<br#######################################################################<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &n= bsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; = &nbsp; &nbsp; &nbsp;#<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;CLEANUP&nbsp; &nbsp; &nbsp; &nbsp; &nb= sp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &= nbsp;#<br#&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs= p; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &n= bsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; = &nbsp; &nbsp; &nbsp;#<br#######################################################################<br+ logs=3D(./*log ./*/logs)<br+ [[ -n ./ovirt-system-tests/logs ]]<br+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs<br./ovirt-system-tests/logs/<br./ovirt-system-tests/logs/<a href=3D"http://mocker-fedora-23-x86_64.fc23.he= _basic_suite_4.0.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s= h/he_basic_suite_4.0.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s= tderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo= t.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui= ld.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta= te.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std= out_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/" rel=3D"norefe= rrer" target=3D"_blank">mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/= <br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s= h/he_basic_suite_4.0.sh.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s= tderr.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/<br= ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo= t.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui= ld.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta= te.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std= out_stderr.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/</a><br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log<br./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l= og<br+ rm -rf ./ovirt-system-tests/logs<br+ failed=3Dfalse<br+ mock_confs=3D("$WORKSPACE"/*/mocker*)<br+ for mock_conf_file in '"${mock_confs[@]}"'<br+ [[ -n &lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes= ts/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferre= r" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-... /ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg</a>&gt; ]]<br+ echo 'Cleaning up mock '<brCleaning up mock<br+ mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg<br+ mock_root=3Dmocker-fedora-23-x86_64.fc23<br+ my_mock=3D/usr/bin/mock<br+ my_mock+=3D' --configdir=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/ov= irt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D= "_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-sy= stem-tests</a>'&gt;<br+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'<br+ my_mock+=3D' --resultdir=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/ov= irt_4.0_he-system-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://je= nkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/</a>'&gt;&...+ echo 'Killing all mock orphan processes, if any.'<brKilling all mock orphan processes, if any.<br+ /usr/bin/mock --configdir=3D&lt;<a href=3D"http://jenkins.ovirt.org/job/o= virt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target= =3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt= -system-tests</a>&gt; --root=3Dmocker-fedora-23-x86_64.fc23 --resultdir=3D&= lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/&q... r= el=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_= he-system-tests/ws/</a>&gt; --orphanskill<brWARNING: Could not find required logging config file: &lt;<a href=3D"http:/= /jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logg= ing.ini" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/= ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini</a>.&gt; Using = default...<brINFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...<brStart: init plugins<brINFO: selinux enabled<brFinish: init plugins<brStart: run<brWARNING: Process ID 115551 still running in chroot. Killing...<brWARNING: Process ID 115576 still running in chroot. Killing...<brWARNING: Process ID 115577 still running in chroot. Killing...<brWARNING: Process ID 115578 still running in chroot. Killing...<brWARNING: Process ID 116634 still running in chroot. Killing...<brFinish: run<br++ grep -Po '(?&lt;=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' &lt;<a= href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ov... stem-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferrer" target=3D"_= blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/... em-tests/mocker-fedora-23-x86_64.fc23.cfg</a>&gt;<br+ mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]<br+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br++ mount<br++ awk '{print $3}'<br++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br+ [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/proc ]]<br+ echo 'Found mounted dirs inside the chroot . Trying to umount.'<brFound mounted dirs inside the chroot . Trying to umount.<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/proc<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/sys<brsh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/dev/shm<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/dev/pts<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/var/cache/yum<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root&lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_h= e-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D"_blank">= http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test= s</a>&gt;<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/run/libvirt<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/var/lib/lago<brumount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/var/lib/lago: not mounted<br+ echo 'ERROR:&nbsp; Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621= 56a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'<brERROR:&nbsp; Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a= 935ea8b988eb73b2ad-91644/root/var/lib/lago.<br+ failed=3Dtrue<br+ for mount in '"${mounts[@]}"'<br+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98= 8eb73b2ad-91644/root/proc/filesystems<brumount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44/root/proc/filesystems: mountpoint not found<br+ echo 'ERROR:&nbsp; Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621= 56a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'<brERROR:&nbsp; Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a= 935ea8b988eb73b2ad-91644/root/proc/filesystems.<br+ failed=3Dtrue<br+ for mock_root in '/var/lib/mock/*'<br+ this_chroot_failed=3Dfalse<br+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br++ mount<br++ awk '{print $3}'<br++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916= 44<br+ :<br+ [[ -n '' ]]<br+ false<br+ sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2= ad-91644<br+ true<br+ echo Aborting.<brAborting.<br+ exit 1<brPOST BUILD TASK : FAILURE<brEND OF POST BUILD TASK : 1<brRecording test results<brERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t= est report files were found. Configuration error?<brArchiving artifacts<br_______________________________________________<brInfra mailing list<br<a href=3D"mailto:Infra@ovirt.org" target=3D&quot;_blank&quot;&gt;Infra(a)ovirt.org&lt;/a&gt;&lt;br= <a href=3D"http://lists.ovirt.org/mailman/listinfo/infra" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/in...<br<br</blockquote></div</div><div><br></div></div></body></html------=_Part_9639957_1092846933.1468912250886--