Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32

------=_Part_9639957_1092846933.1468912250886
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Eyal,=20

The last failed run failed on:=20
15:50:02 [ INFO ] Extracting disk image from OVF archive (could take a few =
minutes depending on archive size)=20
21:35:04 Build timed out (after 360 minutes). Marking the build as failed.=
=20

So it basically got stuck while extracting the OVF image.=20

Some previous runs failed mostly on either:=20
a) broken ovirt-engine-appliance build=20
b) ovirt-engine-appliance missing from the yum repo=20

We need to make sure that the process of building and publishing the ovirt-=
engine-appliance works flawlessly e.g. build ovirt-engine, publish it into =
the repo so that the build of the appliance can work, then publish it to th=
e repo as well.=20
This is extra important as the hosted-engine flow installation will probabl=
y become the default one, and without synced ovirt appliance we can't reall=
y test the changes in the engine.=20

Thanks in advance,=20
Lev Veyde.=20

----- Original Message -----

From: "Eyal Edri" <eedri@redhat.com>=20
To: jenkins@jenkins.phx.ovirt.org=20
Cc: "infra" <infra@ovirt.org>, "Lev Veyde" <lveyde@redhat.com>, sbonazzo@re=
dhat.com=20
Sent: Tuesday, July 19, 2016 8:26:22 AM=20
Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #32=20



Lev, this test is a bit flaky going from stable to failure quite often, can=
 you check what is causing it?=20
On Jul 19, 2016 12:35 AM, < jenkins@jenkins.phx.ovirt.org > wrote:=20


See < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes >=
=20

Changes:=20

[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0=
=20

[Sandro Bonazzola] vdsm: avoid fc24 out of master=20

[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing=20

------------------------------------------=20
[...truncated 620 lines...]=20

WORKSPACE=3D"$PWD"=20
OVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"=20
TESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"=20

rm -rf "$WORKSPACE/exported-artifacts"=20
mkdir -p "$WORKSPACE/exported-artifacts"=20

if [[ -d "$TESTS_LOGS" ]]; then=20
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"=20
fi=20

[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.=
sh=20
+ echo shell_scripts/ system_tests.collect_logs.sh=20
shell_scripts/ system_tests.collect_logs.sh=20
+ VERSION=3D4.0=20
+ SUITE_TYPE=3D=20
+ WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ =
>=20
+ OVIRT_SUITE=3D4.0=20
+ TESTS_LOGS=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/=
ovirt-system-tests/exported-artifacts >=20
+ rm -rf < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifa=
ct/exported-artifacts >=20
+ mkdir -p < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/arti=
fact/exported-artifacts >=20
+ [[ -d < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s=
ystem-tests/exported-artifacts > ]]=20
POST BUILD TASK : SUCCESS=20
END OF POST BUILD TASK : 0=20
Match found for :.* : True=20
Logical operation result is TRUE=20
Running script : #!/bin/bash -xe=20
echo "shell-scripts/mock_cleanup.sh"=20

shopt -s nullglob=20


WORKSPACE=3D"$PWD"=20

# Make clear this is the cleanup, helps reading the jenkins logs=20
cat <<EOC=20
_______________________________________________________________________=20
#######################################################################=20
# #=20
# CLEANUP #=20
# #=20
#######################################################################=20
EOC=20


# Archive the logs, we want them anyway=20
logs=3D(=20
./*log=20
./*/logs=20
)=20
if [[ "$logs" ]]; then=20
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"=20
rm -rf "${logs[@]}"=20
fi=20

# stop any processes running inside the chroot=20
failed=3Dfalse=20
mock_confs=3D("$WORKSPACE"/*/mocker*)=20
# Clean current jobs mockroot if any=20
for mock_conf_file in "${mock_confs[@]}"; do=20
[[ "$mock_conf_file" ]] || continue=20
echo "Cleaning up mock $mock_conf"=20
mock_root=3D"${mock_conf_file##*/}"=20
mock_root=3D"${mock_root%.*}"=20
my_mock=3D"/usr/bin/mock"=20
my_mock+=3D" --configdir=3D${mock_conf_file%/*}"=20
my_mock+=3D" --root=3D${mock_root}"=20
my_mock+=3D" --resultdir=3D$WORKSPACE"=20

#TODO: investigate why mock --clean fails to umount certain dirs sometimes,=
=20
#so we can use it instead of manually doing all this.=20
echo "Killing all mock orphan processes, if any."=20
$my_mock \=20
--orphanskill \=20
|| {=20
echo "ERROR: Failed to kill orphans on $chroot."=20
failed=3Dtrue=20
}=20

mock_root=3D"$(\=20
grep \=20
-Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*" \=20
"$mock_conf_file" \=20
)" || :=20
[[ "$mock_root" ]] || continue=20
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20
if [[ "$mounts" ]]; then=20
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."=20
fi=20
for mount in "${mounts[@]}"; do=20
sudo umount --lazy "$mount" \=20
|| {=20
echo "ERROR: Failed to umount $mount."=20
failed=3Dtrue=20
}=20
done=20
done=20

# Clean any leftover chroot from other jobs=20
for mock_root in /var/lib/mock/*; do=20
this_chroot_failed=3Dfalse=20
mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || :=20
if [[ "$mounts" ]]; then=20
echo "Found mounted dirs inside the chroot $mock_root." \=20
"Trying to umount."=20
fi=20
for mount in "${mounts[@]}"; do=20
sudo umount --lazy "$mount" \=20
|| {=20
echo "ERROR: Failed to umount $mount."=20
failed=3Dtrue=20
this_chroot_failed=3Dtrue=20
}=20
done=20
if ! $this_chroot_failed; then=20
sudo rm -rf "$mock_root"=20
fi=20
done=20

if $failed; then=20
echo "Aborting."=20
exit 1=20
fi=20

# remove mock system cache, we will setup proxies to do the caching and thi=
s=20
# takes lots of space between runs=20
shopt -u nullglob=20
sudo rm -Rf /var/cache/mock/*=20

# restore the permissions in the working dir, as sometimes it leaves files=
=20
# owned by root and then the 'cleanup workspace' from jenkins job fails to=
=20
# clean and breaks the jobs=20
sudo chown -R "$USER" "$WORKSPACE"=20

[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.=
sh=20
+ echo shell-scripts/mock_cleanup.sh=20
shell-scripts/mock_cleanup.sh=20
+ shopt -s nullglob=20
+ WORKSPACE=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ =
>=20
+ cat=20
_______________________________________________________________________=20
#######################################################################=20
# #=20
# CLEANUP #=20
# #=20
#######################################################################=20
+ logs=3D(./*log ./*/logs)=20
+ [[ -n ./ovirt-system-tests/logs ]]=20
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs=20
./ovirt-system-tests/logs/=20
./ovirt-system-tests/logs/ mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.=
sh/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log=20
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l=
og=20
+ rm -rf ./ovirt-system-tests/logs=20
+ failed=3Dfalse=20
+ mock_confs=3D("$WORKSPACE"/*/mocker*)=20
+ for mock_conf_file in '"${mock_confs[@]}"'=20
+ [[ -n < http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-s=
ystem-tests/mocker-fedora-23-x86_64.fc23.cfg > ]]=20
+ echo 'Cleaning up mock '=20
Cleaning up mock=20
+ mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg=20
+ mock_root=3Dmocker-fedora-23-x86_64.fc23=20
+ my_mock=3D/usr/bin/mock=20
+ my_mock+=3D' --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy=
stem-tests/ws/ovirt-system-tests '>=20
+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'=20
+ my_mock+=3D' --resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-sy=
stem-tests/ws/ '>=20
+ echo 'Killing all mock orphan processes, if any.'=20
Killing all mock orphan processes, if any.=20
+ /usr/bin/mock --configdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-s=
ystem-tests/ws/ovirt-system-tests > --root=3Dmocker-fedora-23-x86_64.fc23 -=
-resultdir=3D< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >=
 --orphanskill=20
WARNING: Could not find required logging config file: < http://jenkins.ovir=
t.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini .> Us=
ing default...=20
INFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...=20
Start: init plugins=20
INFO: selinux enabled=20
Finish: init plugins=20
Start: run=20
WARNING: Process ID 115551 still running in chroot. Killing...=20
WARNING: Process ID 115576 still running in chroot. Killing...=20
WARNING: Process ID 115577 still running in chroot. Killing...=20
WARNING: Process ID 115578 still running in chroot. Killing...=20
WARNING: Process ID 116634 still running in chroot. Killing...=20
Finish: run=20
++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' < http://=
jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/mocke=
r-fedora-23-x86_64.fc23.cfg >=20
+ mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20
+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]=20
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20
++ mount=20
++ awk '{print $3}'=20
++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad=20
+ [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc ]]=20
+ echo 'Found mounted dirs inside the chroot . Trying to umount.'=20
Found mounted dirs inside the chroot . Trying to umount.=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/sys=20
sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/shm=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/pts=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/cache/yum=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root< http://jenkins.ovirt.org/job/ovirt_4.0_he-system-test=
s/ws/ovirt-system-tests >=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/run/libvirt=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/lib/lago=20
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/var/lib/lago: not mounted=20
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa=
4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'=20
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8=
b988eb73b2ad-91644/root/var/lib/lago.=20
+ failed=3Dtrue=20
+ for mount in '"${mounts[@]}"'=20
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc/filesystems=20
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc/filesystems: mountpoint not found=20
+ echo 'ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa=
4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'=20
ERROR: Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8=
b988eb73b2ad-91644/root/proc/filesystems.=20
+ failed=3Dtrue=20
+ for mock_root in '/var/lib/mock/*'=20
+ this_chroot_failed=3Dfalse=20
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))=20
++ mount=20
++ awk '{print $3}'=20
++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44=20
+ :=20
+ [[ -n '' ]]=20
+ false=20
+ sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2=
ad-91644=20
+ true=20
+ echo Aborting.=20
Aborting.=20
+ exit 1=20
POST BUILD TASK : FAILURE=20
END OF POST BUILD TASK : 1=20
Recording test results=20
ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t=
est report files were found. Configuration error?=20
Archiving artifacts=20
_______________________________________________=20
Infra mailing list=20
Infra@ovirt.org=20
http://lists.ovirt.org/mailman/listinfo/infra=20







------=_Part_9639957_1092846933.1468912250886
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: times new roman, new york, times, se=
rif; font-size: 12pt; color: #000000"><div>Hi Eyal,<br></div><div><br></div=
><div>The last failed run failed on: </div><div><span class=3D"timestamp"><=
b>15:50:02</b> </span>[ INFO ] Extracting disk image from OVF archive (coul=
d take a few minutes depending on archive size)</div><div><span class=3D"ti=
mestamp"><b>21:35:04</b> </span>Build timed out (after 360 minutes). Markin=
g the build as failed.<br></div><div><br></div><div>So it basically got stu=
ck while extracting the OVF image.<br></div><div><br></div><div>Some previo=
us runs failed mostly on either:<br></div><div>a) broken ovirt-engine-appli=
ance build<br></div><div>b) ovirt-engine-appliance missing from the yum rep=
o<br></div><div><br></div><div>We need to make sure that the process of bui=
lding and publishing the ovirt-engine-appliance works flawlessly e.g. build=
 ovirt-engine, publish it into the repo so that the build of the appliance =
can work, then publish it to the repo as well.<br></div><div>This is extra =
important as the hosted-engine flow installation will probably become the d=
efault one, and without synced ovirt appliance we can't really test the cha=
nges in the engine.<br></div><div><br></div><div><span name=3D"x"></span>Th=
anks in advance,<br>Lev Veyde.<span name=3D"x"></span><br></div><div><br></=
div><hr id=3D"zwchr"><div style=3D"color:#000;font-weight:normal;font-style=
:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-si=
ze:12pt;"><b>From: </b>"Eyal Edri" <eedri@redhat.com><br><b>To: </b>j=
enkins@jenkins.phx.ovirt.org<br><b>Cc: </b>"infra" <infra@ovirt.org>,=
 "Lev Veyde" <lveyde@redhat.com>, sbonazzo@redhat.com<br><b>Sent: </b=
>Tuesday, July 19, 2016 8:26:22 AM<br><b>Subject: </b>Re: Build failed in J=
enkins: ovirt_4.0_he-system-tests #32<br><div><br></div><p dir=3D"ltr">Lev,=
  this test is a bit flaky going from stable to failure quite often,&n=
bsp; can you check what is causing it? </p>
<div class=3D"gmail_quote">On Jul 19, 2016 12:35 AM,  <<a href=3D"mailto=
:jenkins@jenkins.phx.ovirt.org" target=3D"_blank">jenkins@jenkins.phx.ovirt=
.org</a>> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">See <<a href=3D"ht=
tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/changes" rel=3D"nor=
eferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system=
-tests/32/changes</a>><br>
<br>
Changes:<br>
<br>
[Lev Veyde] ovirt-system-tests: Add automation for he_iscsi_basic_suite_4.0=
<br>
<br>
[Sandro Bonazzola] vdsm: avoid fc24 out of master<br>
<br>
[Sandro Bonazzola] ovirt-engine: add 3.6.8 branch testing<br>
<br>
------------------------------------------<br>
[...truncated 620 lines...]<br>
<br>
WORKSPACE=3D"$PWD"<br>
OVIRT_SUITE=3D"$SUITE_TYPE_suite_$VERSION"<br>
TESTS_LOGS=3D"$WORKSPACE/ovirt-system-tests/exported-artifacts"<br>
<br>
rm -rf "$WORKSPACE/exported-artifacts"<br>
mkdir -p "$WORKSPACE/exported-artifacts"<br>
<br>
if [[ -d "$TESTS_LOGS" ]]; then<br>
    mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"<br>
fi<br>
<br>
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson1764906258788527221.=
sh<br>
+ echo shell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D=
"noreferrer" target=3D"_blank">system_tests.collect_logs.sh</a><br>
shell_scripts/<a href=3D"http://system_tests.collect_logs.sh" rel=3D"norefe=
rrer" target=3D"_blank">system_tests.collect_logs.sh</a><br>
+ VERSION=3D4.0<br>
+ SUITE_TYPE=3D<br>
+ WORKSPACE=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst=
em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org=
/job/ovirt_4.0_he-system-tests/ws/</a>><br>
+ OVIRT_SUITE=3D4.0<br>
+ TESTS_LOGS=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-sys=
tem-tests/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" targ=
et=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovi=
rt-system-tests/exported-artifacts</a>><br>
+ rm -rf <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-te=
sts/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">ht=
tp://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported-a=
rtifacts</a>><br>
+ mkdir -p <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-=
tests/32/artifact/exported-artifacts" rel=3D"noreferrer" target=3D"_blank">=
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/32/artifact/exported=
-artifacts</a>><br>
+ [[ -d <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes=
ts/ws/ovirt-system-tests/exported-artifacts" rel=3D"noreferrer" target=3D"_=
blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-syst=
em-tests/exported-artifacts</a>> ]]<br>
POST BUILD TASK : SUCCESS<br>
END OF POST BUILD TASK : 0<br>
Match found for :.* : True<br>
Logical operation result is TRUE<br>
Running script  : #!/bin/bash -xe<br>
echo "shell-scripts/mock_cleanup.sh"<br>
<br>
shopt -s nullglob<br>
<br>
<br>
WORKSPACE=3D"$PWD"<br>
<br>
# Make clear this is the cleanup, helps reading the jenkins logs<br>
cat <<EOC<br>
_______________________________________________________________________<br>
#######################################################################<br>
#                    &nbs=
p;                     &n=
bsp;                     =
     #<br>
#                    &nbs=
p;          CLEANUP        &nb=
sp;                     &=
nbsp;#<br>
#                    &nbs=
p;                     &n=
bsp;                     =
     #<br>
#######################################################################<br>
EOC<br>
<br>
<br>
# Archive the logs, we want them anyway<br>
logs=3D(<br>
    ./*log<br>
    ./*/logs<br>
)<br>
if [[ "$logs" ]]; then<br>
    tar cvzf exported-artifacts/logs.tgz "${logs[@]}"<br>
    rm -rf "${logs[@]}"<br>
fi<br>
<br>
# stop any processes running inside the chroot<br>
failed=3Dfalse<br>
mock_confs=3D("$WORKSPACE"/*/mocker*)<br>
# Clean current jobs mockroot if any<br>
for mock_conf_file in "${mock_confs[@]}"; do<br>
    [[ "$mock_conf_file" ]] || continue<br>
    echo "Cleaning up mock $mock_conf"<br>
    mock_root=3D"${mock_conf_file##*/}"<br>
    mock_root=3D"${mock_root%.*}"<br>
    my_mock=3D"/usr/bin/mock"<br>
    my_mock+=3D" --configdir=3D${mock_conf_file%/*}"<br>
    my_mock+=3D" --root=3D${mock_root}"<br>
    my_mock+=3D" --resultdir=3D$WORKSPACE"<br>
<br>
    #TODO: investigate why mock --clean fails to umount certain d=
irs sometimes,<br>
    #so we can use it instead of manually doing all this.<br>
    echo "Killing all mock orphan processes, if any."<br>
    $my_mock \<br>
        --orphanskill \<br>
    || {<br>
        echo "ERROR:  Failed to kill orphans on $c=
hroot."<br>
        failed=3Dtrue<br>
    }<br>
<br>
    mock_root=3D"$(\<br>
        grep \<br>
            -Po "(?<=3Dconfig_opts\['root'=
\] =3D ')[^']*" \<br>
            "$mock_conf_file" \<br>
    )" || :<br>
    [[ "$mock_root" ]] || continue<br>
    mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||=
 :<br>
    if [[ "$mounts" ]]; then<br>
        echo "Found mounted dirs inside the chroot $chr=
oot. Trying to umount."<br>
    fi<br>
    for mount in "${mounts[@]}"; do<br>
        sudo umount --lazy "$mount" \<br>
        || {<br>
            echo "ERROR:  Failed to umou=
nt $mount."<br>
            failed=3Dtrue<br>
        }<br>
    done<br>
done<br>
<br>
# Clean any leftover chroot from other jobs<br>
for mock_root in /var/lib/mock/*; do<br>
    this_chroot_failed=3Dfalse<br>
    mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) ||=
 :<br>
    if [[ "$mounts" ]]; then<br>
        echo "Found mounted dirs inside the chroot $moc=
k_root." \<br>
             "Trying to umount."<br>
    fi<br>
    for mount in "${mounts[@]}"; do<br>
        sudo umount --lazy "$mount" \<br>
        || {<br>
            echo "ERROR:  Failed to umou=
nt $mount."<br>
            failed=3Dtrue<br>
            this_chroot_failed=3Dtrue<br>
        }<br>
    done<br>
    if ! $this_chroot_failed; then<br>
        sudo rm -rf "$mock_root"<br>
    fi<br>
done<br>
<br>
if $failed; then<br>
    echo "Aborting."<br>
    exit 1<br>
fi<br>
<br>
# remove mock system cache, we will setup proxies to do the caching and thi=
s<br>
# takes lots of space between runs<br>
shopt -u nullglob<br>
sudo rm -Rf /var/cache/mock/*<br>
<br>
# restore the permissions in the working dir, as sometimes it leaves files<=
br>
# owned by root and then the 'cleanup workspace' from jenkins job fails to<=
br>
# clean and breaks the jobs<br>
sudo chown -R "$USER" "$WORKSPACE"<br>
<br>
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5198775129414653216.=
sh<br>
+ echo shell-scripts/mock_cleanup.sh<br>
shell-scripts/mock_cleanup.sh<br>
+ shopt -s nullglob<br>
+ WORKSPACE=3D<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-syst=
em-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org=
/job/ovirt_4.0_he-system-tests/ws/</a>><br>
+ cat<br>
_______________________________________________________________________<br>
#######################################################################<br>
#                    &nbs=
p;                     &n=
bsp;                     =
     #<br>
#                    &nbs=
p;          CLEANUP        &nb=
sp;                     &=
nbsp;#<br>
#                    &nbs=
p;                     &n=
bsp;                     =
     #<br>
#######################################################################<br>
+ logs=3D(./*log ./*/logs)<br>
+ [[ -n ./ovirt-system-tests/logs ]]<br>
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs<br>
./ovirt-system-tests/logs/<br>
./ovirt-system-tests/logs/<a href=3D"http://mocker-fedora-23-x86_64.fc23.he=
_basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/" rel=3D"norefe=
rrer" target=3D"_blank">mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.sh/=
<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.he_basic_suite_4.0.s=
h/he_basic_suite_4.0.sh.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_s=
tderr.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/<br=
>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/roo=
t.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/bui=
ld.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/sta=
te.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/std=
out_stderr.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/</a><br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log<br>
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.l=
og<br>
+ rm -rf ./ovirt-system-tests/logs<br>
+ failed=3Dfalse<br>
+ mock_confs=3D("$WORKSPACE"/*/mocker*)<br>
+ for mock_conf_file in '"${mock_confs[@]}"'<br>
+ [[ -n <<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tes=
ts/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferre=
r" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests=
/ws/ovirt-system-tests/mocker-fedora-23-x86_64.fc23.cfg</a>> ]]<br>
+ echo 'Cleaning up mock '<br>
Cleaning up mock<br>
+ mock_root=3Dmocker-fedora-23-x86_64.fc23.cfg<br>
+ mock_root=3Dmocker-fedora-23-x86_64.fc23<br>
+ my_mock=3D/usr/bin/mock<br>
+ my_mock+=3D' --configdir=3D<<a href=3D"http://jenkins.ovirt.org/job/ov=
irt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D=
"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-sy=
stem-tests</a>'><br>
+ my_mock+=3D' --root=3Dmocker-fedora-23-x86_64.fc23'<br>
+ my_mock+=3D' --resultdir=3D<<a href=3D"http://jenkins.ovirt.org/job/ov=
irt_4.0_he-system-tests/ws/" rel=3D"noreferrer" target=3D"_blank">http://je=
nkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/</a>'><br>
+ echo 'Killing all mock orphan processes, if any.'<br>
Killing all mock orphan processes, if any.<br>
+ /usr/bin/mock --configdir=3D<<a href=3D"http://jenkins.ovirt.org/job/o=
virt_4.0_he-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=
=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt=
-system-tests</a>> --root=3Dmocker-fedora-23-x86_64.fc23 --resultdir=3D&=
lt;<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/" r=
el=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/ovirt_4.0_=
he-system-tests/ws/</a>> --orphanskill<br>
WARNING: Could not find required logging config file: <<a href=3D"http:/=
/jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logg=
ing.ini" rel=3D"noreferrer" target=3D"_blank">http://jenkins.ovirt.org/job/=
ovirt_4.0_he-system-tests/ws/ovirt-system-tests/logging.ini</a>.> Using =
default...<br>
INFO: mock.py version 1.2.17 starting (python version =3D 3.4.3)...<br>
Start: init plugins<br>
INFO: selinux enabled<br>
Finish: init plugins<br>
Start: run<br>
WARNING: Process ID 115551 still running in chroot. Killing...<br>
WARNING: Process ID 115576 still running in chroot. Killing...<br>
WARNING: Process ID 115577 still running in chroot. Killing...<br>
WARNING: Process ID 115578 still running in chroot. Killing...<br>
WARNING: Process ID 116634 still running in chroot. Killing...<br>
Finish: run<br>
++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' <<a=
 href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-sy=
stem-tests/mocker-fedora-23-x86_64.fc23.cfg" rel=3D"noreferrer" target=3D"_=
blank">http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-syst=
em-tests/mocker-fedora-23-x86_64.fc23.cfg</a>><br>
+ mock_root=3Dfedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br>
+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]<br>
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br>
++ mount<br>
++ awk '{print $3}'<br>
++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad<br>
+ [[ -n /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc ]]<br>
+ echo 'Found mounted dirs inside the chroot . Trying to umount.'<br>
Found mounted dirs inside the chroot . Trying to umount.<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/sys<br>
sh: [115551: 1 (255)] tcsetattr: Inappropriate ioctl for device<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/shm<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/dev/pts<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/cache/yum<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root<<a href=3D"http://jenkins.ovirt.org/job/ovirt_4.0_h=
e-system-tests/ws/ovirt-system-tests" rel=3D"noreferrer" target=3D"_blank">=
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test=
s</a>><br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/run/libvirt<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/var/lib/lago<br>
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/var/lib/lago: not mounted<br>
+ echo 'ERROR:  Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621=
56a2fa4a935ea8b988eb73b2ad-91644/root/var/lib/lago.'<br>
ERROR:  Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a=
935ea8b988eb73b2ad-91644/root/var/lib/lago.<br>
+ failed=3Dtrue<br>
+ for mount in '"${mounts[@]}"'<br>
+ sudo umount --lazy /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b98=
8eb73b2ad-91644/root/proc/filesystems<br>
umount: /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44/root/proc/filesystems: mountpoint not found<br>
+ echo 'ERROR:  Failed to umount /var/lib/mock/fedora-23-x86_64-0c3621=
56a2fa4a935ea8b988eb73b2ad-91644/root/proc/filesystems.'<br>
ERROR:  Failed to umount /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a=
935ea8b988eb73b2ad-91644/root/proc/filesystems.<br>
+ failed=3Dtrue<br>
+ for mock_root in '/var/lib/mock/*'<br>
+ this_chroot_failed=3Dfalse<br>
+ mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))<br>
++ mount<br>
++ awk '{print $3}'<br>
++ grep /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad-916=
44<br>
+ :<br>
+ [[ -n '' ]]<br>
+ false<br>
+ sudo rm -rf /var/lib/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2=
ad-91644<br>
+ true<br>
+ echo Aborting.<br>
Aborting.<br>
+ exit 1<br>
POST BUILD TASK : FAILURE<br>
END OF POST BUILD TASK : 1<br>
Recording test results<br>
ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: No t=
est report files were found. Configuration error?<br>
Archiving artifacts<br>
_______________________________________________<br>
Infra mailing list<br>
<a href=3D"mailto:Infra@ovirt.org" target=3D"_blank">Infra@ovirt.org</a><br=
>
<a href=3D"http://lists.ovirt.org/mailman/listinfo/infra" rel=3D"noreferrer=
" target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/infra</a><br>
<br>
<br>
</blockquote></div>
</div><div><br></div></div></body></html>
------=_Part_9639957_1092846933.1468912250886--