Re: Another job failure: "Command failed. See logs for output"

Hi David, =20 I think you've got me wrong. I was referring to what Eyal said about increasing the history we keep. Not to the problem itself.

I still have no idea what the problem is. I thought about attaching the logs of the Mock to the build so we'll have them archived on the master. I'll look at the mock runner script to see where it is writing this logs =

we'll be able to archive them.

=20 Thanks Gil =20 =20 On Tue, May 17, 2016 at 12:20 PM, dcaro <dcaro(a)redhat.com&gt; wrote: =20 > On 05/17 10:36, Gil Shinar wrote: > > Hi, > > > > I have tried to debug this issue yesterday with the help of Barak. > > I don't think that the problem is history as I saw another failure li=

> > this yesterday (it happens from time to time). Problem is the slave it > > builds on being taken by another build and cleans the workspace. I tr=

> to > > look for logs under the workspace and they missing probably because a > newer > > build is being built on the same slave. > > How did you find out about that? > That has happened before when the same slave was added twice on jenkins > with > different names, but the behavior was way different (you'd get things l=

> the > dir does not exist and such, not errors when running yum inside mock) > > > > > Gil > > > > On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > > > > > Can we increase the history for builds on that job? We should be a=

> to > > > debug jobs at least a week back. Artifacts are not needed > > > On May 16, 2016 11:42 PM, "David Caro" <dcaro(a)redhat.com&gt; wrote: > > > > > >> On 05/15 23:23, Nir Soffer wrote: > > >> > Another instance: > > >> > > > >> > 19:38:15 Start: yum install > > >> > 19:38:22 ERROR: Command failed. See logs for output. > > >> > > >> That means that there was an issue with the yum repos, most common > causes > > >> are: > > >> > > >> * The repos were actually down (mirrors fail once a day usually, w=

> > >> syncing > > >> the rpms, though the cause is just a guess) > > >> * Repoproxy (that we use to cache rpms) was overloaded > > >> > > >> Both jobs histories are now gone, in order to be able to debug tho=

> > >> issues > > >> (for the next time), try looking into the mock logs, under the > 'logs.tgz' > > >> file > > >> that's archived in the job, then under the path: > > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > > >> or > > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/ > > >> > > >> Usually in the log named 'root.log' (check which one has more size=

> > >> > > >> > > > >> > > > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/co=

> > >> > > >> > > > >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer(a)redhat.com&gt; > > >> wrote: > > >> > > > > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/co=

> > >> > > > > >> > > 19:30:56 Start: yum install > > >> > > 19:31:03 ERROR: Command failed. See logs for output. > > >> > > > > >> > > Including the to make sure it will not disapper > > >> > > ---- > > >> > > > > >> > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 > > >> > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 > nested) > > >> in > > >> > > workspace > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 > > >> > > 19:30:49 Fetching changes from the remote Git repository > > >> > > 19:30:49 > git config remote.origin.url > > >> > > git://gerrit.ovirt.org/vdsm.git # timeout=3D10 > > >> > > 19:30:49 Cleaning workspace > > >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 > > >> > > 19:30:49 Resetting working tree > > >> > > 19:30:49 > git reset --hard # timeout=3D10 > > >> > > 19:30:49 > git clean -fdx # timeout=3D10 > > >> > > 19:30:49 Pruning obsolete local branches > > >> > > 19:30:49 Fetching upstream changes from git:// > > >> gerrit.ovirt.org/vdsm.git > > >> > > 19:30:49 > git --version # timeout=3D10 > > >> > > 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress > > >> > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune > > >> > > 19:30:52 > git rev-parse > > >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D=

> > >> > > 19:30:52 Checking out Revision > > >> 0940208483f3a21261eb5d725348e65c23becdc0 (master) > > >> > > 19:30:52 > git config core.sparsecheckout # timeout=3D10 > > >> > > 19:30:52 > git checkout -f > 0940208483f3a21261eb5d725348e65c23becdc0 > > >> > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 > > >> > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4=

> # > > >> timeout=3D10 > > >> > > 19:30:52 > git branch -a # timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # > timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse > remotes/origin/ovirt-3.5-gluster^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # > > >> timeout=3D10 > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # > > >> timeout=3D10 > > >> > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 > > >> > > 19:30:53 Fetching changes from the remote Git repository > > >> > > 19:30:53 > git config remote.origin.url > > >> > > git://gerrit.ovirt.org/jenkins.git # timeout=3D10 > > >> > > 19:30:53 Cleaning workspace > > >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 > > >> > > 19:30:53 Resetting working tree > > >> > > 19:30:53 > git reset --hard # timeout=3D10 > > >> > > 19:30:53 > git clean -fdx # timeout=3D10 > > >> > > 19:30:53 Pruning obsolete local branches > > >> > > 19:30:53 Fetching upstream changes from git:// > > >> gerrit.ovirt.org/jenkins.git > > >> > > 19:30:53 > git --version # timeout=3D10 > > >> > > 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress > > >> > > git://gerrit.ovirt.org/jenkins.git > > >> +refs/heads/*:refs/remotes/origin/* > > >> > > --prune > > >> > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 > > >> > > 19:30:53 Checking out Revision > > >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) > > >> > > 19:30:53 > git config core.sparsecheckout # timeout=3D10 > > >> > > 19:30:53 > git checkout -f > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > > >> > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22dd=

> # > > >> timeout=3D10 > > >> > > 19:30:53 > git branch -a # timeout=3D10 > > >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # > timeout=3D10 > > >> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > >> > > /tmp/hudson8265618520208925180.sh > > >> > > 19:30:53 shell-scripts/cleanup_slave.sh > > >> > > 19:30:53 > > >> ################################################################### > > >> > > 19:30:53 # Cleaning up slave > > >> # > > >> > > 19:30:53 > > >> ################################################################### > > >> > > 19:30:53 Filesystem Size Used > Avail > > >> > > Use% Mounted on > > >> > > 19:30:53 devtmpfs 3.9G 0 > > >> 3.9G 0% /dev > > >> > > 19:30:53 tmpfs 3.9G 0 > 3.9G > > >> > > 0% /dev/shm > > >> > > 19:30:53 tmpfs 3.9G 2.1M > > >> 3.9G 1% /run > > >> > > 19:30:53 tmpfs 3.9G 0 > 3.9G > > >> > > 0% /sys/fs/cgroup > > >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G > > >> 27G 20% / > > >> > > 19:30:53 tmpfs 3.9G 244K > > >> 3.9G 1% /tmp > > >> > > 19:30:53 /dev/vda1 988M 94M > > >> 828M 11% /boot > > >> > > 19:30:53 tmpfs 799M 0 > 799M > > >> > > 0% /run/user/1000 > > >> > > 19:30:53 > > >> --------------------------------------------------------------- > > >> > > 19:30:53 Cleaning up postgres databases > > >> > > 19:30:53 Postgres installation not found, skipping > > >> > > 19:30:53 Cleaning up journal logs (if any) > > >> > > 19:30:53 Redirecting to /bin/systemctl restart > > >> systemd-journald.service > > >> > > 19:30:54 Cleaning up /var/tmp > > >> > > 19:30:54 done > > >> > > 19:30:54 Emptying some common logs > > >> > > 19:30:54 /var/log/wtmp > > >> > > 19:30:54 Done > > >> > > 19:30:54 Making sure there are no device mappings... > > >> > > 19:30:54 Removing the used loop devices... > > >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.servi=

> > >> > > 19:30:54 > > >> --------------------------------------------------------------- > > >> > > 19:30:54 Filesystem Size Used > Avail > > >> > > Use% Mounted on > > >> > > 19:30:54 devtmpfs 3.9G 0 > > >> 3.9G 0% /dev > > >> > > 19:30:54 tmpfs 3.9G 0 > 3.9G > > >> > > 0% /dev/shm > > >> > > 19:30:54 tmpfs 3.9G 2.1M > > >> 3.9G 1% /run > > >> > > 19:30:54 tmpfs 3.9G 0 > 3.9G > > >> > > 0% /sys/fs/cgroup > > >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G > > >> 27G 20% / > > >> > > 19:30:54 tmpfs 3.9G 244K > > >> 3.9G 1% /tmp > > >> > > 19:30:54 /dev/vda1 988M 94M > > >> 828M 11% /boot > > >> > > 19:30:54 tmpfs 799M 0 > 799M > > >> > > 0% /run/user/1000 > > >> > > 19:30:54 > > >> ################################################################### > > >> > > 19:30:54 # Slave cleanup done > > >> # > > >> > > 19:30:54 > > >> ################################################################### > > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > >> > > /tmp/hudson7354644880288923842.sh > > >> > > 19:30:54 + echo shell-scripts/global_setup.sh > > >> > > 19:30:54 shell-scripts/global_setup.sh > > >> > > 19:30:54 + rm -rf > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > >> > > 19:30:54 + mkdir -p > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > >> > > /tmp/hudson4023353013258685526.sh > > >> > > 19:30:54 + echo shell-scripts/mock_setup.sh > > >> > > 19:30:54 shell-scripts/mock_setup.sh > > >> > > 19:30:54 + shopt -s nullglob > > >> > > 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] > > >> > > 19:30:54 + failed=3Dfalse > > >> > > 19:30:54 + > > >> > mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/m=

> > >> > > 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) > > >> > > 19:30:54 + false > > >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts > > >> > > 19:30:54 + mkdir -p mock exported-artifacts > > >> > > 19:30:54 + chgrp mock mock > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-art=

> > >> > > 19:30:54 + chmod g+rws mock > > >> > > 19:30:54 + sudo touch > > >> > > 19:30:54 + : > > >> > > 19:30:54 + sudo yum clean all > > >> > > 19:30:54 Yum command has been deprecated, redirecting to > '/usr/bin/dnf > > >> > > clean all'. > > >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. > > >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run: > > >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf=

> > >> migrate' > > >> > > 19:30:54 > > >> > > 19:30:55 Cleaning repos: updates ci-tools fedora > > >> > > 19:30:55 Cleaning up Everything > > >> > > 19:30:55 + exit 0 > > >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > >> > > /tmp/hudson4405890361943185034.sh > > >> > > 19:30:55 shell-scripts/standard_check_patch.sh > > >> > > 19:30:55 > ########################################################## > > >> > > 19:30:55 > ########################################################## > > >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: > > >> el7:epel-7-x86_64 > > >> > > 19:30:55 > ########################################################## > > >> > > 19:30:55 > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for > script: > > >> > > automation/check-patch.sh > > >> > > 19:30:55 > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > >> > > 19:30:55 Using base mock conf > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock=

> > >> > > 19:30:55 Using proxified config > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock=

> > >> > > 19:30:55 Generating temporary mock conf > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-=

> > >> > > 19:30:55 Skipping mount points > > >> > > 19:30:55 Using temp chroot =3D > > >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:30:55 Adding repo ovirt-snapshot -> > > >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 > > >> > > 19:30:55 Adding repo ovirt-snapshot-static -> > > >> > > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 > > >> > > 19:30:55 Adding repo centos-glusterfs -> > > >> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ > > >> > > 19:30:55 Adding repo centos-ovirt40-candidate -> > > >> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ > > >> > > 19:30:55 Adding repo lago -> > > >> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 > > >> > > 19:30:55 Adding repo epel -> > > >> http://download.fedoraproject.org/pub/epel/7/x86_64 > > >> > > 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot > > >> > > 19:30:55 mock \ > > >> > > 19:30:55 > > >> > > > > >> > --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

> > >> > > \ > > >> > > 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ > > >> > > 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.=

> \ > > >> > > 19:30:55 --init > > >> > > 19:30:55 WARNING: Could not find required logging config file: > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging=

> > >> > > Using default... > > >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version=

> > >> 3.4.3)... > > >> > > 19:30:55 Start: init plugins > > >> > > 19:30:55 INFO: selinux enabled > > >> > > 19:30:55 Finish: init plugins > > >> > > 19:30:55 Start: run > > >> > > 19:30:55 Start: clean chroot > > >> > > 19:30:55 Finish: clean chroot > > >> > > 19:30:55 Start: chroot init > > >> > > 19:30:55 INFO: calling preinit hooks > > >> > > 19:30:55 INFO: enabled root cache > > >> > > 19:30:55 INFO: enabled yum cache > > >> > > 19:30:55 Start: cleaning yum metadata > > >> > > 19:30:55 Finish: cleaning yum metadata > > >> > > 19:30:55 INFO: enabled ccache > > >> > > 19:30:55 Mock Version: 1.2.14 > > >> > > 19:30:55 INFO: Mock Version: 1.2.14 > > >> > > 19:30:56 Start: yum install > > >> > > 19:31:03 ERROR: Command failed. See logs for output. > > >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot > > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/r=

> > >> > > --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dno=

> > >> > > 19:31:03 Init took 8 seconds > > >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

> > >> > > 19:31:03 > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patc=

> > >> > > chroot finished > > >> > > 19:31:03 @@ took 8 seconds > > >> > > 19:31:03 @@ rc =3D 1 > > >> > > 19:31:03 > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > >> > > 19:31:03 > ########################################################## > > >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: > > >> el7:epel-7-x86_64 > > >> > > 19:31:03 ## took 8 seconds > > >> > > 19:31:03 ## rc =3D 1 > > >> > > 19:31:03 > ########################################################## > > >> > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > > >> > > 19:31:03 ##! Last 20 log enties: > > >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > >> > > 19:31:03 ##! > > >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot > > >> > > 19:31:03 mock \ > > >> > > 19:31:03 > > >> > > > > >> > --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

> > >> > > \ > > >> > > 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ > > >> > > 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.=

> > >> > > 19:31:03 Init took 8 seconds > > >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

> > >> > > 19:31:03 ##! > > >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > >> > > 19:31:03 > ##!######################################################## > > >> > > 19:31:03 Build step 'Execute shell' marked build as failure > > >> > > 19:31:03 Performing Post build task... > > >> > > 19:31:03 Match found for :.* : True > > >> > > 19:31:03 Logical operation result is TRUE > > >> > > 19:31:03 Running script : #!/bin/bash -xe > > >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh" > > >> > > 19:31:03 > > >> > > 19:31:03 shopt -s nullglob > > >> > > 19:31:03 > > >> > > 19:31:03 > > >> > > 19:31:03 WORKSPACE=3D"$PWD" > > >> > > 19:31:03 > > >> > > 19:31:03 # Make clear this is the cleanup, helps reading the > jenkins > > >> logs > > >> > > 19:31:03 cat <<EOC > > >> > > 19:31:03 > > >> > _______________________________________________________________________ > > >> > > 19:31:03 > > >> > ####################################################################### > > >> > > 19:31:03 # > > >> # > > >> > > 19:31:03 # CLEANUP > > >> # > > >> > > 19:31:03 # > > >> # > > >> > > 19:31:03 > > >> > ####################################################################### > > >> > > 19:31:03 EOC > > >> > > 19:31:03 > > >> > > 19:31:03 > > >> > > 19:31:03 # Archive the logs, we want them anyway > > >> > > 19:31:03 logs=3D( > > >> > > 19:31:03 ./*log > > >> > > 19:31:03 ./*/logs > > >> > > 19:31:03 ) > > >> > > 19:31:03 if [[ "$logs" ]]; then > > >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > > >> > > 19:31:03 rm -rf "${logs[@]}" > > >> > > 19:31:03 fi > > >> > > 19:31:03 > > >> > > 19:31:03 # stop any processes running inside the chroot > > >> > > 19:31:03 failed=3Dfalse > > >> > > 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) > > >> > > 19:31:03 # Clean current jobs mockroot if any > > >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do > > >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue > > >> > > 19:31:03 echo "Cleaning up mock $mock_conf" > > >> > > 19:31:03 mock_root=3D"${mock_conf_file##*/}" > > >> > > 19:31:03 mock_root=3D"${mock_root%.*}" > > >> > > 19:31:03 my_mock=3D"/usr/bin/mock" > > >> > > 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" > > >> > > 19:31:03 my_mock+=3D" --root=3D${mock_root}" > > >> > > 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" > > >> > > 19:31:03 > > >> > > 19:31:03 #TODO: investigate why mock --clean fails to umou=

> > >> > > certain dirs sometimes, > > >> > > 19:31:03 #so we can use it instead of manually doing all t=

> > >> > > 19:31:03 echo "Killing all mock orphan processes, if any." > > >> > > 19:31:03 $my_mock \ > > >> > > 19:31:03 --orphanskill \ > > >> > > 19:31:03 || { > > >> > > 19:31:03 echo "ERROR: Failed to kill orphans on $chro=

> > >> > > 19:31:03 failed=3Dtrue > > >> > > 19:31:03 } > > >> > > 19:31:03 > > >> > > 19:31:03 mock_root=3D"$(\ > > >> > > 19:31:03 grep \ > > >> > > 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^=

> > >> > > 19:31:03 "$mock_conf_file" \ > > >> > > 19:31:03 )" || : > > >> > > 19:31:03 [[ "$mock_root" ]] || continue > > >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep > "$mock_root")) > > >> || : > > >> > > 19:31:03 if [[ "$mounts" ]]; then > > >> > > 19:31:03 echo "Found mounted dirs inside the chroot > $chroot. > > >> > > Trying to umount." > > >> > > 19:31:03 fi > > >> > > 19:31:03 for mount in "${mounts[@]}"; do > > >> > > 19:31:03 sudo umount "$mount" \ > > >> > > 19:31:03 || { > > >> > > 19:31:03 echo "ERROR: Failed to umount $mount." > > >> > > 19:31:03 failed=3Dtrue > > >> > > 19:31:03 } > > >> > > 19:31:03 done > > >> > > 19:31:03 done > > >> > > 19:31:03 > > >> > > 19:31:03 # Clean any leftover chroot from other jobs > > >> > > 19:31:03 for mock_root in /var/lib/mock/*; do > > >> > > 19:31:03 this_chroot_failed=3Dfalse > > >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep > "$mock_root")) > > >> || : > > >> > > 19:31:03 if [[ "$mounts" ]]; then > > >> > > 19:31:03 echo "Found mounted dirs inside the chroot > > >> $mock_root." \ > > >> > > 19:31:03 "Trying to umount." > > >> > > 19:31:03 fi > > >> > > 19:31:03 for mount in "${mounts[@]}"; do > > >> > > 19:31:03 sudo umount "$mount" \ > > >> > > 19:31:03 || { > > >> > > 19:31:03 echo "ERROR: Failed to umount $mount." > > >> > > 19:31:03 failed=3Dtrue > > >> > > 19:31:03 this_chroot_failed=3Dtrue > > >> > > 19:31:03 } > > >> > > 19:31:03 done > > >> > > 19:31:03 if ! $this_chroot_failed; then > > >> > > 19:31:03 sudo rm -rf "$mock_root" > > >> > > 19:31:03 fi > > >> > > 19:31:03 done > > >> > > 19:31:03 > > >> > > 19:31:03 if $failed; then > > >> > > 19:31:03 echo "Aborting." > > >> > > 19:31:03 exit 1 > > >> > > 19:31:03 fi > > >> > > 19:31:03 > > >> > > 19:31:03 # remove mock system cache, we will setup proxies to =

> the > > >> > > caching and this > > >> > > 19:31:03 # takes lots of space between runs > > >> > > 19:31:03 shopt -u nullglob > > >> > > 19:31:03 sudo rm -Rf /var/cache/mock/* > > >> > > 19:31:03 > > >> > > 19:31:03 # restore the permissions in the working dir, as > sometimes it > > >> > > leaves files > > >> > > 19:31:03 # owned by root and then the 'cleanup workspace' from > jenkins > > >> > > job fails to > > >> > > 19:31:03 # clean and breaks the jobs > > >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" > > >> > > 19:31:03 > > >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > >> > > /tmp/hudson3681642653853563663.sh > > >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh > > >> > > 19:31:03 shell-scripts/mock_cleanup.sh > > >> > > 19:31:03 + shopt -s nullglob > > >> > > 19:31:03 + > > >> WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8=

> > >> > > 19:31:03 + cat > > >> > > 19:31:03 > > >> > _______________________________________________________________________ > > >> > > 19:31:03 > > >> > ####################################################################### > > >> > > 19:31:03 # > > >> # > > >> > > 19:31:03 # CLEANUP > > >> # > > >> > > 19:31:03 # > > >> # > > >> > > 19:31:03 > > >> > ####################################################################### > > >> > > 19:31:03 + logs=3D(./*log ./*/logs) > > >> > > 19:31:03 + [[ -n ./vdsm/logs ]] > > >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs > > >> > > 19:31:03 ./vdsm/logs/ > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log > > >> > > 19:31:03 > ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log > > >> > > 19:31:03 + rm -rf ./vdsm/logs > > >> > > 19:31:03 + failed=3Dfalse > > >> > > 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) > > >> > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' > > >> > > 19:31:03 + [[ -n > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-=

> > >> > > ]] > > >> > > 19:31:03 + echo 'Cleaning up mock ' > > >> > > 19:31:03 Cleaning up mock > > >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg > > >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 > > >> > > 19:31:03 + my_mock=3D/usr/bin/mock > > >> > > 19:31:03 + my_mock+=3D' > > >> > > > > >> > --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6=

> > >> > > 19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' > > >> > > 19:31:03 + my_mock+=3D' > > >> > > > > >> > --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6=

> > >> > > 19:31:03 + echo 'Killing all mock orphan processes, if any.' > > >> > > 19:31:03 Killing all mock orphan processes, if any. > > >> > > 19:31:03 + /usr/bin/mock > > >> > > > > >> > --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6=

> > >> > > --root=3Dmocker-epel-7-x86_64.el7 > > >> > > > --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > --orphanskill > > >> > > 19:31:04 WARNING: Could not find required logging config file: > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging=

> > >> > > Using default... > > >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version=

> > >> 3.4.3)... > > >> > > 19:31:04 Start: init plugins > > >> > > 19:31:04 INFO: selinux enabled > > >> > > 19:31:04 Finish: init plugins > > >> > > 19:31:04 Start: run > > >> > > 19:31:04 Finish: run > > >> > > 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D > '\'')[^'\'']*' > > >> > > > > >> > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-=

> > >> > > 19:31:04 + > mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8f=

> > >> > > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep > "$mock_root")) > > >> > > 19:31:04 ++ mount > > >> > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:31:04 ++ awk '{print $3}' > > >> > > 19:31:04 + : > > >> > > 19:31:04 + [[ -n '' ]] > > >> > > 19:31:04 + for mock_root in '/var/lib/mock/*' > > >> > > 19:31:04 + this_chroot_failed=3Dfalse > > >> > > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep > "$mock_root")) > > >> > > 19:31:04 ++ mount > > >> > > 19:31:04 ++ grep > > >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:31:04 ++ awk '{print $3}' > > >> > > 19:31:04 + : > > >> > > 19:31:04 + [[ -n '' ]] > > >> > > 19:31:04 + false > > >> > > 19:31:04 + sudo rm -rf > > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:31:04 + false > > >> > > 19:31:04 + shopt -u nullglob > > >> > > 19:31:04 + sudo rm -Rf > > >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > >> > > 19:31:04 + sudo chown -R jenkins > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > 19:31:04 POST BUILD TASK : SUCCESS > > >> > > 19:31:04 END OF POST BUILD TASK : 0 > > >> > > 19:31:04 Archiving artifacts > > >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure > > >> > > 19:31:04 Started calculate disk usage of build > > >> > > 19:31:04 Finished Calculation of disk usage of build in 0 seco=

> > >> > > 19:31:04 Started calculate disk usage of workspace > > >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 > seconds > > >> > > 19:31:04 Finished: FAILURE > > >> > _______________________________________________ > > >> > Infra mailing list > > >> > Infra(a)ovirt.org > > >> > http://lists.ovirt.org/mailman/listinfo/infra > > >> > > >> -- > > >> David Caro > > >> > > >> Red Hat S.L. > > >> Continuous Integration Engineer - EMEA ENG Virtualization R&D > > >> > > >> Tel.: +420 532 294 605 > > >> Email: dcaro(a)redhat.com > > >> IRC: dcaro|dcaroest@{freenode|oftc|redhat} > > >> Web: www.redhat.com > > >> RHT Global #: 82-62605 > > >> > > >> _______________________________________________ > > >> Infra mailing list > > >> Infra(a)ovirt.org > > >> http://lists.ovirt.org/mailman/listinfo/infra > > >> > > >> > > > _______________________________________________ > > > Infra mailing list > > > Infra(a)ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dcaro(a)redhat.com > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > Web: www.redhat.com > RHT Global #: 82-62605 >

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuVuAAoJEEBxx+HSYmnD8KsIAJt1699HIBsr31Ud8Wm8+XN3 45uDC4Pd1W2VJUzDdW6JpaTNU2ywE1YgAOSG6OgC+kBxa5+zyhWks7MQYzbPJx4F VsyH8Ph/zTUjbuLmYt6WQQRuSaJZTSfXjgRAK3wiMRVklVHGH/C3fRgyuysuFssV 3RpaE5RFrAuBwkMdzauRUUBbeo3mjSfrHCHxCTdNYRll/AQcwyLWzYTBfZ2ZXiYi Sc+gQ5pMr10Nagkl+me3itcTdKjcOPyiO9xltq6fQ3DPkQc57G4odiCikEBBpyeK HwgzIFgnZQbDdHNuSfe8pQ0Yys/O56m2tz6nXx6NdelcEgz1XWFp/OAYw64+AA8= =guUe -----END PGP SIGNATURE-----