--t5NgoZwlhlUmGr82
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 05/17 11:20, dcaro wrote:
On 05/17 10:36, Gil Shinar wrote:
> Hi,
>=20
> I have tried to debug this issue yesterday with the help of Barak.
> I don't think that the problem is history as I saw another failure like
> this yesterday (it happens from time to time). Problem is the slave it
> builds on being taken by another build and cleans the workspace. I trie=
d to
> look for logs under the workspace and they missing probably
because a n=
ewer
> build is being built on the same slave.
=20
How did you find out about that?
That has happened before when the same slave was added twice on jenkins w=
ith
different names, but the behavior was way different (you'd get
things lik=
e the
dir does not exist and such, not errors when running yum inside mock)
An example of yum failure:
http://jenkins.ovirt.org/view/Master%20branch%20per%20project/view/vdsm/job=
/vdsm_master_check-patch-el7-x86_64/1523/
You can see there the log in the console:
03:00:58 Start: yum install
03:01:07 ERROR: Command failed. See logs for output.
03:01:07 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_=
64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-=
build --setopt=3Dtsflags=3Dnocontexts
Then, in the archived artifacts logs.tgz, under
vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log, you can see the error log:
DEBUG util.py:474: Executing command: ['/usr/bin/yum-deprecated', '--insta=
llroot', '/var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root=
/', '--releasever', '7', 'install', '@buildsys-build',
'--setopt=3Dtsflags=
=3Dnocontexts'] with env {'SHELL': '/bin/bash', 'CCACHE_DIR':
'/tmp/ccache'=
, 'HOME': '/builddir', 'PATH':
'/usr/bin:/bin:/usr/sbin:/sbin', 'CCACHE_UMA=
SK': '002', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND':
'printf "\x1b]0;<mock-chr=
oot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8',
'TERM': 'vt100', 'HOSTNAME'=
: 'mock', 'LD_PRELOAD': '/tmp/tmpxqactgv2/$LIB/nosync.so'} and
shell False
DEBUG util.py:399: Yum command has been deprecated, use dnf instead.
DEBUG util.py:399: See 'man dnf' and 'man yum2dnf' for more information.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/bbf11fe5f13b4c6d0a9a242b422885c6ed41e05ae718d4ea3abfc3550ef4f243-comp=
s-epel7.xml.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/4e6f7cf18ae8bc3553da486c7847ac8f9a50671406d26c72a4d0765f914c5c76-upda=
teinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim=
ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim=
ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim=
ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: One of the configured repositories failed ("Custom epe=
l"),
DEBUG util.py:399: and yum doesn't have enough cached data to continue. A=
t this point the only
DEBUG util.py:399: safe thing yum can do is fail. There are a few ways to=
work "fix" this:
DEBUG util.py:399: 1. Contact the upstream for the repository and get=
them to fix the problem.
DEBUG util.py:399: 2. Reconfigure the baseurl/etc. for the repository=
, to point to a working
DEBUG util.py:399: upstream. This is most often useful if you are =
using a newer
DEBUG util.py:399: distribution release than is supported by the r=
epository (and the
DEBUG util.py:399: packages for the previous distribution release =
still work).
DEBUG util.py:399: 3. Disable the repository, so yum won't use it by =
default. Yum will then
DEBUG util.py:399: just ignore the repository until you permanentl=
y enable it again or use
DEBUG util.py:399: --enablerepo for temporary usage:
DEBUG util.py:399: yum-config-manager --disable epel
DEBUG util.py:399: 4. Configure the failing repository to be skipped,=
if it is unavailable.
DEBUG util.py:399: Note that yum will try to contact the repo. whe=
n it runs most commands,
DEBUG util.py:399: so will have to try and fail each time (and thu=
s. yum will be be much
DEBUG util.py:399: slower). If it is a very temporary problem thou=
gh, this is often a nice
DEBUG util.py:399: compromise:
DEBUG util.py:399: yum-config-manager --save --setopt=3Depel.s=
kip_if_unavailable=3Dtrue
DEBUG util.py:399: failure: repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f=
4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz from epel: [Errno 256] No mor=
e mirrors to try.
DEBUG util.py:399:
http://download.fedoraproject.org/pub/epel/7/x86_64/rep=
odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim=
ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:521: Child return code was: 1
That usually means that the mirrors were in the middle of an update, and the
md5sums in the metadata were not consistent at that point it time (happens =
=66rom
time to time).
We could try to minimize that issue handling a mirror ourselves, though that
would mean another service to maintain (with it's own potential issues, like
not being updated and such, but that's another subject)
=20
>=20
> Gil
>=20
> On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri(a)redhat.com> wrote:
>=20
> > Can we increase the history for builds on that job? We should be abl=
e
to
> > debug jobs at least a week back. Artifacts are not needed
> > On May 16, 2016 11:42 PM, "David Caro" <dcaro(a)redhat.com>
wrote:
> >
> >> On 05/15 23:23, Nir Soffer wrote:
> >> > Another instance:
> >> >
> >> > 19:38:15 Start: yum install
> >> > 19:38:22 ERROR: Command failed. See logs for output.
> >>
> >> That means that there was an issue with the yum repos, most common c=
auses
> >> are:
> >>
> >> * The repos were actually down (mirrors fail once a day usually, whi=
le
> >> syncing
> >> the rpms, though the cause is just a guess)
> >> * Repoproxy (that we use to cache rpms) was overloaded
> >>
> >> Both jobs histories are now gone, in order to be able to debug those
> >> issues
> >> (for the next time), try looking into the mock logs, under the 'logs=
=2Etgz'
> >> file
> >> that's archived in the job, then under the path:
> >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> >> or
> >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
> >>
> >> Usually in the log named 'root.log' (check which one has more
size).
> >>
> >> >
> >> >
> >>
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213=
/console
> >>
> >> >
> >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer
<nsoffer(a)redhat.com>
> >> wrote:
> >> > >
> >>
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209=
/console
> >> > >
> >> > > 19:30:56 Start: yum install
> >> > > 19:31:03 ERROR: Command failed. See logs for output.
> >> > >
> >> > > Including the to make sure it will not disapper
> >> > > ----
> >> > >
> >> > > 19:30:48 Triggered by Gerrit:
https://gerrit.ovirt.org/56550
> >> > > 19:30:48 Building remotely on
fc23-vm07.phx.ovirt.org (fc23 nest=
ed)
> >> in
> >> > > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x8=
6_64
> >> > > 19:30:48 > git rev-parse
--is-inside-work-tree # timeout=3D10
> >> > > 19:30:49 Fetching changes from the remote Git repository
> >> > > 19:30:49 > git config remote.origin.url
> >> > >
git://gerrit.ovirt.org/vdsm.git # timeout=3D10
> >> > > 19:30:49 Cleaning workspace
> >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=3D10
> >> > > 19:30:49 Resetting working tree
> >> > > 19:30:49 > git reset --hard # timeout=3D10
> >> > > 19:30:49 > git clean -fdx # timeout=3D10
> >> > > 19:30:49 Pruning obsolete local branches
> >> > > 19:30:49 Fetching upstream changes from git://
> >>
gerrit.ovirt.org/vdsm.git
> >> > > 19:30:49 > git --version # timeout=3D10
> >> > > 19:30:49 > git -c core.askpass=3Dtrue fetch --tags
--progress
> >> > >
git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune
> >> > > 19:30:52 > git rev-parse
> >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10
> >> > > 19:30:52 Checking out Revision
> >> 0940208483f3a21261eb5d725348e65c23becdc0 (master)
> >> > > 19:30:52 > git config core.sparsecheckout # timeout=3D10
> >> > > 19:30:52 > git checkout -f
0940208483f3a21261eb5d725348e65c23be=
cdc0
> >> > > 19:30:52 > git rev-parse
FETCH_HEAD^{commit} # timeout=3D10
> >> > > 19:30:52 > git rev-list
a09c577837b939096f97108fdbbcafe5980d4a0=
d #
> >> timeout=3D10
> >> > > 19:30:52 > git branch -a # timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} #
timeo=
ut=3D10
> >> > > 19:30:52 > git rev-parse
remotes/origin/ovirt-3.1^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse
remotes/origin/ovirt-3.5-gluster^{comm=
it} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} #
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit}
#
> >> timeout=3D10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit}
#
> >> timeout=3D10
> >> > > 19:30:53 > git rev-parse --is-inside-work-tree #
timeout=3D10
> >> > > 19:30:53 Fetching changes from the remote Git repository
> >> > > 19:30:53 > git config remote.origin.url
> >> > >
git://gerrit.ovirt.org/jenkins.git # timeout=3D10
> >> > > 19:30:53 Cleaning workspace
> >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=3D10
> >> > > 19:30:53 Resetting working tree
> >> > > 19:30:53 > git reset --hard # timeout=3D10
> >> > > 19:30:53 > git clean -fdx # timeout=3D10
> >> > > 19:30:53 Pruning obsolete local branches
> >> > > 19:30:53 Fetching upstream changes from git://
> >>
gerrit.ovirt.org/jenkins.git
> >> > > 19:30:53 > git --version # timeout=3D10
> >> > > 19:30:53 > git -c core.askpass=3Dtrue fetch --tags
--progress
> >> > >
git://gerrit.ovirt.org/jenkins.git
> >> +refs/heads/*:refs/remotes/origin/*
> >> > > --prune
> >> > > 19:30:53 > git rev-parse origin/master^{commit} #
timeout=3D10
> >> > > 19:30:53 Checking out Revision
> >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master)
> >> > > 19:30:53 > git config core.sparsecheckout # timeout=3D10
> >> > > 19:30:53 > git checkout -f
68a517c3ccc1fe58a6c52646e3a99f3ab22d=
dab9
> >> > > 19:30:53 > git rev-list
68a517c3ccc1fe58a6c52646e3a99f3ab22ddab=
9 #
> >> timeout=3D10
> >> > > 19:30:53 > git branch -a # timeout=3D10
> >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} #
timeo=
ut=3D10
> >> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64]
$ /bin/bash -e
> >> > > /tmp/hudson8265618520208925180.sh
> >> > > 19:30:53 shell-scripts/cleanup_slave.sh
> >> > > 19:30:53
> >> ###################################################################
> >> > > 19:30:53 # Cleaning up slave
> >> #
> >> > > 19:30:53
> >> ###################################################################
> >> > > 19:30:53 Filesystem Size Used =
Avail
> >> > > Use% Mounted on
> >> > > 19:30:53 devtmpfs 3.9G 0
> >> 3.9G 0% /dev
> >> > > 19:30:53 tmpfs 3.9G 0 =
3.9G
> >> > > 0% /dev/shm
> >> > > 19:30:53 tmpfs 3.9G 2.1M
> >> 3.9G 1% /run
> >> > > 19:30:53 tmpfs 3.9G 0 =
3.9G
> >> > > 0% /sys/fs/cgroup
> >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> >> 27G 20% /
> >> > > 19:30:53 tmpfs 3.9G 244K
> >> 3.9G 1% /tmp
> >> > > 19:30:53 /dev/vda1 988M 94M
> >> 828M 11% /boot
> >> > > 19:30:53 tmpfs 799M 0 =
799M
> >> > > 0% /run/user/1000
> >> > > 19:30:53
> >> ---------------------------------------------------------------
> >> > > 19:30:53 Cleaning up postgres databases
> >> > > 19:30:53 Postgres installation not found, skipping
> >> > > 19:30:53 Cleaning up journal logs (if any)
> >> > > 19:30:53 Redirecting to /bin/systemctl restart
> >> systemd-journald.service
> >> > > 19:30:54 Cleaning up /var/tmp
> >> > > 19:30:54 done
> >> > > 19:30:54 Emptying some common logs
> >> > > 19:30:54 /var/log/wtmp
> >> > > 19:30:54 Done
> >> > > 19:30:54 Making sure there are no device mappings...
> >> > > 19:30:54 Removing the used loop devices...
> >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service
> >> > > 19:30:54
> >> ---------------------------------------------------------------
> >> > > 19:30:54 Filesystem Size Used =
Avail
> >> > > Use% Mounted on
> >> > > 19:30:54 devtmpfs 3.9G 0
> >> 3.9G 0% /dev
> >> > > 19:30:54 tmpfs 3.9G 0 =
3.9G
> >> > > 0% /dev/shm
> >> > > 19:30:54 tmpfs 3.9G 2.1M
> >> 3.9G 1% /run
> >> > > 19:30:54 tmpfs 3.9G 0 =
3.9G
> >> > > 0% /sys/fs/cgroup
> >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> >> 27G 20% /
> >> > > 19:30:54 tmpfs 3.9G 244K
> >> 3.9G 1% /tmp
> >> > > 19:30:54 /dev/vda1 988M 94M
> >> 828M 11% /boot
> >> > > 19:30:54 tmpfs 799M 0 =
799M
> >> > > 0% /run/user/1000
> >> > > 19:30:54
> >> ###################################################################
> >> > > 19:30:54 # Slave cleanup done
> >> #
> >> > > 19:30:54
> >> ###################################################################
> >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson7354644880288923842.sh
> >> > > 19:30:54 + echo shell-scripts/global_setup.sh
> >> > > 19:30:54 shell-scripts/global_setup.sh
> >> > > 19:30:54 + rm -rf
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> >> > > 19:30:54 + mkdir -p
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson4023353013258685526.sh
> >> > > 19:30:54 + echo shell-scripts/mock_setup.sh
> >> > > 19:30:54 shell-scripts/mock_setup.sh
> >> > > 19:30:54 + shopt -s nullglob
> >> > > 19:30:54 + [[ '' =3D=3D \t\r\u\e ]]
> >> > > 19:30:54 + failed=3Dfalse
> >> > > 19:30:54 +
> >> mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6=
4/mock
> >> > > 19:30:54 +
chroots=3D("$WORKSPACE"/mock/*)
> >> > > 19:30:54 + false
> >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts
> >> > > 19:30:54 + mkdir -p mock exported-artifacts
> >> > > 19:30:54 + chgrp mock mock
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-=
artifacts
> >> > > 19:30:54 + chmod g+rws mock
> >> > > 19:30:54 + sudo touch
> >> > > 19:30:54 + :
> >> > > 19:30:54 + sudo yum clean all
> >> > > 19:30:54 Yum command has been deprecated, redirecting to
'/usr/b=
in/dnf
> >> > > clean all'.
> >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more
information.
> >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run:
> >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate
&& dnf-2
> >> migrate'
> >> > > 19:30:54
> >> > > 19:30:55 Cleaning repos: updates ci-tools fedora
> >> > > 19:30:55 Cleaning up Everything
> >> > > 19:30:55 + exit 0
> >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e
> >> > > /tmp/hudson4405890361943185034.sh
> >> > > 19:30:55 shell-scripts/standard_check_patch.sh
> >> > > 19:30:55 #######################################################=
###
> >> > > 19:30:55
#######################################################=
###
> > >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env:
> > >> el7:epel-7-x86_64
> >> > > 19:30:55
#######################################################=
###
> >> > > 19:30:55
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for scri=
pt:
> >> > > automation/check-patch.sh
> >> > > 19:30:55
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:30:55 Using base mock conf
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m=
ock_configs/epel-7-x86_64.cfg
> >> > > 19:30:55 Using proxified config
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m=
ock_configs/epel-7-x86_64_proxied.cfg
> >> > > 19:30:55 Generating temporary mock conf
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock=
er-epel-7-x86_64.el7
> >> > > 19:30:55 Skipping mount points
> >> > > 19:30:55 Using temp chroot =3D
> >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:30:55 Adding repo ovirt-snapshot ->
> >> > >
http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7
> >> > > 19:30:55 Adding repo ovirt-snapshot-static ->
> >> > >
http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/=
el7
> >> > > 19:30:55 Adding repo centos-glusterfs ->
> >> > >
http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/
> >> > > 19:30:55 Adding repo centos-ovirt40-candidate ->
> >> > >
http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/
> >> > > 19:30:55 Adding repo lago ->
> >> > >
http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7
> >> > > 19:30:55 Adding repo epel ->
> >>
http://download.fedoraproject.org/pub/epel/7/x86_64
> >> > > 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot
> >> > > 19:30:55 mock \
> >> > > 19:30:55
> >> > >
> >> --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x=
86_64/vdsm"
> >> > > \
> >> > > 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \
> >> > > 19:30:55
--resultdir=3D"logs/mocker-epel-7-x86_64.el7.in=
it" \
> >> > > 19:30:55 --init
> >> > > 19:30:55 WARNING: Could not find required logging config file:
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg=
ing.ini.
> >> > > Using default...
> >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version =
=3D
> >> 3.4.3)...
> >> > > 19:30:55 Start: init plugins
> >> > > 19:30:55 INFO: selinux enabled
> >> > > 19:30:55 Finish: init plugins
> >> > > 19:30:55 Start: run
> >> > > 19:30:55 Start: clean chroot
> >> > > 19:30:55 Finish: clean chroot
> >> > > 19:30:55 Start: chroot init
> >> > > 19:30:55 INFO: calling preinit hooks
> >> > > 19:30:55 INFO: enabled root cache
> >> > > 19:30:55 INFO: enabled yum cache
> >> > > 19:30:55 Start: cleaning yum metadata
> >> > > 19:30:55 Finish: cleaning yum metadata
> >> > > 19:30:55 INFO: enabled ccache
> >> > > 19:30:55 Mock Version: 1.2.14
> >> > > 19:30:55 INFO: Mock Version: 1.2.14
> >> > > 19:30:56 Start: yum install
> >> > > 19:31:03 ERROR: Command failed. See logs for output.
> >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot
> >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/roo=
t/
> >> > > --releasever 7 install @buildsys-build
--setopt=3Dtsflags=3Dnoco=
ntexts
> >> > > 19:31:03 Init took 8 seconds
> >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> > > 19:31:03
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.=
sh
> >> > > chroot finished
> >> > > 19:31:03 @@ took 8 seconds
> >> > > 19:31:03 @@ rc =3D 1
> >> > > 19:31:03
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:31:03 #######################################################=
###
> >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016
Finished env:
> >> el7:epel-7-x86_64
> >> > > 19:31:03 ## took 8 seconds
> >> > > 19:31:03 ## rc =3D 1
> >> > > 19:31:03 #######################################################=
###
> >> > > 19:31:03 ##! ERROR
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> >> > > 19:31:03 ##! Last 20 log enties:
> >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log
> >> > > 19:31:03 ##!
> >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot
> >> > > 19:31:03 mock \
> >> > > 19:31:03
> >> > >
> >> --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x=
86_64/vdsm"
> >> > > \
> >> > > 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \
> >> > > 19:31:03
--resultdir=3D"logs/mocker-epel-7-x86_64.el7.in=
it" \
> > >> > > 19:31:03 --init
> >> > > 19:31:03 Init took 8 seconds
> >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> > > 19:31:03 ##!
> >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >> > > 19:31:03 ##!####################################################=
####
> >> > > 19:31:03 Build step 'Execute shell'
marked build as failure
> >> > > 19:31:03 Performing Post build task...
> >> > > 19:31:03 Match found for :.* : True
> >> > > 19:31:03 Logical operation result is TRUE
> >> > > 19:31:03 Running script : #!/bin/bash -xe
> >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh"
> >> > > 19:31:03
> >> > > 19:31:03 shopt -s nullglob
> >> > > 19:31:03
> >> > > 19:31:03
> >> > > 19:31:03 WORKSPACE=3D"$PWD"
> >> > > 19:31:03
> >> > > 19:31:03 # Make clear this is the cleanup, helps reading the jen=
kins
> >> logs
> >> > > 19:31:03 cat <<EOC
> >> > > 19:31:03
> >> ____________________________________________________________________=
___
> >> > > 19:31:03
> >> ####################################################################=
###
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03 # CLEANUP
> > >> #
> > >> > > 19:31:03 #
> > >> #
> >> > > 19:31:03
> >> ####################################################################=
###
> >> > > 19:31:03 EOC
> >> > > 19:31:03
> >> > > 19:31:03
> >> > > 19:31:03 # Archive the logs, we want them anyway
> >> > > 19:31:03 logs=3D(
> >> > > 19:31:03 ./*log
> >> > > 19:31:03 ./*/logs
> >> > > 19:31:03 )
> >> > > 19:31:03 if [[ "$logs" ]]; then
> >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz
"${logs[@]}"
> >> > > 19:31:03 rm -rf "${logs[@]}"
> >> > > 19:31:03 fi
> >> > > 19:31:03
> >> > > 19:31:03 # stop any processes running inside the chroot
> >> > > 19:31:03 failed=3Dfalse
> >> > > 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*)
> >> > > 19:31:03 # Clean current jobs mockroot if any
> >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do
> >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue
> >> > > 19:31:03 echo "Cleaning up mock $mock_conf"
> >> > > 19:31:03 mock_root=3D"${mock_conf_file##*/}"
> >> > > 19:31:03 mock_root=3D"${mock_root%.*}"
> >> > > 19:31:03 my_mock=3D"/usr/bin/mock"
> >> > > 19:31:03 my_mock+=3D"
--configdir=3D${mock_conf_file%/*}"
> >> > > 19:31:03 my_mock+=3D" --root=3D${mock_root}"
> >> > > 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE"
> >> > > 19:31:03
> >> > > 19:31:03 #TODO: investigate why mock --clean fails to umount
> >> > > certain dirs sometimes,
> >> > > 19:31:03 #so we can use it instead of manually doing all thi=
s.
> >> > > 19:31:03 echo "Killing all mock
orphan processes, if any."
> >> > > 19:31:03 $my_mock \
> >> > > 19:31:03 --orphanskill \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to kill orphans on
$chroot=
=2E"
> >> > > 19:31:03 failed=3Dtrue
> >> > > 19:31:03 }
> >> > > 19:31:03
> >> > > 19:31:03 mock_root=3D"$(\
> >> > > 19:31:03 grep \
> >> > > 19:31:03 -Po
"(?<=3Dconfig_opts\['root'\] =3D ')[^']=
*" \
> >> > > 19:31:03
"$mock_conf_file" \
> >> > > 19:31:03 )" || :
> >> > > 19:31:03 [[ "$mock_root" ]] || continue
> >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep
"$mock_=
root"))
> >> || :
> >> > > 19:31:03 if [[ "$mounts" ]]; then
> >> > > 19:31:03 echo "Found mounted dirs inside the chroot
$chr=
oot.
> >> > > Trying to umount."
> >> > > 19:31:03 fi
> >> > > 19:31:03 for mount in "${mounts[@]}"; do
> >> > > 19:31:03 sudo umount "$mount" \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to umount
$mount."
> >> > > 19:31:03 failed=3Dtrue
> >> > > 19:31:03 }
> >> > > 19:31:03 done
> >> > > 19:31:03 done
> >> > > 19:31:03
> >> > > 19:31:03 # Clean any leftover chroot from other jobs
> >> > > 19:31:03 for mock_root in /var/lib/mock/*; do
> >> > > 19:31:03 this_chroot_failed=3Dfalse
> >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep
"$mock_=
root"))
> >> || :
> >> > > 19:31:03 if [[ "$mounts" ]]; then
> >> > > 19:31:03 echo "Found mounted dirs inside the chroot
> >> $mock_root." \
> >> > > 19:31:03 "Trying to umount."
> >> > > 19:31:03 fi
> >> > > 19:31:03 for mount in "${mounts[@]}"; do
> >> > > 19:31:03 sudo umount "$mount" \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to umount
$mount."
> >> > > 19:31:03 failed=3Dtrue
> >> > > 19:31:03 this_chroot_failed=3Dtrue
> >> > > 19:31:03 }
> >> > > 19:31:03 done
> >> > > 19:31:03 if ! $this_chroot_failed; then
> >> > > 19:31:03 sudo rm -rf "$mock_root"
> >> > > 19:31:03 fi
> >> > > 19:31:03 done
> >> > > 19:31:03
> >> > > 19:31:03 if $failed; then
> >> > > 19:31:03 echo "Aborting."
> >> > > 19:31:03 exit 1
> >> > > 19:31:03 fi
> >> > > 19:31:03
> >> > > 19:31:03 # remove mock system cache, we will setup proxies to do=
the
> >> > > caching and this
> >> > > 19:31:03 # takes lots of space between runs
> >> > > 19:31:03 shopt -u nullglob
> >> > > 19:31:03 sudo rm -Rf /var/cache/mock/*
> >> > > 19:31:03
> >> > > 19:31:03 # restore the permissions in the working dir, as someti=
mes it
> >> > > leaves files
> >> > > 19:31:03 # owned by root and then the 'cleanup workspace'
from j=
enkins
> >> > > job fails to
> >> > > 19:31:03 # clean and breaks the jobs
> >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE"
> >> > > 19:31:03
> >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson3681642653853563663.sh
> >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh
> >> > > 19:31:03 shell-scripts/mock_cleanup.sh
> >> > > 19:31:03 + shopt -s nullglob
> >> > > 19:31:03 +
> >> WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=
64
> >> > > 19:31:03 + cat
> >> > > 19:31:03
> >> ____________________________________________________________________=
___
> >> > > 19:31:03
> >> ####################################################################=
###
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03 # CLEANUP
> > >> #
> > >> > > 19:31:03 #
> > >> #
> >> > > 19:31:03
> >> ####################################################################=
###
> >> > > 19:31:03 + logs=3D(./*log ./*/logs)
> >> > > 19:31:03 + [[ -n ./vdsm/logs ]]
> >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs
> >> > > 19:31:03 ./vdsm/logs/
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr=
=2Elog
> >> > > 19:31:03
./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log
> >> > > 19:31:03 + rm -rf ./vdsm/logs
> >> > > 19:31:03 + failed=3Dfalse
> >> > > 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*)
> >> > > 19:31:03 + for mock_conf_file in
'"${mock_confs[@]}"'
> >> > > 19:31:03 + [[ -n
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock=
er-epel-7-x86_64.el7.cfg
> >> > > ]]
> >> > > 19:31:03 + echo 'Cleaning up mock '
> >> > > 19:31:03 Cleaning up mock
> >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg
> >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7
> >> > > 19:31:03 + my_mock=3D/usr/bin/mock
> >> > > 19:31:03 + my_mock+=3D'
> >> > >
> >> --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8=
6_64/vdsm'
> >> > > 19:31:03 + my_mock+=3D'
--root=3Dmocker-epel-7-x86_64.el7'
> >> > > 19:31:03 + my_mock+=3D'
> >> > >
> >> --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8=
6_64'
> >> > > 19:31:03 + echo 'Killing all mock orphan
processes, if any.'
> >> > > 19:31:03 Killing all mock orphan processes, if any.
> >> > > 19:31:03 + /usr/bin/mock
> >> > >
> >> --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8=
6_64/vdsm
> >> > > --root=3Dmocker-epel-7-x86_64.el7
> >> > > --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el=
7-x86_64
> >> > > --orphanskill
> >> > > 19:31:04 WARNING: Could not find required logging config file:
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg=
ing.ini.
> >> > > Using default...
> >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version =
=3D
> >> 3.4.3)...
> >> > > 19:31:04 Start: init plugins
> >> > > 19:31:04 INFO: selinux enabled
> >> > > 19:31:04 Finish: init plugins
> >> > > 19:31:04 Start: run
> >> > > 19:31:04 Finish: run
> >> > > 19:31:04 ++ grep -Po
'(?<=3Dconfig_opts\['\''root'\''\] =3D '\''=
)[^'\'']*'
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock=
er-epel-7-x86_64.el7.cfg
> >> > > 19:31:04 +
mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00e=
dd8fc
> >> > > 19:31:04 + [[ -n
epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc =
]]
> >> > > 19:31:04 + mounts=3D($(mount | awk
'{print $3}' | grep "$mock_ro=
ot"))
> > >> > > 19:31:04 ++ mount
> > >> > > 19:31:04 ++ grep
epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 ++ awk '{print $3}'
> > >> > > 19:31:04 + :
> > >> > > 19:31:04 + [[ -n '' ]]
> > >> > > 19:31:04 + for mock_root in '/var/lib/mock/*'
> > >> > > 19:31:04 + this_chroot_failed=3Dfalse
> >> > > 19:31:04 + mounts=3D($(mount | awk
'{print $3}' | grep "$mock_ro=
ot"))
> >> > > 19:31:04 ++ mount
> >> > > 19:31:04 ++ grep
> >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 ++ awk '{print $3}'
> >> > > 19:31:04 + :
> >> > > 19:31:04 + [[ -n '' ]]
> >> > > 19:31:04 + false
> >> > > 19:31:04 + sudo rm -rf
> >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 + false
> >> > > 19:31:04 + shopt -u nullglob
> >> > > 19:31:04 + sudo rm -Rf
> >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 + sudo chown -R jenkins
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > > 19:31:04 POST BUILD TASK : SUCCESS
> >> > > 19:31:04 END OF POST BUILD TASK : 0
> >> > > 19:31:04 Archiving artifacts
> >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as
failure
> >> > > 19:31:04 Started calculate disk usage of build
> >> > > 19:31:04 Finished Calculation of disk usage of build in 0
seconds
> >> > > 19:31:04 Started calculate disk usage of workspace
> >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 se=
conds
> >> > > 19:31:04 Finished: FAILURE
> >> > _______________________________________________
> >> > Infra mailing list
> >> > Infra(a)ovirt.org
> >> >
http://lists.ovirt.org/mailman/listinfo/infra
> >>
> >> --
> >> David Caro
> >>
> >> Red Hat S.L.
> >> Continuous Integration Engineer - EMEA ENG Virtualization R&D
> >>
> >> Tel.: +420 532 294 605
> >> Email: dcaro(a)redhat.com
> >> IRC: dcaro|dcaroest@{freenode|oftc|redhat}
> >> Web:
www.redhat.com
> >> RHT Global #: 82-62605
> >>
> >> _______________________________________________
> >> Infra mailing list
> >> Infra(a)ovirt.org
> >>
http://lists.ovirt.org/mailman/listinfo/infra
> >>
> >>
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> >
http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
=20
--=20
David Caro
=20
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
=20
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web:
www.redhat.com
RHT Global #: 82-62605
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web:
www.redhat.com
RHT Global #: 82-62605
--t5NgoZwlhlUmGr82
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJXOuUyAAoJEEBxx+HSYmnDQAQH/Ax1p+bYMX9BXpNUewDwHsbm
cfDpl2FOYO37OpFCWpnnrWzoZByq1pgpaaub43efKodCbfTzgX1B7Yo9nw/zI1eA
unxBBsnNHHt9K/4dR2ybgCKTR10tlb7ejCaODwIYv7SWGnepFZGlzKQxr7OTQP2I
UrjBmU58Z1+hW/3YsGkHqB/ZQcD0lcYzBigtwoj0Ce3QZn43508B7uBVZgCVKXmM
Vt04CtYQEB79aCdNCU/iYBUxg11bQCjjJntXaOXfFlcYXYQ7DTIxnZhXoXPFHyCj
gkWBTZEbyCoZfjI1868l2jMyR3hoZWPEdrK3v9SkF5CJo8EyUxzeWE3rbSTB9R8=
=6ebz
-----END PGP SIGNATURE-----
--t5NgoZwlhlUmGr82--