On Tue, May 17, 2016 at 12:32 PM, dcaro <dcaro@redhat.com> wrote:

On 05/17 11:20, dcaro wrote:
> On 05/17 10:36, Gil Shinar wrote:
> > Hi,
> >
> > I have tried to debug this issue yesterday with the help of Barak.
> > I don't think that the problem is history as I saw another failure like
> > this yesterday (it happens from time to time). Problem is the slave it
> > builds on being taken by another build and cleans the workspace. I tried to
> > look for logs under the workspace and they missing probably because a newer
> > build is being built on the same slave.
>
> How did you find out about that?
> That has happened before when the same slave was added twice on jenkins with
> different names, but the behavior was way different (you'd get things like the
> dir does not exist and such, not errors when running yum inside mock)

An example of yum failure:

http://jenkins.ovirt.org/view/Master%20branch%20per%20project/view/vdsm/job/vdsm_master_check-patch-el7-x86_64/1523/

You can see there the log in the console:

03:00:58 Start: yum install
03:01:07 ERROR: Command failed. See logs for output.
03:01:07 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts

Then, in the archived artifacts logs.tgz, under
vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log, you can see the error log:

DEBUG util.py:474: Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/', '--releasever', '7', 'install', '@buildsys-build', '--setopt=tsflags=nocontexts'] with env {'SHELL': '/bin/bash', 'CCACHE_DIR': '/tmp/ccache', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'CCACHE_UMASK': '002', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'HOSTNAME': 'mock', 'LD_PRELOAD': '/tmp/tmpxqactgv2/$LIB/nosync.so'} and shell False
DEBUG util.py:399: Yum command has been deprecated, use dnf instead.
DEBUG util.py:399: See 'man dnf' and 'man yum2dnf' for more information.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/bbf11fe5f13b4c6d0a9a242b422885c6ed41e05ae718d4ea3abfc3550ef4f243-comps-epel7.xml.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/4e6f7cf18ae8bc3553da486c7847ac8f9a50671406d26c72a4d0765f914c5c76-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:399: Trying other mirror.
DEBUG util.py:399: One of the configured repositories failed ("Custom epel"),
DEBUG util.py:399: and yum doesn't have enough cached data to continue. At this point the only
DEBUG util.py:399: safe thing yum can do is fail. There are a few ways to work "fix" this:
DEBUG util.py:399: 1. Contact the upstream for the repository and get them to fix the problem.
DEBUG util.py:399: 2. Reconfigure the baseurl/etc. for the repository, to point to a working
DEBUG util.py:399: upstream. This is most often useful if you are using a newer
DEBUG util.py:399: distribution release than is supported by the repository (and the
DEBUG util.py:399: packages for the previous distribution release still work).
DEBUG util.py:399: 3. Disable the repository, so yum won't use it by default. Yum will then
DEBUG util.py:399: just ignore the repository until you permanently enable it again or use
DEBUG util.py:399: --enablerepo for temporary usage:
DEBUG util.py:399: yum-config-manager --disable epel
DEBUG util.py:399: 4. Configure the failing repository to be skipped, if it is unavailable.
DEBUG util.py:399: Note that yum will try to contact the repo. when it runs most commands,
DEBUG util.py:399: so will have to try and fail each time (and thus. yum will be be much
DEBUG util.py:399: slower). If it is a very temporary problem though, this is often a nice
DEBUG util.py:399: compromise:
DEBUG util.py:399: yum-config-manager --save --setopt=epel.skip_if_unavailable=true
DEBUG util.py:399: failure: repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz from epel: [Errno 256] No more mirrors to try.
DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:521: Child return code was: 1

That usually means that the mirrors were in the middle of an update, and the
md5sums in the metadata were not consistent at that point it time (happens from
time to time).

We could try to minimize that issue handling a mirror ourselves, though that
would mean another service to maintain (with it's own potential issues, like
not being updated and such, but that's another subject)

>
> >
> > Gil
> >
> > On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
> >
> > > Can we increase the history for builds on that job? We should be able to
> > > debug jobs at least a week back. Artifacts are not needed
> > > On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
> > >
> > >> On 05/15 23:23, Nir Soffer wrote:
> > >> > Another instance:
> > >> >
> > >> > 19:38:15 Start: yum install
> > >> > 19:38:22 ERROR: Command failed. See logs for output.
> > >>
> > >> That means that there was an issue with the yum repos, most common causes
> > >> are:
> > >>
> > >> * The repos were actually down (mirrors fail once a day usually, while
> > >> syncing
> > >> the rpms, though the cause is just a guess)
> > >> * Repoproxy (that we use to cache rpms) was overloaded
> > >>
> > >> Both jobs histories are now gone, in order to be able to debug those
> > >> issues
> > >> (for the next time), try looking into the mock logs, under the 'logs.tgz'
> > >> file
> > >> that's archived in the job, then under the path:
> > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> > >> or
> > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
> > >>
> > >> Usually in the log named 'root.log' (check which one has more size).
> > >>
> > >> >
> > >> >
> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
> > >>
> > >> >
> > >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
> > >> wrote:
> > >> > >
> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
> > >> > >
> > >> > > 19:30:56 Start: yum install
> > >> > > 19:31:03 ERROR: Command failed. See logs for output.
> > >> > >
> > >> > > Including the to make sure it will not disapper
> > >> > > ----
> > >> > >
> > >> > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550
> > >> > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested)
> > >> in
> > >> > > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > >> > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10
> > >> > > 19:30:49 Fetching changes from the remote Git repository
> > >> > > 19:30:49 > git config remote.origin.url
> > >> > > git://gerrit.ovirt.org/vdsm.git # timeout=10
> > >> > > 19:30:49 Cleaning workspace
> > >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=10
> > >> > > 19:30:49 Resetting working tree
> > >> > > 19:30:49 > git reset --hard # timeout=10
> > >> > > 19:30:49 > git clean -fdx # timeout=10
> > >> > > 19:30:49 Pruning obsolete local branches
> > >> > > 19:30:49 Fetching upstream changes from git://
> > >> gerrit.ovirt.org/vdsm.git
> > >> > > 19:30:49 > git --version # timeout=10
> > >> > > 19:30:49 > git -c core.askpass=true fetch --tags --progress
> > >> > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune
> > >> > > 19:30:52 > git rev-parse
> > >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10
> > >> > > 19:30:52 Checking out Revision
> > >> 0940208483f3a21261eb5d725348e65c23becdc0 (master)
> > >> > > 19:30:52 > git config core.sparsecheckout # timeout=10
> > >> > > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0
> > >> > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10
> > >> > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d #
> > >> timeout=10
> > >> > > 19:30:52 > git branch -a # timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} #
> > >> timeout=10
> > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} #
> > >> timeout=10
> > >> > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10
> > >> > > 19:30:53 Fetching changes from the remote Git repository
> > >> > > 19:30:53 > git config remote.origin.url
> > >> > > git://gerrit.ovirt.org/jenkins.git # timeout=10
> > >> > > 19:30:53 Cleaning workspace
> > >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=10
> > >> > > 19:30:53 Resetting working tree
> > >> > > 19:30:53 > git reset --hard # timeout=10
> > >> > > 19:30:53 > git clean -fdx # timeout=10
> > >> > > 19:30:53 Pruning obsolete local branches
> > >> > > 19:30:53 Fetching upstream changes from git://
> > >> gerrit.ovirt.org/jenkins.git
> > >> > > 19:30:53 > git --version # timeout=10
> > >> > > 19:30:53 > git -c core.askpass=true fetch --tags --progress
> > >> > > git://gerrit.ovirt.org/jenkins.git
> > >> +refs/heads/*:refs/remotes/origin/*
> > >> > > --prune
> > >> > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=10
> > >> > > 19:30:53 Checking out Revision
> > >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master)
> > >> > > 19:30:53 > git config core.sparsecheckout # timeout=10
> > >> > > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9
> > >> > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 #
> > >> timeout=10
> > >> > > 19:30:53 > git branch -a # timeout=10
> > >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10
> > >> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e
> > >> > > /tmp/hudson8265618520208925180.sh
> > >> > > 19:30:53 shell-scripts/cleanup_slave.sh
> > >> > > 19:30:53
> > >> ###################################################################
> > >> > > 19:30:53 # Cleaning up slave
> > >> #
> > >> > > 19:30:53
> > >> ###################################################################
> > >> > > 19:30:53 Filesystem Size Used Avail
> > >> > > Use% Mounted on
> > >> > > 19:30:53 devtmpfs 3.9G 0
> > >> 3.9G 0% /dev
> > >> > > 19:30:53 tmpfs 3.9G 0 3.9G
> > >> > > 0% /dev/shm
> > >> > > 19:30:53 tmpfs 3.9G 2.1M
> > >> 3.9G 1% /run
> > >> > > 19:30:53 tmpfs 3.9G 0 3.9G
> > >> > > 0% /sys/fs/cgroup
> > >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> > >> 27G 20% /
> > >> > > 19:30:53 tmpfs 3.9G 244K
> > >> 3.9G 1% /tmp
> > >> > > 19:30:53 /dev/vda1 988M 94M
> > >> 828M 11% /boot
> > >> > > 19:30:53 tmpfs 799M 0 799M
> > >> > > 0% /run/user/1000
> > >> > > 19:30:53
> > >> ---------------------------------------------------------------
> > >> > > 19:30:53 Cleaning up postgres databases
> > >> > > 19:30:53 Postgres installation not found, skipping
> > >> > > 19:30:53 Cleaning up journal logs (if any)
> > >> > > 19:30:53 Redirecting to /bin/systemctl restart
> > >> systemd-journald.service
> > >> > > 19:30:54 Cleaning up /var/tmp
> > >> > > 19:30:54 done
> > >> > > 19:30:54 Emptying some common logs
> > >> > > 19:30:54 /var/log/wtmp
> > >> > > 19:30:54 Done
> > >> > > 19:30:54 Making sure there are no device mappings...
> > >> > > 19:30:54 Removing the used loop devices...
> > >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service
> > >> > > 19:30:54
> > >> ---------------------------------------------------------------
> > >> > > 19:30:54 Filesystem Size Used Avail
> > >> > > Use% Mounted on
> > >> > > 19:30:54 devtmpfs 3.9G 0
> > >> 3.9G 0% /dev
> > >> > > 19:30:54 tmpfs 3.9G 0 3.9G
> > >> > > 0% /dev/shm
> > >> > > 19:30:54 tmpfs 3.9G 2.1M
> > >> 3.9G 1% /run
> > >> > > 19:30:54 tmpfs 3.9G 0 3.9G
> > >> > > 0% /sys/fs/cgroup
> > >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> > >> 27G 20% /
> > >> > > 19:30:54 tmpfs 3.9G 244K
> > >> 3.9G 1% /tmp
> > >> > > 19:30:54 /dev/vda1 988M 94M
> > >> 828M 11% /boot
> > >> > > 19:30:54 tmpfs 799M 0 799M
> > >> > > 0% /run/user/1000
> > >> > > 19:30:54
> > >> ###################################################################
> > >> > > 19:30:54 # Slave cleanup done
> > >> #
> > >> > > 19:30:54
> > >> ###################################################################
> > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> > >> > > /tmp/hudson7354644880288923842.sh
> > >> > > 19:30:54 + echo shell-scripts/global_setup.sh
> > >> > > 19:30:54 shell-scripts/global_setup.sh
> > >> > > 19:30:54 + rm -rf
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> > >> > > 19:30:54 + mkdir -p
> > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> > >> > > /tmp/hudson4023353013258685526.sh
> > >> > > 19:30:54 + echo shell-scripts/mock_setup.sh
> > >> > > 19:30:54 shell-scripts/mock_setup.sh
> > >> > > 19:30:54 + shopt -s nullglob
> > >> > > 19:30:54 + [[ '' == \t\r\u\e ]]
> > >> > > 19:30:54 + failed=false
> > >> > > 19:30:54 +
> > >> mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock
> > >> > > 19:30:54 + chroots=("$WORKSPACE"/mock/*)
> > >> > > 19:30:54 + false
> > >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts
> > >> > > 19:30:54 + mkdir -p mock exported-artifacts
> > >> > > 19:30:54 + chgrp mock mock
> > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
> > >> > > 19:30:54 + chmod g+rws mock
> > >> > > 19:30:54 + sudo touch
> > >> > > 19:30:54 + :
> > >> > > 19:30:54 + sudo yum clean all
> > >> > > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf
> > >> > > clean all'.
> > >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information.
> > >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run:
> > >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2
> > >> migrate'
> > >> > > 19:30:54
> > >> > > 19:30:55 Cleaning repos: updates ci-tools fedora
> > >> > > 19:30:55 Cleaning up Everything
> > >> > > 19:30:55 + exit 0
> > >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e
> > >> > > /tmp/hudson4405890361943185034.sh
> > >> > > 19:30:55 shell-scripts/standard_check_patch.sh
> > >> > > 19:30:55 ##########################################################
> > >> > > 19:30:55 ##########################################################
> > >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env:
> > >> el7:epel-7-x86_64
> > >> > > 19:30:55 ##########################################################
> > >> > > 19:30:55
> > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script:
> > >> > > automation/check-patch.sh
> > >> > > 19:30:55
> > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > >> > > 19:30:55 Using base mock conf
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
> > >> > > 19:30:55 Using proxified config
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
> > >> > > 19:30:55 Generating temporary mock conf
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
> > >> > > 19:30:55 Skipping mount points
> > >> > > 19:30:55 Using temp chroot =
> > >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:30:55 Adding repo ovirt-snapshot ->
> > >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7
> > >> > > 19:30:55 Adding repo ovirt-snapshot-static ->
> > >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7
> > >> > > 19:30:55 Adding repo centos-glusterfs ->
> > >> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/
> > >> > > 19:30:55 Adding repo centos-ovirt40-candidate ->
> > >> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/
> > >> > > 19:30:55 Adding repo lago ->
> > >> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7
> > >> > > 19:30:55 Adding repo epel ->
> > >> http://download.fedoraproject.org/pub/epel/7/x86_64
> > >> > > 19:30:55 ========== Initializing chroot
> > >> > > 19:30:55 mock \
> > >> > > 19:30:55
> > >> > >
> > >> --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> > >> > > \
> > >> > > 19:30:55 --root="mocker-epel-7-x86_64.el7" \
> > >> > > 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \
> > >> > > 19:30:55 --init
> > >> > > 19:30:55 WARNING: Could not find required logging config file:
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> > >> > > Using default...
> > >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version =
> > >> 3.4.3)...
> > >> > > 19:30:55 Start: init plugins
> > >> > > 19:30:55 INFO: selinux enabled
> > >> > > 19:30:55 Finish: init plugins
> > >> > > 19:30:55 Start: run
> > >> > > 19:30:55 Start: clean chroot
> > >> > > 19:30:55 Finish: clean chroot
> > >> > > 19:30:55 Start: chroot init
> > >> > > 19:30:55 INFO: calling preinit hooks
> > >> > > 19:30:55 INFO: enabled root cache
> > >> > > 19:30:55 INFO: enabled yum cache
> > >> > > 19:30:55 Start: cleaning yum metadata
> > >> > > 19:30:55 Finish: cleaning yum metadata
> > >> > > 19:30:55 INFO: enabled ccache
> > >> > > 19:30:55 Mock Version: 1.2.14
> > >> > > 19:30:55 INFO: Mock Version: 1.2.14
> > >> > > 19:30:56 Start: yum install
> > >> > > 19:31:03 ERROR: Command failed. See logs for output.
> > >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot
> > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/
> > >> > > --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts
> > >> > > 19:31:03 Init took 8 seconds
> > >> > > 19:31:03 ============================
> > >> > > 19:31:03
> > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh
> > >> > > chroot finished
> > >> > > 19:31:03 @@ took 8 seconds
> > >> > > 19:31:03 @@ rc = 1
> > >> > > 19:31:03
> > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > >> > > 19:31:03 ##########################################################
> > >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env:
> > >> el7:epel-7-x86_64
> > >> > > 19:31:03 ## took 8 seconds
> > >> > > 19:31:03 ## rc = 1
> > >> > > 19:31:03 ##########################################################
> > >> > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> > >> > > 19:31:03 ##! Last 20 log enties:
> > >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log
> > >> > > 19:31:03 ##!
> > >> > > 19:31:03 ========== Initializing chroot
> > >> > > 19:31:03 mock \
> > >> > > 19:31:03
> > >> > >
> > >> --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> > >> > > \
> > >> > > 19:31:03 --root="mocker-epel-7-x86_64.el7" \
> > >> > > 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \
> > >> > > 19:31:03 --init
> > >> > > 19:31:03 Init took 8 seconds
> > >> > > 19:31:03 ============================
> > >> > > 19:31:03 ##!
> > >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >> > > 19:31:03 ##!########################################################
> > >> > > 19:31:03 Build step 'Execute shell' marked build as failure
> > >> > > 19:31:03 Performing Post build task...
> > >> > > 19:31:03 Match found for :.* : True
> > >> > > 19:31:03 Logical operation result is TRUE
> > >> > > 19:31:03 Running script : #!/bin/bash -xe
> > >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh"
> > >> > > 19:31:03
> > >> > > 19:31:03 shopt -s nullglob
> > >> > > 19:31:03
> > >> > > 19:31:03
> > >> > > 19:31:03 WORKSPACE="$PWD"
> > >> > > 19:31:03
> > >> > > 19:31:03 # Make clear this is the cleanup, helps reading the jenkins
> > >> logs
> > >> > > 19:31:03 cat <<EOC
> > >> > > 19:31:03
> > >> _______________________________________________________________________
> > >> > > 19:31:03
> > >> #######################################################################
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03 # CLEANUP
> > >> #
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03
> > >> #######################################################################
> > >> > > 19:31:03 EOC
> > >> > > 19:31:03
> > >> > > 19:31:03
> > >> > > 19:31:03 # Archive the logs, we want them anyway
> > >> > > 19:31:03 logs=(
> > >> > > 19:31:03 ./*log
> > >> > > 19:31:03 ./*/logs
> > >> > > 19:31:03 )
> > >> > > 19:31:03 if [[ "$logs" ]]; then
> > >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
> > >> > > 19:31:03 rm -rf "${logs[@]}"
> > >> > > 19:31:03 fi
> > >> > > 19:31:03
> > >> > > 19:31:03 # stop any processes running inside the chroot
> > >> > > 19:31:03 failed=false
> > >> > > 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*)
> > >> > > 19:31:03 # Clean current jobs mockroot if any
> > >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do
> > >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue
> > >> > > 19:31:03 echo "Cleaning up mock $mock_conf"
> > >> > > 19:31:03 mock_root="${mock_conf_file##*/}"
> > >> > > 19:31:03 mock_root="${mock_root%.*}"
> > >> > > 19:31:03 my_mock="/usr/bin/mock"
> > >> > > 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}"
> > >> > > 19:31:03 my_mock+=" --root=${mock_root}"
> > >> > > 19:31:03 my_mock+=" --resultdir=$WORKSPACE"
> > >> > > 19:31:03
> > >> > > 19:31:03 #TODO: investigate why mock --clean fails to umount
> > >> > > certain dirs sometimes,
> > >> > > 19:31:03 #so we can use it instead of manually doing all this.
> > >> > > 19:31:03 echo "Killing all mock orphan processes, if any."
> > >> > > 19:31:03 $my_mock \
> > >> > > 19:31:03 --orphanskill \
> > >> > > 19:31:03 || {
> > >> > > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot."
> > >> > > 19:31:03 failed=true
> > >> > > 19:31:03 }
> > >> > > 19:31:03
> > >> > > 19:31:03 mock_root="$(\
> > >> > > 19:31:03 grep \
> > >> > > 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \
> > >> > > 19:31:03 "$mock_conf_file" \
> > >> > > 19:31:03 )" || :
> > >> > > 19:31:03 [[ "$mock_root" ]] || continue
> > >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> > >> || :
> > >> > > 19:31:03 if [[ "$mounts" ]]; then
> > >> > > 19:31:03 echo "Found mounted dirs inside the chroot $chroot.
> > >> > > Trying to umount."
> > >> > > 19:31:03 fi
> > >> > > 19:31:03 for mount in "${mounts[@]}"; do
> > >> > > 19:31:03 sudo umount "$mount" \
> > >> > > 19:31:03 || {
> > >> > > 19:31:03 echo "ERROR: Failed to umount $mount."
> > >> > > 19:31:03 failed=true
> > >> > > 19:31:03 }
> > >> > > 19:31:03 done
> > >> > > 19:31:03 done
> > >> > > 19:31:03
> > >> > > 19:31:03 # Clean any leftover chroot from other jobs
> > >> > > 19:31:03 for mock_root in /var/lib/mock/*; do
> > >> > > 19:31:03 this_chroot_failed=false
> > >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> > >> || :
> > >> > > 19:31:03 if [[ "$mounts" ]]; then
> > >> > > 19:31:03 echo "Found mounted dirs inside the chroot
> > >> $mock_root." \
> > >> > > 19:31:03 "Trying to umount."
> > >> > > 19:31:03 fi
> > >> > > 19:31:03 for mount in "${mounts[@]}"; do
> > >> > > 19:31:03 sudo umount "$mount" \
> > >> > > 19:31:03 || {
> > >> > > 19:31:03 echo "ERROR: Failed to umount $mount."
> > >> > > 19:31:03 failed=true
> > >> > > 19:31:03 this_chroot_failed=true
> > >> > > 19:31:03 }
> > >> > > 19:31:03 done
> > >> > > 19:31:03 if ! $this_chroot_failed; then
> > >> > > 19:31:03 sudo rm -rf "$mock_root"
> > >> > > 19:31:03 fi
> > >> > > 19:31:03 done
> > >> > > 19:31:03
> > >> > > 19:31:03 if $failed; then
> > >> > > 19:31:03 echo "Aborting."
> > >> > > 19:31:03 exit 1
> > >> > > 19:31:03 fi
> > >> > > 19:31:03
> > >> > > 19:31:03 # remove mock system cache, we will setup proxies to do the
> > >> > > caching and this
> > >> > > 19:31:03 # takes lots of space between runs
> > >> > > 19:31:03 shopt -u nullglob
> > >> > > 19:31:03 sudo rm -Rf /var/cache/mock/*
> > >> > > 19:31:03
> > >> > > 19:31:03 # restore the permissions in the working dir, as sometimes it
> > >> > > leaves files
> > >> > > 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins
> > >> > > job fails to
> > >> > > 19:31:03 # clean and breaks the jobs
> > >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE"
> > >> > > 19:31:03
> > >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> > >> > > /tmp/hudson3681642653853563663.sh
> > >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh
> > >> > > 19:31:03 shell-scripts/mock_cleanup.sh
> > >> > > 19:31:03 + shopt -s nullglob
> > >> > > 19:31:03 +
> > >> WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > >> > > 19:31:03 + cat
> > >> > > 19:31:03
> > >> _______________________________________________________________________
> > >> > > 19:31:03
> > >> #######################################################################
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03 # CLEANUP
> > >> #
> > >> > > 19:31:03 #
> > >> #
> > >> > > 19:31:03
> > >> #######################################################################
> > >> > > 19:31:03 + logs=(./*log ./*/logs)
> > >> > > 19:31:03 + [[ -n ./vdsm/logs ]]
> > >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs
> > >> > > 19:31:03 ./vdsm/logs/
> > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log
> > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log
> > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log
> > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log
> > >> > > 19:31:03 + rm -rf ./vdsm/logs
> > >> > > 19:31:03 + failed=false
> > >> > > 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*)
> > >> > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"'
> > >> > > 19:31:03 + [[ -n
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> > >> > > ]]
> > >> > > 19:31:03 + echo 'Cleaning up mock '
> > >> > > 19:31:03 Cleaning up mock
> > >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg
> > >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7
> > >> > > 19:31:03 + my_mock=/usr/bin/mock
> > >> > > 19:31:03 + my_mock+='
> > >> > >
> > >> --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
> > >> > > 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7'
> > >> > > 19:31:03 + my_mock+='
> > >> > >
> > >> --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
> > >> > > 19:31:03 + echo 'Killing all mock orphan processes, if any.'
> > >> > > 19:31:03 Killing all mock orphan processes, if any.
> > >> > > 19:31:03 + /usr/bin/mock
> > >> > >
> > >> --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
> > >> > > --root=mocker-epel-7-x86_64.el7
> > >> > > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > >> > > --orphanskill
> > >> > > 19:31:04 WARNING: Could not find required logging config file:
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> > >> > > Using default...
> > >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version =
> > >> 3.4.3)...
> > >> > > 19:31:04 Start: init plugins
> > >> > > 19:31:04 INFO: selinux enabled
> > >> > > 19:31:04 Finish: init plugins
> > >> > > 19:31:04 Start: run
> > >> > > 19:31:04 Finish: run
> > >> > > 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
> > >> > >
> > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> > >> > > 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]]
> > >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> > >> > > 19:31:04 ++ mount
> > >> > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 ++ awk '{print $3}'
> > >> > > 19:31:04 + :
> > >> > > 19:31:04 + [[ -n '' ]]
> > >> > > 19:31:04 + for mock_root in '/var/lib/mock/*'
> > >> > > 19:31:04 + this_chroot_failed=false
> > >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> > >> > > 19:31:04 ++ mount
> > >> > > 19:31:04 ++ grep
> > >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 ++ awk '{print $3}'
> > >> > > 19:31:04 + :
> > >> > > 19:31:04 + [[ -n '' ]]
> > >> > > 19:31:04 + false
> > >> > > 19:31:04 + sudo rm -rf
> > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 + false
> > >> > > 19:31:04 + shopt -u nullglob
> > >> > > 19:31:04 + sudo rm -Rf
> > >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> > >> > > 19:31:04 + sudo chown -R jenkins
> > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > >> > > 19:31:04 POST BUILD TASK : SUCCESS
> > >> > > 19:31:04 END OF POST BUILD TASK : 0
> > >> > > 19:31:04 Archiving artifacts
> > >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure
> > >> > > 19:31:04 Started calculate disk usage of build
> > >> > > 19:31:04 Finished Calculation of disk usage of build in 0 seconds
> > >> > > 19:31:04 Started calculate disk usage of workspace
> > >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds
> > >> > > 19:31:04 Finished: FAILURE
> > >> > _______________________________________________
> > >> > Infra mailing list
> > >> > Infra@ovirt.org
> > >> > http://lists.ovirt.org/mailman/listinfo/infra
> > >>
> > >> --
> > >> David Caro
> > >>
> > >> Red Hat S.L.
> > >> Continuous Integration Engineer - EMEA ENG Virtualization R&D
> > >>
> > >> Tel.: +420 532 294 605
> > >> Email: dcaro@redhat.com
> > >> IRC: dcaro|dcaroest@{freenode|oftc|redhat}
> > >> Web: www.redhat.com
> > >> RHT Global #: 82-62605
> > >>
> > >> _______________________________________________
> > >> Infra mailing list
> > >> Infra@ovirt.org
> > >> http://lists.ovirt.org/mailman/listinfo/infra
> > >>
> > >>
> > > _______________________________________________
> > > Infra mailing list
> > > Infra@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> > >
> > >
>
> --
> David Caro
>
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>
> Tel.: +420 532 294 605
> Email: dcaro@redhat.com
> IRC: dcaro|dcaroest@{freenode|oftc|redhat}
> Web: www.redhat.com
> RHT Global #: 82-62605

--
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro@redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605