Another job failure: "Command failed. See logs for output"
dcaro
dcaro at redhat.com
Tue May 17 09:20:28 UTC 2016
On 05/17 10:36, Gil Shinar wrote:
> Hi,
>
> I have tried to debug this issue yesterday with the help of Barak.
> I don't think that the problem is history as I saw another failure like
> this yesterday (it happens from time to time). Problem is the slave it
> builds on being taken by another build and cleans the workspace. I tried to
> look for logs under the workspace and they missing probably because a newer
> build is being built on the same slave.
How did you find out about that?
That has happened before when the same slave was added twice on jenkins with
different names, but the behavior was way different (you'd get things like the
dir does not exist and such, not errors when running yum inside mock)
>
> Gil
>
> On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri at redhat.com> wrote:
>
> > Can we increase the history for builds on that job? We should be able to
> > debug jobs at least a week back. Artifacts are not needed
> > On May 16, 2016 11:42 PM, "David Caro" <dcaro at redhat.com> wrote:
> >
> >> On 05/15 23:23, Nir Soffer wrote:
> >> > Another instance:
> >> >
> >> > 19:38:15 Start: yum install
> >> > 19:38:22 ERROR: Command failed. See logs for output.
> >>
> >> That means that there was an issue with the yum repos, most common causes
> >> are:
> >>
> >> * The repos were actually down (mirrors fail once a day usually, while
> >> syncing
> >> the rpms, though the cause is just a guess)
> >> * Repoproxy (that we use to cache rpms) was overloaded
> >>
> >> Both jobs histories are now gone, in order to be able to debug those
> >> issues
> >> (for the next time), try looking into the mock logs, under the 'logs.tgz'
> >> file
> >> that's archived in the job, then under the path:
> >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> >> or
> >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
> >>
> >> Usually in the log named 'root.log' (check which one has more size).
> >>
> >> >
> >> >
> >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
> >>
> >> >
> >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer at redhat.com>
> >> wrote:
> >> > >
> >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
> >> > >
> >> > > 19:30:56 Start: yum install
> >> > > 19:31:03 ERROR: Command failed. See logs for output.
> >> > >
> >> > > Including the to make sure it will not disapper
> >> > > ----
> >> > >
> >> > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550
> >> > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested)
> >> in
> >> > > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10
> >> > > 19:30:49 Fetching changes from the remote Git repository
> >> > > 19:30:49 > git config remote.origin.url
> >> > > git://gerrit.ovirt.org/vdsm.git # timeout=10
> >> > > 19:30:49 Cleaning workspace
> >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=10
> >> > > 19:30:49 Resetting working tree
> >> > > 19:30:49 > git reset --hard # timeout=10
> >> > > 19:30:49 > git clean -fdx # timeout=10
> >> > > 19:30:49 Pruning obsolete local branches
> >> > > 19:30:49 Fetching upstream changes from git://
> >> gerrit.ovirt.org/vdsm.git
> >> > > 19:30:49 > git --version # timeout=10
> >> > > 19:30:49 > git -c core.askpass=true fetch --tags --progress
> >> > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune
> >> > > 19:30:52 > git rev-parse
> >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10
> >> > > 19:30:52 Checking out Revision
> >> 0940208483f3a21261eb5d725348e65c23becdc0 (master)
> >> > > 19:30:52 > git config core.sparsecheckout # timeout=10
> >> > > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0
> >> > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10
> >> > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d #
> >> timeout=10
> >> > > 19:30:52 > git branch -a # timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} #
> >> timeout=10
> >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} #
> >> timeout=10
> >> > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10
> >> > > 19:30:53 Fetching changes from the remote Git repository
> >> > > 19:30:53 > git config remote.origin.url
> >> > > git://gerrit.ovirt.org/jenkins.git # timeout=10
> >> > > 19:30:53 Cleaning workspace
> >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=10
> >> > > 19:30:53 Resetting working tree
> >> > > 19:30:53 > git reset --hard # timeout=10
> >> > > 19:30:53 > git clean -fdx # timeout=10
> >> > > 19:30:53 Pruning obsolete local branches
> >> > > 19:30:53 Fetching upstream changes from git://
> >> gerrit.ovirt.org/jenkins.git
> >> > > 19:30:53 > git --version # timeout=10
> >> > > 19:30:53 > git -c core.askpass=true fetch --tags --progress
> >> > > git://gerrit.ovirt.org/jenkins.git
> >> +refs/heads/*:refs/remotes/origin/*
> >> > > --prune
> >> > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=10
> >> > > 19:30:53 Checking out Revision
> >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master)
> >> > > 19:30:53 > git config core.sparsecheckout # timeout=10
> >> > > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9
> >> > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 #
> >> timeout=10
> >> > > 19:30:53 > git branch -a # timeout=10
> >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10
> >> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e
> >> > > /tmp/hudson8265618520208925180.sh
> >> > > 19:30:53 shell-scripts/cleanup_slave.sh
> >> > > 19:30:53
> >> ###################################################################
> >> > > 19:30:53 # Cleaning up slave
> >> #
> >> > > 19:30:53
> >> ###################################################################
> >> > > 19:30:53 Filesystem Size Used Avail
> >> > > Use% Mounted on
> >> > > 19:30:53 devtmpfs 3.9G 0
> >> 3.9G 0% /dev
> >> > > 19:30:53 tmpfs 3.9G 0 3.9G
> >> > > 0% /dev/shm
> >> > > 19:30:53 tmpfs 3.9G 2.1M
> >> 3.9G 1% /run
> >> > > 19:30:53 tmpfs 3.9G 0 3.9G
> >> > > 0% /sys/fs/cgroup
> >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> >> 27G 20% /
> >> > > 19:30:53 tmpfs 3.9G 244K
> >> 3.9G 1% /tmp
> >> > > 19:30:53 /dev/vda1 988M 94M
> >> 828M 11% /boot
> >> > > 19:30:53 tmpfs 799M 0 799M
> >> > > 0% /run/user/1000
> >> > > 19:30:53
> >> ---------------------------------------------------------------
> >> > > 19:30:53 Cleaning up postgres databases
> >> > > 19:30:53 Postgres installation not found, skipping
> >> > > 19:30:53 Cleaning up journal logs (if any)
> >> > > 19:30:53 Redirecting to /bin/systemctl restart
> >> systemd-journald.service
> >> > > 19:30:54 Cleaning up /var/tmp
> >> > > 19:30:54 done
> >> > > 19:30:54 Emptying some common logs
> >> > > 19:30:54 /var/log/wtmp
> >> > > 19:30:54 Done
> >> > > 19:30:54 Making sure there are no device mappings...
> >> > > 19:30:54 Removing the used loop devices...
> >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service
> >> > > 19:30:54
> >> ---------------------------------------------------------------
> >> > > 19:30:54 Filesystem Size Used Avail
> >> > > Use% Mounted on
> >> > > 19:30:54 devtmpfs 3.9G 0
> >> 3.9G 0% /dev
> >> > > 19:30:54 tmpfs 3.9G 0 3.9G
> >> > > 0% /dev/shm
> >> > > 19:30:54 tmpfs 3.9G 2.1M
> >> 3.9G 1% /run
> >> > > 19:30:54 tmpfs 3.9G 0 3.9G
> >> > > 0% /sys/fs/cgroup
> >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G
> >> 27G 20% /
> >> > > 19:30:54 tmpfs 3.9G 244K
> >> 3.9G 1% /tmp
> >> > > 19:30:54 /dev/vda1 988M 94M
> >> 828M 11% /boot
> >> > > 19:30:54 tmpfs 799M 0 799M
> >> > > 0% /run/user/1000
> >> > > 19:30:54
> >> ###################################################################
> >> > > 19:30:54 # Slave cleanup done
> >> #
> >> > > 19:30:54
> >> ###################################################################
> >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson7354644880288923842.sh
> >> > > 19:30:54 + echo shell-scripts/global_setup.sh
> >> > > 19:30:54 shell-scripts/global_setup.sh
> >> > > 19:30:54 + rm -rf
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> >> > > 19:30:54 + mkdir -p
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp
> >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson4023353013258685526.sh
> >> > > 19:30:54 + echo shell-scripts/mock_setup.sh
> >> > > 19:30:54 shell-scripts/mock_setup.sh
> >> > > 19:30:54 + shopt -s nullglob
> >> > > 19:30:54 + [[ '' == \t\r\u\e ]]
> >> > > 19:30:54 + failed=false
> >> > > 19:30:54 +
> >> mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock
> >> > > 19:30:54 + chroots=("$WORKSPACE"/mock/*)
> >> > > 19:30:54 + false
> >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts
> >> > > 19:30:54 + mkdir -p mock exported-artifacts
> >> > > 19:30:54 + chgrp mock mock
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
> >> > > 19:30:54 + chmod g+rws mock
> >> > > 19:30:54 + sudo touch
> >> > > 19:30:54 + :
> >> > > 19:30:54 + sudo yum clean all
> >> > > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf
> >> > > clean all'.
> >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information.
> >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run:
> >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2
> >> migrate'
> >> > > 19:30:54
> >> > > 19:30:55 Cleaning repos: updates ci-tools fedora
> >> > > 19:30:55 Cleaning up Everything
> >> > > 19:30:55 + exit 0
> >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e
> >> > > /tmp/hudson4405890361943185034.sh
> >> > > 19:30:55 shell-scripts/standard_check_patch.sh
> >> > > 19:30:55 ##########################################################
> >> > > 19:30:55 ##########################################################
> >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env:
> >> el7:epel-7-x86_64
> >> > > 19:30:55 ##########################################################
> >> > > 19:30:55
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script:
> >> > > automation/check-patch.sh
> >> > > 19:30:55
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:30:55 Using base mock conf
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
> >> > > 19:30:55 Using proxified config
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
> >> > > 19:30:55 Generating temporary mock conf
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
> >> > > 19:30:55 Skipping mount points
> >> > > 19:30:55 Using temp chroot =
> >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:30:55 Adding repo ovirt-snapshot ->
> >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7
> >> > > 19:30:55 Adding repo ovirt-snapshot-static ->
> >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7
> >> > > 19:30:55 Adding repo centos-glusterfs ->
> >> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/
> >> > > 19:30:55 Adding repo centos-ovirt40-candidate ->
> >> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/
> >> > > 19:30:55 Adding repo lago ->
> >> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7
> >> > > 19:30:55 Adding repo epel ->
> >> http://download.fedoraproject.org/pub/epel/7/x86_64
> >> > > 19:30:55 ========== Initializing chroot
> >> > > 19:30:55 mock \
> >> > > 19:30:55
> >> > >
> >> --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> >> > > \
> >> > > 19:30:55 --root="mocker-epel-7-x86_64.el7" \
> >> > > 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \
> >> > > 19:30:55 --init
> >> > > 19:30:55 WARNING: Could not find required logging config file:
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> >> > > Using default...
> >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version =
> >> 3.4.3)...
> >> > > 19:30:55 Start: init plugins
> >> > > 19:30:55 INFO: selinux enabled
> >> > > 19:30:55 Finish: init plugins
> >> > > 19:30:55 Start: run
> >> > > 19:30:55 Start: clean chroot
> >> > > 19:30:55 Finish: clean chroot
> >> > > 19:30:55 Start: chroot init
> >> > > 19:30:55 INFO: calling preinit hooks
> >> > > 19:30:55 INFO: enabled root cache
> >> > > 19:30:55 INFO: enabled yum cache
> >> > > 19:30:55 Start: cleaning yum metadata
> >> > > 19:30:55 Finish: cleaning yum metadata
> >> > > 19:30:55 INFO: enabled ccache
> >> > > 19:30:55 Mock Version: 1.2.14
> >> > > 19:30:55 INFO: Mock Version: 1.2.14
> >> > > 19:30:56 Start: yum install
> >> > > 19:31:03 ERROR: Command failed. See logs for output.
> >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot
> >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/
> >> > > --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts
> >> > > 19:31:03 Init took 8 seconds
> >> > > 19:31:03 ============================
> >> > > 19:31:03
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh
> >> > > chroot finished
> >> > > 19:31:03 @@ took 8 seconds
> >> > > 19:31:03 @@ rc = 1
> >> > > 19:31:03
> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> >> > > 19:31:03 ##########################################################
> >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env:
> >> el7:epel-7-x86_64
> >> > > 19:31:03 ## took 8 seconds
> >> > > 19:31:03 ## rc = 1
> >> > > 19:31:03 ##########################################################
> >> > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> >> > > 19:31:03 ##! Last 20 log enties:
> >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log
> >> > > 19:31:03 ##!
> >> > > 19:31:03 ========== Initializing chroot
> >> > > 19:31:03 mock \
> >> > > 19:31:03
> >> > >
> >> --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> >> > > \
> >> > > 19:31:03 --root="mocker-epel-7-x86_64.el7" \
> >> > > 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \
> >> > > 19:31:03 --init
> >> > > 19:31:03 Init took 8 seconds
> >> > > 19:31:03 ============================
> >> > > 19:31:03 ##!
> >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >> > > 19:31:03 ##!########################################################
> >> > > 19:31:03 Build step 'Execute shell' marked build as failure
> >> > > 19:31:03 Performing Post build task...
> >> > > 19:31:03 Match found for :.* : True
> >> > > 19:31:03 Logical operation result is TRUE
> >> > > 19:31:03 Running script : #!/bin/bash -xe
> >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh"
> >> > > 19:31:03
> >> > > 19:31:03 shopt -s nullglob
> >> > > 19:31:03
> >> > > 19:31:03
> >> > > 19:31:03 WORKSPACE="$PWD"
> >> > > 19:31:03
> >> > > 19:31:03 # Make clear this is the cleanup, helps reading the jenkins
> >> logs
> >> > > 19:31:03 cat <<EOC
> >> > > 19:31:03
> >> _______________________________________________________________________
> >> > > 19:31:03
> >> #######################################################################
> >> > > 19:31:03 #
> >> #
> >> > > 19:31:03 # CLEANUP
> >> #
> >> > > 19:31:03 #
> >> #
> >> > > 19:31:03
> >> #######################################################################
> >> > > 19:31:03 EOC
> >> > > 19:31:03
> >> > > 19:31:03
> >> > > 19:31:03 # Archive the logs, we want them anyway
> >> > > 19:31:03 logs=(
> >> > > 19:31:03 ./*log
> >> > > 19:31:03 ./*/logs
> >> > > 19:31:03 )
> >> > > 19:31:03 if [[ "$logs" ]]; then
> >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
> >> > > 19:31:03 rm -rf "${logs[@]}"
> >> > > 19:31:03 fi
> >> > > 19:31:03
> >> > > 19:31:03 # stop any processes running inside the chroot
> >> > > 19:31:03 failed=false
> >> > > 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*)
> >> > > 19:31:03 # Clean current jobs mockroot if any
> >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do
> >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue
> >> > > 19:31:03 echo "Cleaning up mock $mock_conf"
> >> > > 19:31:03 mock_root="${mock_conf_file##*/}"
> >> > > 19:31:03 mock_root="${mock_root%.*}"
> >> > > 19:31:03 my_mock="/usr/bin/mock"
> >> > > 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}"
> >> > > 19:31:03 my_mock+=" --root=${mock_root}"
> >> > > 19:31:03 my_mock+=" --resultdir=$WORKSPACE"
> >> > > 19:31:03
> >> > > 19:31:03 #TODO: investigate why mock --clean fails to umount
> >> > > certain dirs sometimes,
> >> > > 19:31:03 #so we can use it instead of manually doing all this.
> >> > > 19:31:03 echo "Killing all mock orphan processes, if any."
> >> > > 19:31:03 $my_mock \
> >> > > 19:31:03 --orphanskill \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot."
> >> > > 19:31:03 failed=true
> >> > > 19:31:03 }
> >> > > 19:31:03
> >> > > 19:31:03 mock_root="$(\
> >> > > 19:31:03 grep \
> >> > > 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \
> >> > > 19:31:03 "$mock_conf_file" \
> >> > > 19:31:03 )" || :
> >> > > 19:31:03 [[ "$mock_root" ]] || continue
> >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> >> || :
> >> > > 19:31:03 if [[ "$mounts" ]]; then
> >> > > 19:31:03 echo "Found mounted dirs inside the chroot $chroot.
> >> > > Trying to umount."
> >> > > 19:31:03 fi
> >> > > 19:31:03 for mount in "${mounts[@]}"; do
> >> > > 19:31:03 sudo umount "$mount" \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to umount $mount."
> >> > > 19:31:03 failed=true
> >> > > 19:31:03 }
> >> > > 19:31:03 done
> >> > > 19:31:03 done
> >> > > 19:31:03
> >> > > 19:31:03 # Clean any leftover chroot from other jobs
> >> > > 19:31:03 for mock_root in /var/lib/mock/*; do
> >> > > 19:31:03 this_chroot_failed=false
> >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> >> || :
> >> > > 19:31:03 if [[ "$mounts" ]]; then
> >> > > 19:31:03 echo "Found mounted dirs inside the chroot
> >> $mock_root." \
> >> > > 19:31:03 "Trying to umount."
> >> > > 19:31:03 fi
> >> > > 19:31:03 for mount in "${mounts[@]}"; do
> >> > > 19:31:03 sudo umount "$mount" \
> >> > > 19:31:03 || {
> >> > > 19:31:03 echo "ERROR: Failed to umount $mount."
> >> > > 19:31:03 failed=true
> >> > > 19:31:03 this_chroot_failed=true
> >> > > 19:31:03 }
> >> > > 19:31:03 done
> >> > > 19:31:03 if ! $this_chroot_failed; then
> >> > > 19:31:03 sudo rm -rf "$mock_root"
> >> > > 19:31:03 fi
> >> > > 19:31:03 done
> >> > > 19:31:03
> >> > > 19:31:03 if $failed; then
> >> > > 19:31:03 echo "Aborting."
> >> > > 19:31:03 exit 1
> >> > > 19:31:03 fi
> >> > > 19:31:03
> >> > > 19:31:03 # remove mock system cache, we will setup proxies to do the
> >> > > caching and this
> >> > > 19:31:03 # takes lots of space between runs
> >> > > 19:31:03 shopt -u nullglob
> >> > > 19:31:03 sudo rm -Rf /var/cache/mock/*
> >> > > 19:31:03
> >> > > 19:31:03 # restore the permissions in the working dir, as sometimes it
> >> > > leaves files
> >> > > 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins
> >> > > job fails to
> >> > > 19:31:03 # clean and breaks the jobs
> >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE"
> >> > > 19:31:03
> >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe
> >> > > /tmp/hudson3681642653853563663.sh
> >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh
> >> > > 19:31:03 shell-scripts/mock_cleanup.sh
> >> > > 19:31:03 + shopt -s nullglob
> >> > > 19:31:03 +
> >> WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > > 19:31:03 + cat
> >> > > 19:31:03
> >> _______________________________________________________________________
> >> > > 19:31:03
> >> #######################################################################
> >> > > 19:31:03 #
> >> #
> >> > > 19:31:03 # CLEANUP
> >> #
> >> > > 19:31:03 #
> >> #
> >> > > 19:31:03
> >> #######################################################################
> >> > > 19:31:03 + logs=(./*log ./*/logs)
> >> > > 19:31:03 + [[ -n ./vdsm/logs ]]
> >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs
> >> > > 19:31:03 ./vdsm/logs/
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log
> >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log
> >> > > 19:31:03 + rm -rf ./vdsm/logs
> >> > > 19:31:03 + failed=false
> >> > > 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*)
> >> > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"'
> >> > > 19:31:03 + [[ -n
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> >> > > ]]
> >> > > 19:31:03 + echo 'Cleaning up mock '
> >> > > 19:31:03 Cleaning up mock
> >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg
> >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7
> >> > > 19:31:03 + my_mock=/usr/bin/mock
> >> > > 19:31:03 + my_mock+='
> >> > >
> >> --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
> >> > > 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7'
> >> > > 19:31:03 + my_mock+='
> >> > >
> >> --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
> >> > > 19:31:03 + echo 'Killing all mock orphan processes, if any.'
> >> > > 19:31:03 Killing all mock orphan processes, if any.
> >> > > 19:31:03 + /usr/bin/mock
> >> > >
> >> --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
> >> > > --root=mocker-epel-7-x86_64.el7
> >> > > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > > --orphanskill
> >> > > 19:31:04 WARNING: Could not find required logging config file:
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> >> > > Using default...
> >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version =
> >> 3.4.3)...
> >> > > 19:31:04 Start: init plugins
> >> > > 19:31:04 INFO: selinux enabled
> >> > > 19:31:04 Finish: init plugins
> >> > > 19:31:04 Start: run
> >> > > 19:31:04 Finish: run
> >> > > 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
> >> > >
> >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> >> > > 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]]
> >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> >> > > 19:31:04 ++ mount
> >> > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 ++ awk '{print $3}'
> >> > > 19:31:04 + :
> >> > > 19:31:04 + [[ -n '' ]]
> >> > > 19:31:04 + for mock_root in '/var/lib/mock/*'
> >> > > 19:31:04 + this_chroot_failed=false
> >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
> >> > > 19:31:04 ++ mount
> >> > > 19:31:04 ++ grep
> >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 ++ awk '{print $3}'
> >> > > 19:31:04 + :
> >> > > 19:31:04 + [[ -n '' ]]
> >> > > 19:31:04 + false
> >> > > 19:31:04 + sudo rm -rf
> >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 + false
> >> > > 19:31:04 + shopt -u nullglob
> >> > > 19:31:04 + sudo rm -Rf
> >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc
> >> > > 19:31:04 + sudo chown -R jenkins
> >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> >> > > 19:31:04 POST BUILD TASK : SUCCESS
> >> > > 19:31:04 END OF POST BUILD TASK : 0
> >> > > 19:31:04 Archiving artifacts
> >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure
> >> > > 19:31:04 Started calculate disk usage of build
> >> > > 19:31:04 Finished Calculation of disk usage of build in 0 seconds
> >> > > 19:31:04 Started calculate disk usage of workspace
> >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds
> >> > > 19:31:04 Finished: FAILURE
> >> > _______________________________________________
> >> > Infra mailing list
> >> > Infra at ovirt.org
> >> > http://lists.ovirt.org/mailman/listinfo/infra
> >>
> >> --
> >> David Caro
> >>
> >> Red Hat S.L.
> >> Continuous Integration Engineer - EMEA ENG Virtualization R&D
> >>
> >> Tel.: +420 532 294 605
> >> Email: dcaro at redhat.com
> >> IRC: dcaro|dcaroest@{freenode|oftc|redhat}
> >> Web: www.redhat.com
> >> RHT Global #: 82-62605
> >>
> >> _______________________________________________
> >> Infra mailing list
> >> Infra at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/infra
> >>
> >>
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro at redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160517/61680e61/attachment.sig>
More information about the Infra
mailing list