Re: Another job failure: "Command failed. See logs for output"

On 05/17 12:27, Gil Shinar wrote: > Hi David, > > I think you've got me wrong. I was referring to what Eyal said about > increasing the history we keep. Not to the problem itself. Yep, I got you wrong :) > I still have no idea what the problem is. > I thought about attaching the logs of the Mock to the build so we'll have > them archived on the master. > I'll look at the mock runner script to see where it is writing this logs so > we'll be able to archive them. They are already being archived, see the previous email I sent > > Thanks > Gil > > > On Tue, May 17, 2016 at 12:20 PM, dcaro <dcaro(a)redhat.com&gt; wrote: > > > On 05/17 10:36, Gil Shinar wrote: > > > Hi, > > > > > > I have tried to debug this issue yesterday with the help of Barak. > > > I don't think that the problem is history as I saw another failure like > > > this yesterday (it happens from time to time). Problem is the slave it > > > builds on being taken by another build and cleans the workspace. I tried > > to > > > look for logs under the workspace and they missing probably because a > > newer > > > build is being built on the same slave. > > > > How did you find out about that? > > That has happened before when the same slave was added twice on jenkins > > with > > different names, but the behavior was way different (you'd get things like > > the > > dir does not exist and such, not errors when running yum inside mock) > > > > > > > > Gil > > > > > > On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > > > > > > > Can we increase the history for builds on that job? We should be able > > to > > > > debug jobs at least a week back. Artifacts are not needed > > > > On May 16, 2016 11:42 PM, "David Caro" <dcaro(a)redhat.com&gt; wrote: > > > > > > > >> On 05/15 23:23, Nir Soffer wrote: > > > >> > Another instance: > > > >> > > > > >> > 19:38:15 Start: yum install > > > >> > 19:38:22 ERROR: Command failed. See logs for output. > > > >> > > > >> That means that there was an issue with the yum repos, most common > > causes > > > >> are: > > > >> > > > >> * The repos were actually down (mirrors fail once a day usually, while > > > >> syncing > > > >> the rpms, though the cause is just a guess) > > > >> * Repoproxy (that we use to cache rpms) was overloaded > > > >> > > > >> Both jobs histories are now gone, in order to be able to debug those > > > >> issues > > > >> (for the next time), try looking into the mock logs, under the > > 'logs.tgz' > > > >> file > > > >> that's archived in the job, then under the path: > > > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > > > >> or > > > >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/ > > > >> > > > >> Usually in the log named 'root.log' (check which one has more size). > > > >> > > > >> > > > > >> > > > > >> > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console > > > >> > > > >> > > > > >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer < nsoffer(a)redhat.com&gt; > > > >> wrote: > > > >> > > > > > >> > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console > > > >> > > > > > >> > > 19:30:56 Start: yum install > > > >> > > 19:31:03 ERROR: Command failed. See logs for output. > > > >> > > > > > >> > > Including the to make sure it will not disapper > > > >> > > ---- > > > >> > > > > > >> > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 > > > >> > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 > > nested) > > > >> in > > > >> > > workspace > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > > >> > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 > > > >> > > 19:30:49 Fetching changes from the remote Git repository > > > >> > > 19:30:49 > git config remote.origin.url > > > >> > > git://gerrit.ovirt.org/vdsm.git # timeout=10 > > > >> > > 19:30:49 Cleaning workspace > > > >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=10 > > > >> > > 19:30:49 Resetting working tree > > > >> > > 19:30:49 > git reset --hard # timeout=10 > > > >> > > 19:30:49 > git clean -fdx # timeout=10 > > > >> > > 19:30:49 Pruning obsolete local branches > > > >> > > 19:30:49 Fetching upstream changes from git:// > > > >> gerrit.ovirt.org/vdsm.git > > > >> > > 19:30:49 > git --version # timeout=10 > > > >> > > 19:30:49 > git -c core.askpass=true fetch --tags --progress > > > >> > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune > > > >> > > 19:30:52 > git rev-parse > > > >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 > > > >> > > 19:30:52 Checking out Revision > > > >> 0940208483f3a21261eb5d725348e65c23becdc0 (master) > > > >> > > 19:30:52 > git config core.sparsecheckout # timeout=10 > > > >> > > 19:30:52 > git checkout -f > > 0940208483f3a21261eb5d725348e65c23becdc0 > > > >> > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 > > > >> > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d > > # > > > >> timeout=10 > > > >> > > 19:30:52 > git branch -a # timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # > > timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse > > remotes/origin/ovirt-3.5-gluster^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # > > > >> timeout=10 > > > >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # > > > >> timeout=10 > > > >> > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 > > > >> > > 19:30:53 Fetching changes from the remote Git repository > > > >> > > 19:30:53 > git config remote.origin.url > > > >> > > git://gerrit.ovirt.org/jenkins.git # timeout=10 > > > >> > > 19:30:53 Cleaning workspace > > > >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=10 > > > >> > > 19:30:53 Resetting working tree > > > >> > > 19:30:53 > git reset --hard # timeout=10 > > > >> > > 19:30:53 > git clean -fdx # timeout=10 > > > >> > > 19:30:53 Pruning obsolete local branches > > > >> > > 19:30:53 Fetching upstream changes from git:// > > > >> gerrit.ovirt.org/jenkins.git > > > >> > > 19:30:53 > git --version # timeout=10 > > > >> > > 19:30:53 > git -c core.askpass=true fetch --tags --progress > > > >> > > git://gerrit.ovirt.org/jenkins.git > > > >> +refs/heads/*:refs/remotes/origin/* > > > >> > > --prune > > > >> > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 > > > >> > > 19:30:53 Checking out Revision > > > >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) > > > >> > > 19:30:53 > git config core.sparsecheckout # timeout=10 > > > >> > > 19:30:53 > git checkout -f > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > > > >> > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > > # > > > >> timeout=10 > > > >> > > 19:30:53 > git branch -a # timeout=10 > > > >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # > > timeout=10 > > > >> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > > >> > > /tmp/hudson8265618520208925180.sh > > > >> > > 19:30:53 shell-scripts/cleanup_slave.sh > > > >> > > 19:30:53 > > > >> ################################################################### > > > >> > > 19:30:53 # Cleaning up slave > > > >> # > > > >> > > 19:30:53 > > > >> ################################################################### > > > >> > > 19:30:53 Filesystem Size Used > > Avail > > > >> > > Use% Mounted on > > > >> > > 19:30:53 devtmpfs 3.9G 0 > > > >> 3.9G 0% /dev > > > >> > > 19:30:53 tmpfs 3.9G 0 > > 3.9G > > > >> > > 0% /dev/shm > > > >> > > 19:30:53 tmpfs 3.9G 2.1M > > > >> 3.9G 1% /run > > > >> > > 19:30:53 tmpfs 3.9G 0 > > 3.9G > > > >> > > 0% /sys/fs/cgroup > > > >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G > > > >> 27G 20% / > > > >> > > 19:30:53 tmpfs 3.9G 244K > > > >> 3.9G 1% /tmp > > > >> > > 19:30:53 /dev/vda1 988M 94M > > > >> 828M 11% /boot > > > >> > > 19:30:53 tmpfs 799M 0 > > 799M > > > >> > > 0% /run/user/1000 > > > >> > > 19:30:53 > > > >> --------------------------------------------------------------- > > > >> > > 19:30:53 Cleaning up postgres databases > > > >> > > 19:30:53 Postgres installation not found, skipping > > > >> > > 19:30:53 Cleaning up journal logs (if any) > > > >> > > 19:30:53 Redirecting to /bin/systemctl restart > > > >> systemd-journald.service > > > >> > > 19:30:54 Cleaning up /var/tmp > > > >> > > 19:30:54 done > > > >> > > 19:30:54 Emptying some common logs > > > >> > > 19:30:54 /var/log/wtmp > > > >> > > 19:30:54 Done > > > >> > > 19:30:54 Making sure there are no device mappings... > > > >> > > 19:30:54 Removing the used loop devices... > > > >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service > > > >> > > 19:30:54 > > > >> --------------------------------------------------------------- > > > >> > > 19:30:54 Filesystem Size Used > > Avail > > > >> > > Use% Mounted on > > > >> > > 19:30:54 devtmpfs 3.9G 0 > > > >> 3.9G 0% /dev > > > >> > > 19:30:54 tmpfs 3.9G 0 > > 3.9G > > > >> > > 0% /dev/shm > > > >> > > 19:30:54 tmpfs 3.9G 2.1M > > > >> 3.9G 1% /run > > > >> > > 19:30:54 tmpfs 3.9G 0 > > 3.9G > > > >> > > 0% /sys/fs/cgroup > > > >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G > > > >> 27G 20% / > > > >> > > 19:30:54 tmpfs 3.9G 244K > > > >> 3.9G 1% /tmp > > > >> > > 19:30:54 /dev/vda1 988M 94M > > > >> 828M 11% /boot > > > >> > > 19:30:54 tmpfs 799M 0 > > 799M > > > >> > > 0% /run/user/1000 > > > >> > > 19:30:54 > > > >> ################################################################### > > > >> > > 19:30:54 # Slave cleanup done > > > >> # > > > >> > > 19:30:54 > > > >> ################################################################### > > > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > > >> > > /tmp/hudson7354644880288923842.sh > > > >> > > 19:30:54 + echo shell-scripts/global_setup.sh > > > >> > > 19:30:54 shell-scripts/global_setup.sh > > > >> > > 19:30:54 + rm -rf > > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > > >> > > 19:30:54 + mkdir -p > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > > >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > > >> > > /tmp/hudson4023353013258685526.sh > > > >> > > 19:30:54 + echo shell-scripts/mock_setup.sh > > > >> > > 19:30:54 shell-scripts/mock_setup.sh > > > >> > > 19:30:54 + shopt -s nullglob > > > >> > > 19:30:54 + [[ '' == \t\r\u\e ]] > > > >> > > 19:30:54 + failed=false > > > >> > > 19:30:54 + > > > >> > > mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock > > > >> > > 19:30:54 + chroots=("$WORKSPACE"/mock/*) > > > >> > > 19:30:54 + false > > > >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts > > > >> > > 19:30:54 + mkdir -p mock exported-artifacts > > > >> > > 19:30:54 + chgrp mock mock > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts > > > >> > > 19:30:54 + chmod g+rws mock > > > >> > > 19:30:54 + sudo touch > > > >> > > 19:30:54 + : > > > >> > > 19:30:54 + sudo yum clean all > > > >> > > 19:30:54 Yum command has been deprecated, redirecting to > > '/usr/bin/dnf > > > >> > > clean all'. > > > >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. > > > >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run: > > > >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 > > > >> migrate' > > > >> > > 19:30:54 > > > >> > > 19:30:55 Cleaning repos: updates ci-tools fedora > > > >> > > 19:30:55 Cleaning up Everything > > > >> > > 19:30:55 + exit 0 > > > >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > > >> > > /tmp/hudson4405890361943185034.sh > > > >> > > 19:30:55 shell-scripts/standard_check_patch.sh > > > >> > > 19:30:55 > > ########################################################## > > > >> > > 19:30:55 > > ########################################################## > > > >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: > > > >> el7:epel-7-x86_64 > > > >> > > 19:30:55 > > ########################################################## > > > >> > > 19:30:55 > > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > > >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for > > script: > > > >> > > automation/check-patch.sh > > > >> > > 19:30:55 > > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > > >> > > 19:30:55 Using base mock conf > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg > > > >> > > 19:30:55 Using proxified config > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg > > > >> > > 19:30:55 Generating temporary mock conf > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7 > > > >> > > 19:30:55 Skipping mount points > > > >> > > 19:30:55 Using temp chroot = > > > >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:30:55 Adding repo ovirt-snapshot -> > > > >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 > > > >> > > 19:30:55 Adding repo ovirt-snapshot-static -> > > > >> > > > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 > > > >> > > 19:30:55 Adding repo centos-glusterfs -> > > > >> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ > > > >> > > 19:30:55 Adding repo centos-ovirt40-candidate -> > > > >> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ > > > >> > > 19:30:55 Adding repo lago -> > > > >> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 > > > >> > > 19:30:55 Adding repo epel -> > > > >> http://download.fedoraproject.org/pub/epel/7/x86_64 > > > >> > > 19:30:55 ========== Initializing chroot > > > >> > > 19:30:55 mock \ > > > >> > > 19:30:55 > > > >> > > > > > >> > > --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" > > > >> > > \ > > > >> > > 19:30:55 --root="mocker-epel-7-x86_64.el7" \ > > > >> > > 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" > > \ > > > >> > > 19:30:55 --init > > > >> > > 19:30:55 WARNING: Could not find required logging config file: > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. > > > >> > > Using default... > > > >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version = > > > >> 3.4.3)... > > > >> > > 19:30:55 Start: init plugins > > > >> > > 19:30:55 INFO: selinux enabled > > > >> > > 19:30:55 Finish: init plugins > > > >> > > 19:30:55 Start: run > > > >> > > 19:30:55 Start: clean chroot > > > >> > > 19:30:55 Finish: clean chroot > > > >> > > 19:30:55 Start: chroot init > > > >> > > 19:30:55 INFO: calling preinit hooks > > > >> > > 19:30:55 INFO: enabled root cache > > > >> > > 19:30:55 INFO: enabled yum cache > > > >> > > 19:30:55 Start: cleaning yum metadata > > > >> > > 19:30:55 Finish: cleaning yum metadata > > > >> > > 19:30:55 INFO: enabled ccache > > > >> > > 19:30:55 Mock Version: 1.2.14 > > > >> > > 19:30:55 INFO: Mock Version: 1.2.14 > > > >> > > 19:30:56 Start: yum install > > > >> > > 19:31:03 ERROR: Command failed. See logs for output. > > > >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot > > > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ > > > >> > > --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts > > > >> > > 19:31:03 Init took 8 seconds > > > >> > > 19:31:03 ============================ > > > >> > > 19:31:03 > > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > > >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh > > > >> > > chroot finished > > > >> > > 19:31:03 @@ took 8 seconds > > > >> > > 19:31:03 @@ rc = 1 > > > >> > > 19:31:03 > > > >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > > >> > > 19:31:03 > > ########################################################## > > > >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: > > > >> el7:epel-7-x86_64 > > > >> > > 19:31:03 ## took 8 seconds > > > >> > > 19:31:03 ## rc = 1 > > > >> > > 19:31:03 > > ########################################################## > > > >> > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > > > >> > > 19:31:03 ##! Last 20 log enties: > > > >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > > >> > > 19:31:03 ##! > > > >> > > 19:31:03 ========== Initializing chroot > > > >> > > 19:31:03 mock \ > > > >> > > 19:31:03 > > > >> > > > > > >> > > --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" > > > >> > > \ > > > >> > > 19:31:03 --root="mocker-epel-7-x86_64.el7" \ > > > >> > > 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" > > \ > > > >> > > 19:31:03 --init > > > >> > > 19:31:03 Init took 8 seconds > > > >> > > 19:31:03 ============================ > > > >> > > 19:31:03 ##! > > > >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > >> > > 19:31:03 > > ##!######################################################## > > > >> > > 19:31:03 Build step 'Execute shell' marked build as failure > > > >> > > 19:31:03 Performing Post build task... > > > >> > > 19:31:03 Match found for :.* : True > > > >> > > 19:31:03 Logical operation result is TRUE > > > >> > > 19:31:03 Running script : #!/bin/bash -xe > > > >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh" > > > >> > > 19:31:03 > > > >> > > 19:31:03 shopt -s nullglob > > > >> > > 19:31:03 > > > >> > > 19:31:03 > > > >> > > 19:31:03 WORKSPACE="$PWD" > > > >> > > 19:31:03 > > > >> > > 19:31:03 # Make clear this is the cleanup, helps reading the > > jenkins > > > >> logs > > > >> > > 19:31:03 cat <<EOC > > > >> > > 19:31:03 > > > >> > > _______________________________________________________________________ > > > >> > > 19:31:03 > > > >> > > ####################################################################### > > > >> > > 19:31:03 # > > > >> # > > > >> > > 19:31:03 # CLEANUP > > > >> # > > > >> > > 19:31:03 # > > > >> # > > > >> > > 19:31:03 > > > >> > > ####################################################################### > > > >> > > 19:31:03 EOC > > > >> > > 19:31:03 > > > >> > > 19:31:03 > > > >> > > 19:31:03 # Archive the logs, we want them anyway > > > >> > > 19:31:03 logs=( > > > >> > > 19:31:03 ./*log > > > >> > > 19:31:03 ./*/logs > > > >> > > 19:31:03 ) > > > >> > > 19:31:03 if [[ "$logs" ]]; then > > > >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > > > >> > > 19:31:03 rm -rf "${logs[@]}" > > > >> > > 19:31:03 fi > > > >> > > 19:31:03 > > > >> > > 19:31:03 # stop any processes running inside the chroot > > > >> > > 19:31:03 failed=false > > > >> > > 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) > > > >> > > 19:31:03 # Clean current jobs mockroot if any > > > >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do > > > >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue > > > >> > > 19:31:03 echo "Cleaning up mock $mock_conf" > > > >> > > 19:31:03 mock_root="${mock_conf_file##*/}" > > > >> > > 19:31:03 mock_root="${mock_root%.*}" > > > >> > > 19:31:03 my_mock="/usr/bin/mock" > > > >> > > 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" > > > >> > > 19:31:03 my_mock+=" --root=${mock_root}" > > > >> > > 19:31:03 my_mock+=" --resultdir=$WORKSPACE" > > > >> > > 19:31:03 > > > >> > > 19:31:03 #TODO: investigate why mock --clean fails to umount > > > >> > > certain dirs sometimes, > > > >> > > 19:31:03 #so we can use it instead of manually doing all this. > > > >> > > 19:31:03 echo "Killing all mock orphan processes, if any." > > > >> > > 19:31:03 $my_mock \ > > > >> > > 19:31:03 --orphanskill \ > > > >> > > 19:31:03 || { > > > >> > > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." > > > >> > > 19:31:03 failed=true > > > >> > > 19:31:03 } > > > >> > > 19:31:03 > > > >> > > 19:31:03 mock_root="$(\ > > > >> > > 19:31:03 grep \ > > > >> > > 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ > > > >> > > 19:31:03 "$mock_conf_file" \ > > > >> > > 19:31:03 )" || : > > > >> > > 19:31:03 [[ "$mock_root" ]] || continue > > > >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep > > "$mock_root")) > > > >> || : > > > >> > > 19:31:03 if [[ "$mounts" ]]; then > > > >> > > 19:31:03 echo "Found mounted dirs inside the chroot > > $chroot. > > > >> > > Trying to umount." > > > >> > > 19:31:03 fi > > > >> > > 19:31:03 for mount in "${mounts[@]}"; do > > > >> > > 19:31:03 sudo umount "$mount" \ > > > >> > > 19:31:03 || { > > > >> > > 19:31:03 echo "ERROR: Failed to umount $mount." > > > >> > > 19:31:03 failed=true > > > >> > > 19:31:03 } > > > >> > > 19:31:03 done > > > >> > > 19:31:03 done > > > >> > > 19:31:03 > > > >> > > 19:31:03 # Clean any leftover chroot from other jobs > > > >> > > 19:31:03 for mock_root in /var/lib/mock/*; do > > > >> > > 19:31:03 this_chroot_failed=false > > > >> > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep > > "$mock_root")) > > > >> || : > > > >> > > 19:31:03 if [[ "$mounts" ]]; then > > > >> > > 19:31:03 echo "Found mounted dirs inside the chroot > > > >> $mock_root." \ > > > >> > > 19:31:03 "Trying to umount." > > > >> > > 19:31:03 fi > > > >> > > 19:31:03 for mount in "${mounts[@]}"; do > > > >> > > 19:31:03 sudo umount "$mount" \ > > > >> > > 19:31:03 || { > > > >> > > 19:31:03 echo "ERROR: Failed to umount $mount." > > > >> > > 19:31:03 failed=true > > > >> > > 19:31:03 this_chroot_failed=true > > > >> > > 19:31:03 } > > > >> > > 19:31:03 done > > > >> > > 19:31:03 if ! $this_chroot_failed; then > > > >> > > 19:31:03 sudo rm -rf "$mock_root" > > > >> > > 19:31:03 fi > > > >> > > 19:31:03 done > > > >> > > 19:31:03 > > > >> > > 19:31:03 if $failed; then > > > >> > > 19:31:03 echo "Aborting." > > > >> > > 19:31:03 exit 1 > > > >> > > 19:31:03 fi > > > >> > > 19:31:03 > > > >> > > 19:31:03 # remove mock system cache, we will setup proxies to do > > the > > > >> > > caching and this > > > >> > > 19:31:03 # takes lots of space between runs > > > >> > > 19:31:03 shopt -u nullglob > > > >> > > 19:31:03 sudo rm -Rf /var/cache/mock/* > > > >> > > 19:31:03 > > > >> > > 19:31:03 # restore the permissions in the working dir, as > > sometimes it > > > >> > > leaves files > > > >> > > 19:31:03 # owned by root and then the 'cleanup workspace' from > > jenkins > > > >> > > job fails to > > > >> > > 19:31:03 # clean and breaks the jobs > > > >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" > > > >> > > 19:31:03 > > > >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > > >> > > /tmp/hudson3681642653853563663.sh > > > >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh > > > >> > > 19:31:03 shell-scripts/mock_cleanup.sh > > > >> > > 19:31:03 + shopt -s nullglob > > > >> > > 19:31:03 + > > > >> WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > > >> > > 19:31:03 + cat > > > >> > > 19:31:03 > > > >> > > _______________________________________________________________________ > > > >> > > 19:31:03 > > > >> > > ####################################################################### > > > >> > > 19:31:03 # > > > >> # > > > >> > > 19:31:03 # CLEANUP > > > >> # > > > >> > > 19:31:03 # > > > >> # > > > >> > > 19:31:03 > > > >> > > ####################################################################### > > > >> > > 19:31:03 + logs=(./*log ./*/logs) > > > >> > > 19:31:03 + [[ -n ./vdsm/logs ]] > > > >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs > > > >> > > 19:31:03 ./vdsm/logs/ > > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log > > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log > > > >> > > 19:31:03 > > ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > > >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log > > > >> > > 19:31:03 + rm -rf ./vdsm/logs > > > >> > > 19:31:03 + failed=false > > > >> > > 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) > > > >> > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' > > > >> > > 19:31:03 + [[ -n > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg > > > >> > > ]] > > > >> > > 19:31:03 + echo 'Cleaning up mock ' > > > >> > > 19:31:03 Cleaning up mock > > > >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg > > > >> > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 > > > >> > > 19:31:03 + my_mock=/usr/bin/mock > > > >> > > 19:31:03 + my_mock+=' > > > >> > > > > > >> > > --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm' > > > >> > > 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' > > > >> > > 19:31:03 + my_mock+=' > > > >> > > > > > >> > > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64' > > > >> > > 19:31:03 + echo 'Killing all mock orphan processes, if any.' > > > >> > > 19:31:03 Killing all mock orphan processes, if any. > > > >> > > 19:31:03 + /usr/bin/mock > > > >> > > > > > >> > > --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm > > > >> > > --root=mocker-epel-7-x86_64.el7 > > > >> > > > > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > > >> > > --orphanskill > > > >> > > 19:31:04 WARNING: Could not find required logging config file: > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. > > > >> > > Using default... > > > >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version = > > > >> 3.4.3)... > > > >> > > 19:31:04 Start: init plugins > > > >> > > 19:31:04 INFO: selinux enabled > > > >> > > 19:31:04 Finish: init plugins > > > >> > > 19:31:04 Start: run > > > >> > > 19:31:04 Finish: run > > > >> > > 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = > > '\'')[^'\'']*' > > > >> > > > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg > > > >> > > 19:31:04 + > > mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] > > > >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep > > "$mock_root")) > > > >> > > 19:31:04 ++ mount > > > >> > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:31:04 ++ awk '{print $3}' > > > >> > > 19:31:04 + : > > > >> > > 19:31:04 + [[ -n '' ]] > > > >> > > 19:31:04 + for mock_root in '/var/lib/mock/*' > > > >> > > 19:31:04 + this_chroot_failed=false > > > >> > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep > > "$mock_root")) > > > >> > > 19:31:04 ++ mount > > > >> > > 19:31:04 ++ grep > > > >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:31:04 ++ awk '{print $3}' > > > >> > > 19:31:04 + : > > > >> > > 19:31:04 + [[ -n '' ]] > > > >> > > 19:31:04 + false > > > >> > > 19:31:04 + sudo rm -rf > > > >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:31:04 + false > > > >> > > 19:31:04 + shopt -u nullglob > > > >> > > 19:31:04 + sudo rm -Rf > > > >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > > >> > > 19:31:04 + sudo chown -R jenkins > > > >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > > >> > > 19:31:04 POST BUILD TASK : SUCCESS > > > >> > > 19:31:04 END OF POST BUILD TASK : 0 > > > >> > > 19:31:04 Archiving artifacts > > > >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure > > > >> > > 19:31:04 Started calculate disk usage of build > > > >> > > 19:31:04 Finished Calculation of disk usage of build in 0 seconds > > > >> > > 19:31:04 Started calculate disk usage of workspace > > > >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 > > seconds > > > >> > > 19:31:04 Finished: FAILURE > > > >> > _______________________________________________ > > > >> > Infra mailing list > > > >> > Infra(a)ovirt.org > > > >> > http://lists.ovirt.org/mailman/listinfo/infra > > > >> > > > >> -- > > > >> David Caro > > > >> > > > >> Red Hat S.L. > > > >> Continuous Integration Engineer - EMEA ENG Virtualization R&D > > > >> > > > >> Tel.: +420 532 294 605 > > > >> Email: dcaro(a)redhat.com > > > >> IRC: dcaro|dcaroest@{freenode|oftc|redhat} > > > >> Web: www.redhat.com > > > >> RHT Global #: 82-62605 > > > >> > > > >> _______________________________________________ > > > >> Infra mailing list > > > >> Infra(a)ovirt.org > > > >> http://lists.ovirt.org/mailman/listinfo/infra > > > >> > > > >> > > > > _______________________________________________ > > > > Infra mailing list > > > > Infra(a)ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > > > > -- > > David Caro > > > > Red Hat S.L. > > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > > > Tel.: +420 532 294 605 > > Email: dcaro(a)redhat.com > > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > > Web: www.redhat.com > > RHT Global #: 82-62605 > > -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605