Re: Another job failure: "Command failed. See logs for output"

Hi, =20 I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I tried =

look for logs under the workspace and they missing probably because a new=

build is being built on the same slave.

=20 Gil =20 On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: =20 > Can we increase the history for builds on that job? We should be able =

> debug jobs at least a week back. Artifacts are not needed > On May 16, 2016 11:42 PM, "David Caro" <dcaro(a)redhat.com&gt; wrote: > >> On 05/15 23:23, Nir Soffer wrote: >> > Another instance: >> > >> > 19:38:15 Start: yum install >> > 19:38:22 ERROR: Command failed. See logs for output. >> >> That means that there was an issue with the yum repos, most common cau=

>> are: >> >> * The repos were actually down (mirrors fail once a day usually, while >> syncing >> the rpms, though the cause is just a guess) >> * Repoproxy (that we use to cache rpms) was overloaded >> >> Both jobs histories are now gone, in order to be able to debug those >> issues >> (for the next time), try looking into the mock logs, under the 'logs.t=

>> file >> that's archived in the job, then under the path: >> /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ >> or >> /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/ >> >> Usually in the log named 'root.log' (check which one has more size). >> >> > >> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/c=

>> >> > >> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >> wrote: >> > > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/c=

>> > > >> > > 19:30:56 Start: yum install >> > > 19:31:03 ERROR: Command failed. See logs for output. >> > > >> > > Including the to make sure it will not disapper >> > > ---- >> > > >> > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 >> > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) >> in >> > > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

>> > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 >> > > 19:30:49 Fetching changes from the remote Git repository >> > > 19:30:49 > git config remote.origin.url >> > > git://gerrit.ovirt.org/vdsm.git # timeout=3D10 >> > > 19:30:49 Cleaning workspace >> > > 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 >> > > 19:30:49 Resetting working tree >> > > 19:30:49 > git reset --hard # timeout=3D10 >> > > 19:30:49 > git clean -fdx # timeout=3D10 >> > > 19:30:49 Pruning obsolete local branches >> > > 19:30:49 Fetching upstream changes from git:// >> gerrit.ovirt.org/vdsm.git >> > > 19:30:49 > git --version # timeout=3D10 >> > > 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress >> > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune >> > > 19:30:52 > git rev-parse >> > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10 >> > > 19:30:52 Checking out Revision >> 0940208483f3a21261eb5d725348e65c23becdc0 (master) >> > > 19:30:52 > git config core.sparsecheckout # timeout=3D10 >> > > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becd=

>> > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 >> > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # >> timeout=3D10 >> > > 19:30:52 > git branch -a # timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=

>> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit=

>> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # >> timeout=3D10 >> > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # >> timeout=3D10 >> > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 >> > > 19:30:53 Fetching changes from the remote Git repository >> > > 19:30:53 > git config remote.origin.url >> > > git://gerrit.ovirt.org/jenkins.git # timeout=3D10 >> > > 19:30:53 Cleaning workspace >> > > 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 >> > > 19:30:53 Resetting working tree >> > > 19:30:53 > git reset --hard # timeout=3D10 >> > > 19:30:53 > git clean -fdx # timeout=3D10 >> > > 19:30:53 Pruning obsolete local branches >> > > 19:30:53 Fetching upstream changes from git:// >> gerrit.ovirt.org/jenkins.git >> > > 19:30:53 > git --version # timeout=3D10 >> > > 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress >> > > git://gerrit.ovirt.org/jenkins.git >> +refs/heads/*:refs/remotes/origin/* >> > > --prune >> > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 >> > > 19:30:53 Checking out Revision >> > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) >> > > 19:30:53 > git config core.sparsecheckout # timeout=3D10 >> > > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22dda=

>> > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # >> timeout=3D10 >> > > 19:30:53 > git branch -a # timeout=3D10 >> > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=

>> > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e >> > > /tmp/hudson8265618520208925180.sh >> > > 19:30:53 shell-scripts/cleanup_slave.sh >> > > 19:30:53 >> ################################################################### >> > > 19:30:53 # Cleaning up slave >> # >> > > 19:30:53 >> ################################################################### >> > > 19:30:53 Filesystem Size Used Av=

>> > > Use% Mounted on >> > > 19:30:53 devtmpfs 3.9G 0 >> 3.9G 0% /dev >> > > 19:30:53 tmpfs 3.9G 0 3=

>> > > 0% /dev/shm >> > > 19:30:53 tmpfs 3.9G 2.1M >> 3.9G 1% /run >> > > 19:30:53 tmpfs 3.9G 0 3=

>> > > 0% /sys/fs/cgroup >> > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G >> 27G 20% / >> > > 19:30:53 tmpfs 3.9G 244K >> 3.9G 1% /tmp >> > > 19:30:53 /dev/vda1 988M 94M >> 828M 11% /boot >> > > 19:30:53 tmpfs 799M 0 7=

>> > > 0% /run/user/1000 >> > > 19:30:53 >> --------------------------------------------------------------- >> > > 19:30:53 Cleaning up postgres databases >> > > 19:30:53 Postgres installation not found, skipping >> > > 19:30:53 Cleaning up journal logs (if any) >> > > 19:30:53 Redirecting to /bin/systemctl restart >> systemd-journald.service >> > > 19:30:54 Cleaning up /var/tmp >> > > 19:30:54 done >> > > 19:30:54 Emptying some common logs >> > > 19:30:54 /var/log/wtmp >> > > 19:30:54 Done >> > > 19:30:54 Making sure there are no device mappings... >> > > 19:30:54 Removing the used loop devices... >> > > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service >> > > 19:30:54 >> --------------------------------------------------------------- >> > > 19:30:54 Filesystem Size Used Av=

>> > > Use% Mounted on >> > > 19:30:54 devtmpfs 3.9G 0 >> 3.9G 0% /dev >> > > 19:30:54 tmpfs 3.9G 0 3=

>> > > 0% /dev/shm >> > > 19:30:54 tmpfs 3.9G 2.1M >> 3.9G 1% /run >> > > 19:30:54 tmpfs 3.9G 0 3=

>> > > 0% /sys/fs/cgroup >> > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G >> 27G 20% / >> > > 19:30:54 tmpfs 3.9G 244K >> 3.9G 1% /tmp >> > > 19:30:54 /dev/vda1 988M 94M >> 828M 11% /boot >> > > 19:30:54 tmpfs 799M 0 7=

>> > > 0% /run/user/1000 >> > > 19:30:54 >> ################################################################### >> > > 19:30:54 # Slave cleanup done >> # >> > > 19:30:54 >> ################################################################### >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe >> > > /tmp/hudson7354644880288923842.sh >> > > 19:30:54 + echo shell-scripts/global_setup.sh >> > > 19:30:54 shell-scripts/global_setup.sh >> > > 19:30:54 + rm -rf >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp >> > > 19:30:54 + mkdir -p >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp >> > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe >> > > /tmp/hudson4023353013258685526.sh >> > > 19:30:54 + echo shell-scripts/mock_setup.sh >> > > 19:30:54 shell-scripts/mock_setup.sh >> > > 19:30:54 + shopt -s nullglob >> > > 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] >> > > 19:30:54 + failed=3Dfalse >> > > 19:30:54 + >> mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/=

>> > > 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) >> > > 19:30:54 + false >> > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts >> > > 19:30:54 + mkdir -p mock exported-artifacts >> > > 19:30:54 + chgrp mock mock >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-ar=

>> > > 19:30:54 + chmod g+rws mock >> > > 19:30:54 + sudo touch >> > > 19:30:54 + : >> > > 19:30:54 + sudo yum clean all >> > > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin=

>> > > clean all'. >> > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. >> > > 19:30:54 To transfer transaction metadata from yum to DNF, run: >> > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 >> migrate' >> > > 19:30:54 >> > > 19:30:55 Cleaning repos: updates ci-tools fedora >> > > 19:30:55 Cleaning up Everything >> > > 19:30:55 + exit 0 >> > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e >> > > /tmp/hudson4405890361943185034.sh >> > > 19:30:55 shell-scripts/standard_check_patch.sh >> > > 19:30:55 ########################################################## >> > > 19:30:55 ########################################################## >> > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: >> el7:epel-7-x86_64 >> > > 19:30:55 ########################################################## >> > > 19:30:55 >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: >> > > automation/check-patch.sh >> > > 19:30:55 >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> > > 19:30:55 Using base mock conf >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/moc=

>> > > 19:30:55 Using proxified config >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/moc=

>> > > 19:30:55 Generating temporary mock conf >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker=

>> > > 19:30:55 Skipping mount points >> > > 19:30:55 Using temp chroot =3D >> epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc >> > > 19:30:55 Adding repo ovirt-snapshot -> >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 >> > > 19:30:55 Adding repo ovirt-snapshot-static -> >> > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 >> > > 19:30:55 Adding repo centos-glusterfs -> >> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ >> > > 19:30:55 Adding repo centos-ovirt40-candidate -> >> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ >> > > 19:30:55 Adding repo lago -> >> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 >> > > 19:30:55 Adding repo epel -> >> http://download.fedoraproject.org/pub/epel/7/x86_64 >> > > 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot >> > > 19:30:55 mock \ >> > > 19:30:55 >> > > >> --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86=

>> > > \ >> > > 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ >> > > 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init=

>> > > 19:30:55 --init >> > > 19:30:55 WARNING: Could not find required logging config file: >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/loggin=

>> > > Using default... >> > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version =3D >> 3.4.3)... >> > > 19:30:55 Start: init plugins >> > > 19:30:55 INFO: selinux enabled >> > > 19:30:55 Finish: init plugins >> > > 19:30:55 Start: run >> > > 19:30:55 Start: clean chroot >> > > 19:30:55 Finish: clean chroot >> > > 19:30:55 Start: chroot init >> > > 19:30:55 INFO: calling preinit hooks >> > > 19:30:55 INFO: enabled root cache >> > > 19:30:55 INFO: enabled yum cache >> > > 19:30:55 Start: cleaning yum metadata >> > > 19:30:55 Finish: cleaning yum metadata >> > > 19:30:55 INFO: enabled ccache >> > > 19:30:55 Mock Version: 1.2.14 >> > > 19:30:55 INFO: Mock Version: 1.2.14 >> > > 19:30:56 Start: yum install >> > > 19:31:03 ERROR: Command failed. See logs for output. >> > > 19:31:03 # /usr/bin/yum-deprecated --installroot >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ >> > > --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dnocont=

>> > > 19:31:03 Init took 8 seconds >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

>> > > 19:31:03 >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh >> > > chroot finished >> > > 19:31:03 @@ took 8 seconds >> > > 19:31:03 @@ rc =3D 1 >> > > 19:31:03 >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> > > 19:31:03 ########################################################## >> > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: >> el7:epel-7-x86_64 >> > > 19:31:03 ## took 8 seconds >> > > 19:31:03 ## rc =3D 1 >> > > 19:31:03 ########################################################## >> > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv >> > > 19:31:03 ##! Last 20 log enties: >> > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log >> > > 19:31:03 ##! >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot >> > > 19:31:03 mock \ >> > > 19:31:03 >> > > >> --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86=

>> > > \ >> > > 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ >> > > 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init=

>> > > 19:31:03 Init took 8 seconds >> > > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

>> > > 19:31:03 ##! >> > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> > > 19:31:03 ##!######################################################=

>> > > 19:31:03 Build step 'Execute shell' marked build as failure >> > > 19:31:03 Performing Post build task... >> > > 19:31:03 Match found for :.* : True >> > > 19:31:03 Logical operation result is TRUE >> > > 19:31:03 Running script : #!/bin/bash -xe >> > > 19:31:03 echo "shell-scripts/mock_cleanup.sh" >> > > 19:31:03 >> > > 19:31:03 shopt -s nullglob >> > > 19:31:03 >> > > 19:31:03 >> > > 19:31:03 WORKSPACE=3D"$PWD" >> > > 19:31:03 >> > > 19:31:03 # Make clear this is the cleanup, helps reading the jenki=

>> logs >> > > 19:31:03 cat <<EOC >> > > 19:31:03 >> _______________________________________________________________________ >> > > 19:31:03 >> ####################################################################### >> > > 19:31:03 # >> # >> > > 19:31:03 # CLEANUP >> # >> > > 19:31:03 # >> # >> > > 19:31:03 >> ####################################################################### >> > > 19:31:03 EOC >> > > 19:31:03 >> > > 19:31:03 >> > > 19:31:03 # Archive the logs, we want them anyway >> > > 19:31:03 logs=3D( >> > > 19:31:03 ./*log >> > > 19:31:03 ./*/logs >> > > 19:31:03 ) >> > > 19:31:03 if [[ "$logs" ]]; then >> > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" >> > > 19:31:03 rm -rf "${logs[@]}" >> > > 19:31:03 fi >> > > 19:31:03 >> > > 19:31:03 # stop any processes running inside the chroot >> > > 19:31:03 failed=3Dfalse >> > > 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) >> > > 19:31:03 # Clean current jobs mockroot if any >> > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do >> > > 19:31:03 [[ "$mock_conf_file" ]] || continue >> > > 19:31:03 echo "Cleaning up mock $mock_conf" >> > > 19:31:03 mock_root=3D"${mock_conf_file##*/}" >> > > 19:31:03 mock_root=3D"${mock_root%.*}" >> > > 19:31:03 my_mock=3D"/usr/bin/mock" >> > > 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" >> > > 19:31:03 my_mock+=3D" --root=3D${mock_root}" >> > > 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" >> > > 19:31:03 >> > > 19:31:03 #TODO: investigate why mock --clean fails to umount >> > > certain dirs sometimes, >> > > 19:31:03 #so we can use it instead of manually doing all this. >> > > 19:31:03 echo "Killing all mock orphan processes, if any." >> > > 19:31:03 $my_mock \ >> > > 19:31:03 --orphanskill \ >> > > 19:31:03 || { >> > > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." >> > > 19:31:03 failed=3Dtrue >> > > 19:31:03 } >> > > 19:31:03 >> > > 19:31:03 mock_root=3D"$(\ >> > > 19:31:03 grep \ >> > > 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*"=

>> > > 19:31:03 "$mock_conf_file" \ >> > > 19:31:03 )" || : >> > > 19:31:03 [[ "$mock_root" ]] || continue >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro=

>> || : >> > > 19:31:03 if [[ "$mounts" ]]; then >> > > 19:31:03 echo "Found mounted dirs inside the chroot $chroo=

>> > > Trying to umount." >> > > 19:31:03 fi >> > > 19:31:03 for mount in "${mounts[@]}"; do >> > > 19:31:03 sudo umount "$mount" \ >> > > 19:31:03 || { >> > > 19:31:03 echo "ERROR: Failed to umount $mount." >> > > 19:31:03 failed=3Dtrue >> > > 19:31:03 } >> > > 19:31:03 done >> > > 19:31:03 done >> > > 19:31:03 >> > > 19:31:03 # Clean any leftover chroot from other jobs >> > > 19:31:03 for mock_root in /var/lib/mock/*; do >> > > 19:31:03 this_chroot_failed=3Dfalse >> > > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro=

>> || : >> > > 19:31:03 if [[ "$mounts" ]]; then >> > > 19:31:03 echo "Found mounted dirs inside the chroot >> $mock_root." \ >> > > 19:31:03 "Trying to umount." >> > > 19:31:03 fi >> > > 19:31:03 for mount in "${mounts[@]}"; do >> > > 19:31:03 sudo umount "$mount" \ >> > > 19:31:03 || { >> > > 19:31:03 echo "ERROR: Failed to umount $mount." >> > > 19:31:03 failed=3Dtrue >> > > 19:31:03 this_chroot_failed=3Dtrue >> > > 19:31:03 } >> > > 19:31:03 done >> > > 19:31:03 if ! $this_chroot_failed; then >> > > 19:31:03 sudo rm -rf "$mock_root" >> > > 19:31:03 fi >> > > 19:31:03 done >> > > 19:31:03 >> > > 19:31:03 if $failed; then >> > > 19:31:03 echo "Aborting." >> > > 19:31:03 exit 1 >> > > 19:31:03 fi >> > > 19:31:03 >> > > 19:31:03 # remove mock system cache, we will setup proxies to do t=

>> > > caching and this >> > > 19:31:03 # takes lots of space between runs >> > > 19:31:03 shopt -u nullglob >> > > 19:31:03 sudo rm -Rf /var/cache/mock/* >> > > 19:31:03 >> > > 19:31:03 # restore the permissions in the working dir, as sometime=

>> > > leaves files >> > > 19:31:03 # owned by root and then the 'cleanup workspace' from jen=

>> > > job fails to >> > > 19:31:03 # clean and breaks the jobs >> > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" >> > > 19:31:03 >> > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe >> > > /tmp/hudson3681642653853563663.sh >> > > 19:31:03 + echo shell-scripts/mock_cleanup.sh >> > > 19:31:03 shell-scripts/mock_cleanup.sh >> > > 19:31:03 + shopt -s nullglob >> > > 19:31:03 + >> WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >> > > 19:31:03 + cat >> > > 19:31:03 >> _______________________________________________________________________ >> > > 19:31:03 >> ####################################################################### >> > > 19:31:03 # >> # >> > > 19:31:03 # CLEANUP >> # >> > > 19:31:03 # >> # >> > > 19:31:03 >> ####################################################################### >> > > 19:31:03 + logs=3D(./*log ./*/logs) >> > > 19:31:03 + [[ -n ./vdsm/logs ]] >> > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs >> > > 19:31:03 ./vdsm/logs/ >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log >> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.l=

>> > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log >> > > 19:31:03 + rm -rf ./vdsm/logs >> > > 19:31:03 + failed=3Dfalse >> > > 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) >> > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' >> > > 19:31:03 + [[ -n >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker=

>> > > ]] >> > > 19:31:03 + echo 'Cleaning up mock ' >> > > 19:31:03 Cleaning up mock >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg >> > > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 >> > > 19:31:03 + my_mock=3D/usr/bin/mock >> > > 19:31:03 + my_mock+=3D' >> > > >> --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

>> > > 19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' >> > > 19:31:03 + my_mock+=3D' >> > > >> --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

>> > > 19:31:03 + echo 'Killing all mock orphan processes, if any.' >> > > 19:31:03 Killing all mock orphan processes, if any. >> > > 19:31:03 + /usr/bin/mock >> > > >> --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_=

>> > > --root=3Dmocker-epel-7-x86_64.el7 >> > > --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-=

>> > > --orphanskill >> > > 19:31:04 WARNING: Could not find required logging config file: >> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/loggin=

>> > > Using default... >> > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version =3D >> 3.4.3)... >> > > 19:31:04 Start: init plugins >> > > 19:31:04 INFO: selinux enabled >> > > 19:31:04 Finish: init plugins >> > > 19:31:04 Start: run >> > > 19:31:04 Finish: run >> > > 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[=

>> > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker=

>> > > 19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00edd=

>> > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] >> > > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root=

>> > > 19:31:04 ++ mount >> > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc >> > > 19:31:04 ++ awk '{print $3}' >> > > 19:31:04 + : >> > > 19:31:04 + [[ -n '' ]] >> > > 19:31:04 + for mock_root in '/var/lib/mock/*' >> > > 19:31:04 + this_chroot_failed=3Dfalse >> > > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root=

>> > > 19:31:04 ++ mount >> > > 19:31:04 ++ grep >> /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc >> > > 19:31:04 ++ awk '{print $3}' >> > > 19:31:04 + : >> > > 19:31:04 + [[ -n '' ]] >> > > 19:31:04 + false >> > > 19:31:04 + sudo rm -rf >> > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc >> > > 19:31:04 + false >> > > 19:31:04 + shopt -u nullglob >> > > 19:31:04 + sudo rm -Rf >> > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc >> > > 19:31:04 + sudo chown -R jenkins >> > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >> > > 19:31:04 POST BUILD TASK : SUCCESS >> > > 19:31:04 END OF POST BUILD TASK : 0 >> > > 19:31:04 Archiving artifacts >> > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure >> > > 19:31:04 Started calculate disk usage of build >> > > 19:31:04 Finished Calculation of disk usage of build in 0 seconds >> > > 19:31:04 Started calculate disk usage of workspace >> > > 19:31:04 Finished Calculation of disk usage of workspace in 0 seco=

>> > > 19:31:04 Finished: FAILURE >> > _______________________________________________ >> > Infra mailing list >> > Infra(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/infra >> >> -- >> David Caro >> >> Red Hat S.L. >> Continuous Integration Engineer - EMEA ENG Virtualization R&D >> >> Tel.: +420 532 294 605 >> Email: dcaro(a)redhat.com >> IRC: dcaro|dcaroest@{freenode|oftc|redhat} >> Web: www.redhat.com >> RHT Global #: 82-62605 >> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> >> > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > >

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuJcAAoJEEBxx+HSYmnDIFMH/0tZ53spdcSVyRDCfg43a/DW QLxzIajJVmJvVbxZuUl90f14OxZIHs+/o7oYk20KXo2Weq3aYvNFxnYVw7ipSE4k M/B32wiiV4h9a0ocLagNZvB/xuAthjQ0mSG7XIIaGwioGMVh9kY4FAK1t+ue3L8C u4pLzLKaoCVgdfs/HpdkjHPMgTA4IyUN7jjSGL0DAfZXWf+fJxjfTVVMJSS8QYVy gAsDGb53dQ+VrETwMR1smyxWTq2J5t4Lcxjc1nSGxhXoDy05/m9PqUvJBFLR7UIb KpFR2LhsZjQNqhdzqO67BfpavG2M7OPYuqrAivj0rHATCWF60t+VySNLHeezSnQ= =iYSL -----END PGP SIGNATURE-----