--t5NgoZwlhlUmGr82 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/17 11:20, dcaro wrote:
Hi, =20 I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I trie= d to look for logs under the workspace and they missing probably because a n= ewer build is being built on the same slave. =20 How did you find out about that? That has happened before when the same slave was added twice on jenkins w= ith different names, but the behavior was way different (you'd get things lik= e the
On 05/17 10:36, Gil Shinar wrote: dir does not exist and such, not errors when running yum inside mock)
=20
=20 Gil =20 On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote: =20
Can we increase the history for builds on that job? We should be abl= e to debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common c= auses are:
* The repos were actually down (mirrors fail once a day usually, whi= le syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs= =2Etgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213= /console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209= /console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nest=
ed) in
workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=3D10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=3D10 19:30:49 > git clean -fdx # timeout=3D10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=3D10 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=3D10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23be= cdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0= d # timeout=3D10 19:30:52 > git branch -a # timeout=3D10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeo= ut=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{comm= it} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=3D10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=3D10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=3D10 19:30:53 > git clean -fdx # timeout=3D10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=3D10 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=3D10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22d= dab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab= 9 # timeout=3D10 19:30:53 > git branch -a # timeout=3D10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeo= ut=3D10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used = Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 = 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 = 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 = 799M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used = Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 = 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 = 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 = 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] 19:30:54 + failed=3Dfalse 19:30:54 + mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/mock 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-= artifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/b= in/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 #######################################################= ### 19:30:55 #######################################################= ### 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 #######################################################= ### 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for scri=
An example of yum failure: http://jenkins.ovirt.org/view/Master%20branch%20per%20project/view/vdsm/job= /vdsm_master_check-patch-el7-x86_64/1523/ You can see there the log in the console: 03:00:58 Start: yum install 03:01:07 ERROR: Command failed. See logs for output. 03:01:07 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_= 64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-= build --setopt=3Dtsflags=3Dnocontexts Then, in the archived artifacts logs.tgz, under vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log, you can see the error log: DEBUG util.py:474: Executing command: ['/usr/bin/yum-deprecated', '--insta= llroot', '/var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root= /', '--releasever', '7', 'install', '@buildsys-build', '--setopt=3Dtsflags= =3Dnocontexts'] with env {'SHELL': '/bin/bash', 'CCACHE_DIR': '/tmp/ccache'= , 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'CCACHE_UMA= SK': '002', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chr= oot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'HOSTNAME'= : 'mock', 'LD_PRELOAD': '/tmp/tmpxqactgv2/$LIB/nosync.so'} and shell False DEBUG util.py:399: Yum command has been deprecated, use dnf instead. DEBUG util.py:399: See 'man dnf' and 'man yum2dnf' for more information. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/bbf11fe5f13b4c6d0a9a242b422885c6ed41e05ae718d4ea3abfc3550ef4f243-comp= s-epel7.xml.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/4e6f7cf18ae8bc3553da486c7847ac8f9a50671406d26c72a4d0765f914c5c76-upda= teinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: One of the configured repositories failed ("Custom epe= l"), DEBUG util.py:399: and yum doesn't have enough cached data to continue. A= t this point the only DEBUG util.py:399: safe thing yum can do is fail. There are a few ways to= work "fix" this: DEBUG util.py:399: 1. Contact the upstream for the repository and get= them to fix the problem. DEBUG util.py:399: 2. Reconfigure the baseurl/etc. for the repository= , to point to a working DEBUG util.py:399: upstream. This is most often useful if you are = using a newer DEBUG util.py:399: distribution release than is supported by the r= epository (and the DEBUG util.py:399: packages for the previous distribution release = still work). DEBUG util.py:399: 3. Disable the repository, so yum won't use it by = default. Yum will then DEBUG util.py:399: just ignore the repository until you permanentl= y enable it again or use DEBUG util.py:399: --enablerepo for temporary usage: DEBUG util.py:399: yum-config-manager --disable epel DEBUG util.py:399: 4. Configure the failing repository to be skipped,= if it is unavailable. DEBUG util.py:399: Note that yum will try to contact the repo. whe= n it runs most commands, DEBUG util.py:399: so will have to try and fail each time (and thu= s. yum will be be much DEBUG util.py:399: slower). If it is a very temporary problem thou= gh, this is often a nice DEBUG util.py:399: compromise: DEBUG util.py:399: yum-config-manager --save --setopt=3Depel.s= kip_if_unavailable=3Dtrue DEBUG util.py:399: failure: repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f= 4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz from epel: [Errno 256] No mor= e mirrors to try. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:521: Child return code was: 1 That usually means that the mirrors were in the middle of an update, and the md5sums in the metadata were not consistent at that point it time (happens = =66rom time to time). We could try to minimize that issue handling a mirror ourselves, though that would mean another service to maintain (with it's own potential issues, like not being updated and such, but that's another subject) pt:
automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m= ock_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m= ock_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot =3D epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/= el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:30:55 mock \ 19:30:55
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x= 86_64/vdsm"
\ 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.in= it" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg= ing.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = =3D 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/roo= t/ --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dnoco= ntexts 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.= sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc =3D 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 #######################################################= ### 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc =3D 1 19:31:03 #######################################################= ### 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:31:03 mock \ 19:31:03
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x= 86_64/vdsm"
\ 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.in= it" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!####################################################= #### 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE=3D"$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jen= kins logs 19:31:03 cat <<EOC 19:31:03 ____________________________________________________________________=
19:31:03 ####################################################################= ### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################= ### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=3D( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=3Dfalse 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root=3D"${mock_conf_file##*/}" 19:31:03 mock_root=3D"${mock_root%.*}" 19:31:03 my_mock=3D"/usr/bin/mock" 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" 19:31:03 my_mock+=3D" --root=3D${mock_root}" 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all thi= s. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot= =2E" 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 19:31:03 mock_root=3D"$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']= *" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_= root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chr= oot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=3Dfalse 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_= root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 this_chroot_failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do= the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as someti= mes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from j= enkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64 19:31:03 + cat 19:31:03 ____________________________________________________________________=
19:31:03 ####################################################################= ### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################= ### 19:31:03 + logs=3D(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr= =2Elog 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=3Dfalse 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 19:31:03 + my_mock=3D/usr/bin/mock 19:31:03 + my_mock+=3D'
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64/vdsm'
19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=3D'
--resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64'
19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64/vdsm
--root=3Dmocker-epel-7-x86_64.el7 --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el= 7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg= ing.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = =3D 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\''= )[^'\'']*'
19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00e=
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7.cfg dd8fc
19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc = ]] 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=3Dfalse 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 se= conds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
=20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --t5NgoZwlhlUmGr82 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuUyAAoJEEBxx+HSYmnDQAQH/Ax1p+bYMX9BXpNUewDwHsbm cfDpl2FOYO37OpFCWpnnrWzoZByq1pgpaaub43efKodCbfTzgX1B7Yo9nw/zI1eA unxBBsnNHHt9K/4dR2ybgCKTR10tlb7ejCaODwIYv7SWGnepFZGlzKQxr7OTQP2I UrjBmU58Z1+hW/3YsGkHqB/ZQcD0lcYzBigtwoj0Ce3QZn43508B7uBVZgCVKXmM Vt04CtYQEB79aCdNCU/iYBUxg11bQCjjJntXaOXfFlcYXYQ7DTIxnZhXoXPFHyCj gkWBTZEbyCoZfjI1868l2jMyR3hoZWPEdrK3v9SkF5CJo8EyUxzeWE3rbSTB9R8= =6ebz -----END PGP SIGNATURE----- --t5NgoZwlhlUmGr82--