Another job failure: "Command failed. See logs for output"
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. Including the to make sure it will not disapper ---- 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=10 19:30:49 > git clean -fdx # timeout=10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git://gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=10 19:30:49 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 19:30:52 > git branch -a # timeout=10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=10 19:30:53 > git clean -fdx # timeout=10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git://gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=10 19:30:53 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 19:30:53 > git branch -a # timeout=10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53 --------------------------------------------------------------- 19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54 --------------------------------------------------------------- 19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' == \t\r\u\e ]] 19:30:54 + failed=false 19:30:54 + mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock 19:30:54 + chroots=("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts 19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg 19:30:55 Using proxified config /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg 19:30:55 Generating temporary mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7 19:30:55 Skipping mount points 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 ========== Initializing chroot 19:30:55 mock \ 19:30:55 --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" \ 19:30:55 --root="mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc = 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc = 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 ========== Initializing chroot 19:31:03 mock \ 19:31:03 --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" \ 19:31:03 --root="mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE="$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs 19:31:03 cat <<EOC 19:31:03 _______________________________________________________________________ 19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=false 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root="${mock_conf_file##*/}" 19:31:03 mock_root="${mock_root%.*}" 19:31:03 my_mock="/usr/bin/mock" 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" 19:31:03 my_mock+=" --root=${mock_root}" 19:31:03 my_mock+=" --resultdir=$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=true 19:31:03 } 19:31:03 19:31:03 mock_root="$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=false 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 this_chroot_failed=true 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03 _______________________________________________________________________ 19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 + logs=(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=false 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg ]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 19:31:03 + my_mock=/usr/bin/mock 19:31:03 + my_mock+=' --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm' 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=' --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64' 19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm --root=mocker-epel-7-x86_64.el7 --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=false 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
Another instance: 19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output. http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com> wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=10 19:30:49 > git clean -fdx # timeout=10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git://gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=10 19:30:49 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 19:30:52 > git branch -a # timeout=10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=10 19:30:53 > git clean -fdx # timeout=10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git://gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=10 19:30:53 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 19:30:53 > git branch -a # timeout=10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53 --------------------------------------------------------------- 19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54 --------------------------------------------------------------- 19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' == \t\r\u\e ]] 19:30:54 + failed=false 19:30:54 + mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock 19:30:54 + chroots=("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts 19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg 19:30:55 Using proxified config /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg 19:30:55 Generating temporary mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7 19:30:55 Skipping mount points 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 ========== Initializing chroot 19:30:55 mock \ 19:30:55 --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" \ 19:30:55 --root="mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc = 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc = 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 ========== Initializing chroot 19:31:03 mock \ 19:31:03 --configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm" \ 19:31:03 --root="mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE="$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs 19:31:03 cat <<EOC 19:31:03 _______________________________________________________________________ 19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=false 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root="${mock_conf_file##*/}" 19:31:03 mock_root="${mock_root%.*}" 19:31:03 my_mock="/usr/bin/mock" 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" 19:31:03 my_mock+=" --root=${mock_root}" 19:31:03 my_mock+=" --resultdir=$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=true 19:31:03 } 19:31:03 19:31:03 mock_root="$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=false 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 this_chroot_failed=true 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03 _______________________________________________________________________ 19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 + logs=(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=false 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg ]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 19:31:03 + my_mock=/usr/bin/mock 19:31:03 + my_mock+=' --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm' 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=' --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64' 19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock --configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm --root=mocker-epel-7-x86_64.el7 --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini. Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=false 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
--R3G7APHDIzY6R/pk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/15 23:23, Nir Soffer wrote:
Another instance: =20 19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
=20 http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/cons=
That means that there was an issue with the yum repos, most common causes a= re: * The repos were actually down (mirrors fail once a day usually, while sync= ing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.tgz' f= ile that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/ Usually in the log named 'root.log' (check which one has more size). ole
=20 On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com> wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/co= nsole
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=3D10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=3D10 19:30:49 > git clean -fdx # timeout=3D10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git://gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=3D10 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0= (master) 19:30:52 > git config core.sparsecheckout # timeout=3D10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # tim= eout=3D10 19:30:52 > git branch -a # timeout=3D10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # t= imeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout= =3D10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=3D10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=3D10 19:30:53 > git clean -fdx # timeout=3D10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git://gerrit.ovirt.org/jenkins.= git 19:30:53 > git --version # timeout=3D10 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=3D10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # tim= eout=3D10 19:30:53 > git branch -a # timeout=3D10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=3D10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ##############################################################= ##### 19:30:53 # Cleaning up slave = # 19:30:53 ##############################################################= ##### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G = 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G = 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G = 20% / 19:30:53 tmpfs 3.9G 244K 3.9G = 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M = 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53 --------------------------------------------------------------- 19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54 --------------------------------------------------------------- 19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G = 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G = 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G = 20% / 19:30:54 tmpfs 3.9G 244K 3.9G = 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M = 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ##############################################################= ##### 19:30:54 # Slave cleanup done = # 19:30:54 ##############################################################= ##### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x= 86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] 19:30:54 + failed=3Dfalse 19:30:54 + mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-e= l7-x86_64/mock 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-art= ifacts 19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrat= e' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock= _configs/epel-7-x86_64.cfg 19:30:55 Using proxified config /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock= _configs/epel-7-x86_64_proxied.cfg 19:30:55 Generating temporary mock conf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7 19:30:55 Skipping mount points 19:30:55 Using temp chroot =3D epel-7-x86_64-46ef12ce4362729a0f4c411e00= edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel= /7/x86_64 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:30:55 mock \ 19:30:55 --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm" \ 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging= =2Eini. Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version =3D 3.4.= 3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dnocontexts 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc =3D 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc =3D 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:31:03 mock \ 19:31:03 --configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm" \ 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE=3D"$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins lo= gs 19:31:03 cat <<EOC 19:31:03 ______________________________________________________________=
19:31:03 ##############################################################= ######### 19:31:03 # = # 19:31:03 # CLEANUP = # 19:31:03 # = # 19:31:03 ##############################################################= ######### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=3D( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=3Dfalse 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root=3D"${mock_conf_file##*/}" 19:31:03 mock_root=3D"${mock_root%.*}" 19:31:03 my_mock=3D"/usr/bin/mock" 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" 19:31:03 my_mock+=3D" --root=3D${mock_root}" 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 19:31:03 mock_root=3D"$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))= || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=3Dfalse 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root"))= || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root.= " \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 this_chroot_failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-= el7-x86_64 19:31:03 + cat 19:31:03 ______________________________________________________________=
19:31:03 ##############################################################= ######### 19:31:03 # = # 19:31:03 # CLEANUP = # 19:31:03 # = # 19:31:03 ##############################################################= ######### 19:31:03 + logs=3D(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=3Dfalse 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7.cfg ]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 19:31:03 + my_mock=3D/usr/bin/mock 19:31:03 + my_mock+=3D' --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/vdsm' 19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=3D' --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4' 19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock --configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/vdsm --root=3Dmocker-epel-7-x86_64.el7 --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging= =2Eini. Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version =3D 3.4.= 3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\''= ]*' /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7.cfg 19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=3Dfalse 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00= edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --R3G7APHDIzY6R/pk Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOjC2AAoJEEBxx+HSYmnDJZwH/3n5JkrqgDwwMlzrfQlEYpFW Q0bIRGMwF77vJv7LyY8TeNXu/Db1BnmbXVyWJr3ei/K3hu+qjIwXcbHm7MbXrFPS oexIBNiKLu3YK3RJgwxLVrpxyd5cNIwNnLSbJlQKJ14iQKtS8C+AX2ch2NvoHpwY rNOGFuOF90Y+5d8e/YKzIiWM9m2pj929nPw96GBcj+wuUCF52zFzlyXyNRwFR5Z3 IFJ9uUJOIPgL6gueG4go0qed46nrj3+0Kja1b0RjHNAEH1STHOXF8i9CNRY4Lbln 0rt3Iif/QlEnVc3tqM6uA7SaGlL/D4XNUq+tYVVryNzmtNnYTtjrRfwuCSr40BQ= =4pWU -----END PGP SIGNATURE----- --R3G7APHDIzY6R/pk--
Can we increase the history for builds on that job? We should be able to debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com> wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=10 19:30:49 > git clean -fdx # timeout=10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git://
gerrit.ovirt.org/vdsm.git
19:30:49 > git --version # timeout=10 19:30:49 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 19:30:52 > git branch -a # timeout=10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=10 19:30:53 > git clean -fdx # timeout=10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=10 19:30:53 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 19:30:53 > git branch -a # timeout=10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' == \t\r\u\e ]] 19:30:54 + failed=false 19:30:54 + mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock 19:30:54 + chroots=("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 ========== Initializing chroot 19:30:55 mock \ 19:30:55
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:30:55 --root="mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc = 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc = 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 ========== Initializing chroot 19:31:03 mock \ 19:31:03
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:31:03 --root="mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE="$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs 19:31:03 cat <<EOC 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=false 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root="${mock_conf_file##*/}" 19:31:03 mock_root="${mock_root%.*}" 19:31:03 my_mock="/usr/bin/mock" 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" 19:31:03 my_mock+=" --root=${mock_root}" 19:31:03 my_mock+=" --resultdir=$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=true 19:31:03 } 19:31:03 19:31:03 mock_root="$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=false 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 this_chroot_failed=true 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 + logs=(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=false 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 19:31:03 + my_mock=/usr/bin/mock 19:31:03 + my_mock+='
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=' --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64' 19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
--root=mocker-epel-7-x86_64.el7 --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=false 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Hi, I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I tried to look for logs under the workspace and they missing probably because a newer build is being built on the same slave. Gil On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
Can we increase the history for builds on that job? We should be able to debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested)
in
workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=10 19:30:49 > git clean -fdx # timeout=10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=10 19:30:49 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 19:30:52 > git branch -a # timeout=10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=10 19:30:53 > git clean -fdx # timeout=10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=10 19:30:53 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 19:30:53 > git branch -a # timeout=10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' == \t\r\u\e ]] 19:30:54 + failed=false 19:30:54 + mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock 19:30:54 + chroots=("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 ========== Initializing chroot 19:30:55 mock \ 19:30:55
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:30:55 --root="mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc = 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc = 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 ========== Initializing chroot 19:31:03 mock \ 19:31:03
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:31:03 --root="mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE="$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs 19:31:03 cat <<EOC 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=false 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root="${mock_conf_file##*/}" 19:31:03 mock_root="${mock_root%.*}" 19:31:03 my_mock="/usr/bin/mock" 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" 19:31:03 my_mock+=" --root=${mock_root}" 19:31:03 my_mock+=" --resultdir=$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=true 19:31:03 } 19:31:03 19:31:03 mock_root="$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=false 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 this_chroot_failed=true 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 + logs=(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=false 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 19:31:03 + my_mock=/usr/bin/mock 19:31:03 + my_mock+='
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' 19:31:03 + my_mock+='
--resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
--root=mocker-epel-7-x86_64.el7 --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=false 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--iRjOs3ViPWHdlw/I Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/17 10:36, Gil Shinar wrote:
Hi, =20 I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I tried = to look for logs under the workspace and they missing probably because a new= er build is being built on the same slave.
How did you find out about that? That has happened before when the same slave was added twice on jenkins with different names, but the behavior was way different (you'd get things like = the dir does not exist and such, not errors when running yum inside mock)
=20 Gil =20 On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote: =20
Can we increase the history for builds on that job? We should be able = to debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common cau= ses are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.t= gz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/c= onsole
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/c= onsole
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested)
in
workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=3D10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=3D10 19:30:49 > git clean -fdx # timeout=3D10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=3D10 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=3D10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becd= c0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=3D10 19:30:52 > git branch -a # timeout=3D10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout= =3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit= } # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=3D10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=3D10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=3D10 19:30:53 > git clean -fdx # timeout=3D10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=3D10 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=3D10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22dda= b9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=3D10 19:30:53 > git branch -a # timeout=3D10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout= =3D10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Av= ail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3= =2E9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3= =2E9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 7= 99M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used Av= ail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3= =2E9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3= =2E9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 7= 99M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] 19:30:54 + failed=3Dfalse 19:30:54 + mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/= mock 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-ar= tifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin= /dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/moc= k_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/moc= k_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker= -epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot =3D epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:30:55 mock \ 19:30:55
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86= _64/vdsm"
\ 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init= " \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/loggin= g.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version =3D 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dnocont= exts 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc =3D 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc =3D 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:31:03 mock \ 19:31:03
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86= _64/vdsm"
\ 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.init= " \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################= ## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE=3D"$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenki= ns logs 19:31:03 cat <<EOC 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=3D( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=3Dfalse 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root=3D"${mock_conf_file##*/}" 19:31:03 mock_root=3D"${mock_root%.*}" 19:31:03 my_mock=3D"/usr/bin/mock" 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" 19:31:03 my_mock+=3D" --root=3D${mock_root}" 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 19:31:03 mock_root=3D"$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']*"= \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroo= t. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=3Dfalse 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 this_chroot_failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do t= he caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometime= s it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jen= kins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03
19:31:03 ####################################################################### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################### 19:31:03 + logs=3D(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.l= og 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=3Dfalse 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker= -epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 19:31:03 + my_mock=3D/usr/bin/mock 19:31:03 + my_mock+=3D'
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm'
19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=3D'
--resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64'
19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm
--root=3Dmocker-epel-7-x86_64.el7 --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-= x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/loggin= g.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version =3D 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[= ^'\'']*'
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker= -epel-7-x86_64.el7.cfg
19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00edd= 8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root= ")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=3Dfalse 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root= ")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seco= nds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --iRjOs3ViPWHdlw/I Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuJcAAoJEEBxx+HSYmnDIFMH/0tZ53spdcSVyRDCfg43a/DW QLxzIajJVmJvVbxZuUl90f14OxZIHs+/o7oYk20KXo2Weq3aYvNFxnYVw7ipSE4k M/B32wiiV4h9a0ocLagNZvB/xuAthjQ0mSG7XIIaGwioGMVh9kY4FAK1t+ue3L8C u4pLzLKaoCVgdfs/HpdkjHPMgTA4IyUN7jjSGL0DAfZXWf+fJxjfTVVMJSS8QYVy gAsDGb53dQ+VrETwMR1smyxWTq2J5t4Lcxjc1nSGxhXoDy05/m9PqUvJBFLR7UIb KpFR2LhsZjQNqhdzqO67BfpavG2M7OPYuqrAivj0rHATCWF60t+VySNLHeezSnQ= =iYSL -----END PGP SIGNATURE----- --iRjOs3ViPWHdlw/I--
Hi David, I think you've got me wrong. I was referring to what Eyal said about increasing the history we keep. Not to the problem itself. I still have no idea what the problem is. I thought about attaching the logs of the Mock to the build so we'll have them archived on the master. I'll look at the mock runner script to see where it is writing this logs so we'll be able to archive them. Thanks Gil On Tue, May 17, 2016 at 12:20 PM, dcaro <dcaro@redhat.com> wrote:
On 05/17 10:36, Gil Shinar wrote:
Hi,
I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I tried to look for logs under the workspace and they missing probably because a newer build is being built on the same slave.
How did you find out about that? That has happened before when the same slave was added twice on jenkins with different names, but the behavior was way different (you'd get things like the dir does not exist and such, not errors when running yum inside mock)
Gil
On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
Can we increase the history for builds on that job? We should be able
debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23
nested) in
workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=10 19:30:49 > git clean -fdx # timeout=10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=10 19:30:49 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 19:30:52 > git branch -a # timeout=10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=10 19:30:53 > git clean -fdx # timeout=10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=10 19:30:53 > git -c core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 19:30:53 > git branch -a # timeout=10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 799M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' == \t\r\u\e ]] 19:30:54 + failed=false 19:30:54 +
mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock
19:30:54 + chroots=("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 ########################################################## 19:30:55 ########################################################## 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 ########################################################## 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for
to script:
automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static ->
http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7
19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 ========== Initializing chroot 19:30:55 mock \ 19:30:55
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:30:55 --root="mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc = 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 ########################################################## 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc = 1 19:31:03 ########################################################## 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 ========== Initializing chroot 19:31:03 mock \ 19:31:03
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
\ 19:31:03 --root="mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 ============================ 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!######################################################## 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE="$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs 19:31:03 cat <<EOC 19:31:03
19:31:03
#######################################################################
19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03
19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=false 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root="${mock_conf_file##*/}" 19:31:03 mock_root="${mock_root%.*}" 19:31:03 my_mock="/usr/bin/mock" 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" 19:31:03 my_mock+=" --root=${mock_root}" 19:31:03 my_mock+=" --resultdir=$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all this. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." 19:31:03 failed=true 19:31:03 } 19:31:03 19:31:03 mock_root="$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chroot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=false 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=true 19:31:03 this_chroot_failed=true 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do
####################################################################### the
caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as sometimes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:03 + cat 19:31:03
19:31:03
#######################################################################
19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03
#######################################################################
19:31:03 + logs=(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=false 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 19:31:03 + my_mock=/usr/bin/mock 19:31:03 + my_mock+='
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' 19:31:03 + my_mock+='
--resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
--root=mocker-epel-7-x86_64.el7
--resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
--orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*'
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=false 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--FX+Db2fp7WJhXKrW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/17 12:27, Gil Shinar wrote:
Hi David, =20 I think you've got me wrong. I was referring to what Eyal said about increasing the history we keep. Not to the problem itself.
Yep, I got you wrong :)
I still have no idea what the problem is. I thought about attaching the logs of the Mock to the build so we'll have them archived on the master. I'll look at the mock runner script to see where it is writing this logs = so we'll be able to archive them.
They are already being archived, see the previous email I sent
=20 Thanks Gil =20 =20 On Tue, May 17, 2016 at 12:20 PM, dcaro <dcaro@redhat.com> wrote: =20
On 05/17 10:36, Gil Shinar wrote:
Hi,
I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure li= ke this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I tr= ied to look for logs under the workspace and they missing probably because a newer build is being built on the same slave.
How did you find out about that? That has happened before when the same slave was added twice on jenkins with different names, but the behavior was way different (you'd get things l= ike the dir does not exist and such, not errors when running yum inside mock)
Gil
On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
Can we increase the history for builds on that job? We should be a=
debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, w= hile syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug tho= se issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size= ).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/co= nsole
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
>
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/co= nsole
> > 19:30:56 Start: yum install > 19:31:03 ERROR: Command failed. See logs for output. > > Including the to make sure it will not disapper > ---- > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 > 19:30:49 Fetching changes from the remote Git repository > 19:30:49 > git config remote.origin.url > git://gerrit.ovirt.org/vdsm.git # timeout=3D10 > 19:30:49 Cleaning workspace > 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 > 19:30:49 Resetting working tree > 19:30:49 > git reset --hard # timeout=3D10 > 19:30:49 > git clean -fdx # timeout=3D10 > 19:30:49 Pruning obsolete local branches > 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git > 19:30:49 > git --version # timeout=3D10 > 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune > 19:30:52 > git rev-parse > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D= 10 > 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) > 19:30:52 > git config core.sparsecheckout # timeout=3D10 > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4= a0d # timeout=3D10 > 19:30:52 > git branch -a # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=3D10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=3D10 > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 > 19:30:53 Fetching changes from the remote Git repository > 19:30:53 > git config remote.origin.url > git://gerrit.ovirt.org/jenkins.git # timeout=3D10 > 19:30:53 Cleaning workspace > 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 > 19:30:53 Resetting working tree > 19:30:53 > git reset --hard # timeout=3D10 > 19:30:53 > git clean -fdx # timeout=3D10 > 19:30:53 Pruning obsolete local branches > 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git > 19:30:53 > git --version # timeout=3D10 > 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress > git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > --prune > 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 > 19:30:53 Checking out Revision > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) > 19:30:53 > git config core.sparsecheckout # timeout=3D10 > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22dd= ab9 # timeout=3D10 > 19:30:53 > git branch -a # timeout=3D10 > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=3D10 > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > /tmp/hudson8265618520208925180.sh > 19:30:53 shell-scripts/cleanup_slave.sh > 19:30:53 ################################################################### > 19:30:53 # Cleaning up slave # > 19:30:53 ################################################################### > 19:30:53 Filesystem Size Used Avail > Use% Mounted on > 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev > 19:30:53 tmpfs 3.9G 0 3.9G > 0% /dev/shm > 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run > 19:30:53 tmpfs 3.9G 0 3.9G > 0% /sys/fs/cgroup > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp > 19:30:53 /dev/vda1 988M 94M 828M 11% /boot > 19:30:53 tmpfs 799M 0 799M > 0% /run/user/1000 > 19:30:53
> 19:30:53 Cleaning up postgres databases > 19:30:53 Postgres installation not found, skipping > 19:30:53 Cleaning up journal logs (if any) > 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service > 19:30:54 Cleaning up /var/tmp > 19:30:54 done > 19:30:54 Emptying some common logs > 19:30:54 /var/log/wtmp > 19:30:54 Done > 19:30:54 Making sure there are no device mappings... > 19:30:54 Removing the used loop devices... > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.servi= ce > 19:30:54
> 19:30:54 Filesystem Size Used Avail > Use% Mounted on > 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev > 19:30:54 tmpfs 3.9G 0 3.9G > 0% /dev/shm > 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run > 19:30:54 tmpfs 3.9G 0 3.9G > 0% /sys/fs/cgroup > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp > 19:30:54 /dev/vda1 988M 94M 828M 11% /boot > 19:30:54 tmpfs 799M 0 799M > 0% /run/user/1000 > 19:30:54 ################################################################### > 19:30:54 # Slave cleanup done # > 19:30:54 ################################################################### > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson7354644880288923842.sh > 19:30:54 + echo shell-scripts/global_setup.sh > 19:30:54 shell-scripts/global_setup.sh > 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > 19:30:54 + mkdir -p > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson4023353013258685526.sh > 19:30:54 + echo shell-scripts/mock_setup.sh > 19:30:54 shell-scripts/mock_setup.sh > 19:30:54 + shopt -s nullglob > 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] > 19:30:54 + failed=3Dfalse > 19:30:54 +
mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/m= ock
> 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) > 19:30:54 + false > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts > 19:30:54 + mkdir -p mock exported-artifacts > 19:30:54 + chgrp mock mock > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-art= ifacts
> 19:30:54 + chmod g+rws mock > 19:30:54 + sudo touch > 19:30:54 + : > 19:30:54 + sudo yum clean all > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf > clean all'. > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. > 19:30:54 To transfer transaction metadata from yum to DNF, run: > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf= -2 migrate' > 19:30:54 > 19:30:55 Cleaning repos: updates ci-tools fedora > 19:30:55 Cleaning up Everything > 19:30:55 + exit 0 > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > /tmp/hudson4405890361943185034.sh > 19:30:55 shell-scripts/standard_check_patch.sh > 19:30:55 ########################################################## > 19:30:55 ########################################################## > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 > 19:30:55 ########################################################## > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for
ble to script:
> automation/check-patch.sh > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:30:55 Using base mock conf >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock= _configs/epel-7-x86_64.cfg
> 19:30:55 Using proxified config >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock= _configs/epel-7-x86_64_proxied.cfg
> 19:30:55 Generating temporary mock conf >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7
> 19:30:55 Skipping mount points > 19:30:55 Using temp chroot =3D epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:30:55 Adding repo ovirt-snapshot -> > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 > 19:30:55 Adding repo ovirt-snapshot-static -> > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 > 19:30:55 Adding repo centos-glusterfs -> > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ > 19:30:55 Adding repo centos-ovirt40-candidate -> > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ > 19:30:55 Adding repo lago -> > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 > 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 > 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot > 19:30:55 mock \ > 19:30:55 >
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm"
> \ > 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ > 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.= init" \ > 19:30:55 --init > 19:30:55 WARNING: Could not find required logging config file: >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging= =2Eini.
> Using default... > 19:30:55 INFO: mock.py version 1.2.14 starting (python version= =3D 3.4.3)... > 19:30:55 Start: init plugins > 19:30:55 INFO: selinux enabled > 19:30:55 Finish: init plugins > 19:30:55 Start: run > 19:30:55 Start: clean chroot > 19:30:55 Finish: clean chroot > 19:30:55 Start: chroot init > 19:30:55 INFO: calling preinit hooks > 19:30:55 INFO: enabled root cache > 19:30:55 INFO: enabled yum cache > 19:30:55 Start: cleaning yum metadata > 19:30:55 Finish: cleaning yum metadata > 19:30:55 INFO: enabled ccache > 19:30:55 Mock Version: 1.2.14 > 19:30:55 INFO: Mock Version: 1.2.14 > 19:30:56 Start: yum install > 19:31:03 ERROR: Command failed. See logs for output. > 19:31:03 # /usr/bin/yum-deprecated --installroot > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/r= oot/ > --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dno= contexts > 19:31:03 Init took 8 seconds > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patc= h.sh > chroot finished > 19:31:03 @@ took 8 seconds > 19:31:03 @@ rc =3D 1 > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:31:03 ########################################################## > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 > 19:31:03 ## took 8 seconds > 19:31:03 ## rc =3D 1 > 19:31:03 ########################################################## > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > 19:31:03 ##! Last 20 log enties: > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > 19:31:03 ##! > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot > 19:31:03 mock \ > 19:31:03 >
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64/vdsm"
> \ > 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ > 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.= init" \ > 19:31:03 --init > 19:31:03 Init took 8 seconds > 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 19:31:03 ##! > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 19:31:03 ##!######################################################## > 19:31:03 Build step 'Execute shell' marked build as failure > 19:31:03 Performing Post build task... > 19:31:03 Match found for :.* : True > 19:31:03 Logical operation result is TRUE > 19:31:03 Running script : #!/bin/bash -xe > 19:31:03 echo "shell-scripts/mock_cleanup.sh" > 19:31:03 > 19:31:03 shopt -s nullglob > 19:31:03 > 19:31:03 > 19:31:03 WORKSPACE=3D"$PWD" > 19:31:03 > 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs > 19:31:03 cat <<EOC > 19:31:03
> 19:31:03
#######################################################################
> 19:31:03 # # > 19:31:03 # CLEANUP # > 19:31:03 # # > 19:31:03
> 19:31:03 EOC > 19:31:03 > 19:31:03 > 19:31:03 # Archive the logs, we want them anyway > 19:31:03 logs=3D( > 19:31:03 ./*log > 19:31:03 ./*/logs > 19:31:03 ) > 19:31:03 if [[ "$logs" ]]; then > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > 19:31:03 rm -rf "${logs[@]}" > 19:31:03 fi > 19:31:03 > 19:31:03 # stop any processes running inside the chroot > 19:31:03 failed=3Dfalse > 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) > 19:31:03 # Clean current jobs mockroot if any > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do > 19:31:03 [[ "$mock_conf_file" ]] || continue > 19:31:03 echo "Cleaning up mock $mock_conf" > 19:31:03 mock_root=3D"${mock_conf_file##*/}" > 19:31:03 mock_root=3D"${mock_root%.*}" > 19:31:03 my_mock=3D"/usr/bin/mock" > 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" > 19:31:03 my_mock+=3D" --root=3D${mock_root}" > 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" > 19:31:03 > 19:31:03 #TODO: investigate why mock --clean fails to umou= nt > certain dirs sometimes, > 19:31:03 #so we can use it instead of manually doing all t= his. > 19:31:03 echo "Killing all mock orphan processes, if any." > 19:31:03 $my_mock \ > 19:31:03 --orphanskill \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to kill orphans on $chro= ot." > 19:31:03 failed=3Dtrue > 19:31:03 } > 19:31:03 > 19:31:03 mock_root=3D"$(\ > 19:31:03 grep \ > 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^= ']*" \ > 19:31:03 "$mock_conf_file" \ > 19:31:03 )" || : > 19:31:03 [[ "$mock_root" ]] || continue > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || : > 19:31:03 if [[ "$mounts" ]]; then > 19:31:03 echo "Found mounted dirs inside the chroot $chroot. > Trying to umount." > 19:31:03 fi > 19:31:03 for mount in "${mounts[@]}"; do > 19:31:03 sudo umount "$mount" \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to umount $mount." > 19:31:03 failed=3Dtrue > 19:31:03 } > 19:31:03 done > 19:31:03 done > 19:31:03 > 19:31:03 # Clean any leftover chroot from other jobs > 19:31:03 for mock_root in /var/lib/mock/*; do > 19:31:03 this_chroot_failed=3Dfalse > 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) || : > 19:31:03 if [[ "$mounts" ]]; then > 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ > 19:31:03 "Trying to umount." > 19:31:03 fi > 19:31:03 for mount in "${mounts[@]}"; do > 19:31:03 sudo umount "$mount" \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to umount $mount." > 19:31:03 failed=3Dtrue > 19:31:03 this_chroot_failed=3Dtrue > 19:31:03 } > 19:31:03 done > 19:31:03 if ! $this_chroot_failed; then > 19:31:03 sudo rm -rf "$mock_root" > 19:31:03 fi > 19:31:03 done > 19:31:03 > 19:31:03 if $failed; then > 19:31:03 echo "Aborting." > 19:31:03 exit 1 > 19:31:03 fi > 19:31:03 > 19:31:03 # remove mock system cache, we will setup proxies to = do
####################################################################### the
> caching and this > 19:31:03 # takes lots of space between runs > 19:31:03 shopt -u nullglob > 19:31:03 sudo rm -Rf /var/cache/mock/* > 19:31:03 > 19:31:03 # restore the permissions in the working dir, as sometimes it > leaves files > 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins > job fails to > 19:31:03 # clean and breaks the jobs > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" > 19:31:03 > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson3681642653853563663.sh > 19:31:03 + echo shell-scripts/mock_cleanup.sh > 19:31:03 shell-scripts/mock_cleanup.sh > 19:31:03 + shopt -s nullglob > 19:31:03 + WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64 > 19:31:03 + cat > 19:31:03
> 19:31:03
#######################################################################
> 19:31:03 # # > 19:31:03 # CLEANUP # > 19:31:03 # # > 19:31:03
#######################################################################
> 19:31:03 + logs=3D(./*log ./*/logs) > 19:31:03 + [[ -n ./vdsm/logs ]] > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs > 19:31:03 ./vdsm/logs/ > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log > 19:31:03 + rm -rf ./vdsm/logs > 19:31:03 + failed=3Dfalse > 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' > 19:31:03 + [[ -n >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7.cfg
> ]] > 19:31:03 + echo 'Cleaning up mock ' > 19:31:03 Cleaning up mock > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg > 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 > 19:31:03 + my_mock=3D/usr/bin/mock > 19:31:03 + my_mock+=3D' >
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/vdsm'
> 19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' > 19:31:03 + my_mock+=3D' >
--resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4'
> 19:31:03 + echo 'Killing all mock orphan processes, if any.' > 19:31:03 Killing all mock orphan processes, if any. > 19:31:03 + /usr/bin/mock >
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/vdsm
> --root=3Dmocker-epel-7-x86_64.el7 > --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > --orphanskill > 19:31:04 WARNING: Could not find required logging config file: >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging= =2Eini.
> Using default... > 19:31:04 INFO: mock.py version 1.2.14 starting (python version= =3D 3.4.3)... > 19:31:04 Start: init plugins > 19:31:04 INFO: selinux enabled > 19:31:04 Finish: init plugins > 19:31:04 Start: run > 19:31:04 Finish: run > 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\'')[^'\'']*' >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-= epel-7-x86_64.el7.cfg
> 19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8f= c ]] > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) > 19:31:04 ++ mount > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 ++ awk '{print $3}' > 19:31:04 + : > 19:31:04 + [[ -n '' ]] > 19:31:04 + for mock_root in '/var/lib/mock/*' > 19:31:04 + this_chroot_failed=3Dfalse > 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_root")) > 19:31:04 ++ mount > 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 ++ awk '{print $3}' > 19:31:04 + : > 19:31:04 + [[ -n '' ]] > 19:31:04 + false > 19:31:04 + sudo rm -rf > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + false > 19:31:04 + shopt -u nullglob > 19:31:04 + sudo rm -Rf > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + sudo chown -R jenkins > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > 19:31:04 POST BUILD TASK : SUCCESS > 19:31:04 END OF POST BUILD TASK : 0 > 19:31:04 Archiving artifacts > 19:31:04 Build step 'Groovy Postbuild' marked build as failure > 19:31:04 Started calculate disk usage of build > 19:31:04 Finished Calculation of disk usage of build in 0 seco= nds > 19:31:04 Started calculate disk usage of workspace > 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds > 19:31:04 Finished: FAILURE _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --FX+Db2fp7WJhXKrW Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuVuAAoJEEBxx+HSYmnD8KsIAJt1699HIBsr31Ud8Wm8+XN3 45uDC4Pd1W2VJUzDdW6JpaTNU2ywE1YgAOSG6OgC+kBxa5+zyhWks7MQYzbPJx4F VsyH8Ph/zTUjbuLmYt6WQQRuSaJZTSfXjgRAK3wiMRVklVHGH/C3fRgyuysuFssV 3RpaE5RFrAuBwkMdzauRUUBbeo3mjSfrHCHxCTdNYRll/AQcwyLWzYTBfZ2ZXiYi Sc+gQ5pMr10Nagkl+me3itcTdKjcOPyiO9xltq6fQ3DPkQc57G4odiCikEBBpyeK HwgzIFgnZQbDdHNuSfe8pQ0Yys/O56m2tz6nXx6NdelcEgz1XWFp/OAYw64+AA8= =guUe -----END PGP SIGNATURE----- --FX+Db2fp7WJhXKrW--
OK. My bad :-) Thanks Gil On Tue, May 17, 2016 at 12:33 PM, dcaro <dcaro@redhat.com> wrote:
On 05/17 12:27, Gil Shinar wrote:
Hi David,
I think you've got me wrong. I was referring to what Eyal said about increasing the history we keep. Not to the problem itself.
Yep, I got you wrong :)
I still have no idea what the problem is. I thought about attaching the logs of the Mock to the build so we'll have them archived on the master. I'll look at the mock runner script to see where it is writing this logs so we'll be able to archive them.
They are already being archived, see the previous email I sent
Thanks Gil
On Tue, May 17, 2016 at 12:20 PM, dcaro <dcaro@redhat.com> wrote:
On 05/17 10:36, Gil Shinar wrote:
Hi,
I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure
this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I
to
look for logs under the workspace and they missing probably because a newer build is being built on the same slave.
How did you find out about that? That has happened before when the same slave was added twice on jenkins with different names, but the behavior was way different (you'd get things
the dir does not exist and such, not errors when running yum inside mock)
Gil
On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
Can we increase the history for builds on that job? We should be
able to
debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote: > Another instance: > > 19:38:15 Start: yum install > 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug
issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
> >
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
> > On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <
nsoffer@redhat.com>
wrote: > >
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
> > > > 19:30:56 Start: yum install > > 19:31:03 ERROR: Command failed. See logs for output. > > > > Including the to make sure it will not disapper > > ---- > > > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 > > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in > > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 > > 19:30:49 Fetching changes from the remote Git repository > > 19:30:49 > git config remote.origin.url > > git://gerrit.ovirt.org/vdsm.git # timeout=10 > > 19:30:49 Cleaning workspace > > 19:30:49 > git rev-parse --verify HEAD # timeout=10 > > 19:30:49 Resetting working tree > > 19:30:49 > git reset --hard # timeout=10 > > 19:30:49 > git clean -fdx # timeout=10 > > 19:30:49 Pruning obsolete local branches > > 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git > > 19:30:49 > git --version # timeout=10 > > 19:30:49 > git -c core.askpass=true fetch --tags --progress > > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune > > 19:30:52 > git rev-parse > > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 > > 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) > > 19:30:52 > git config core.sparsecheckout # timeout=10 > > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 > > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 > > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 > > 19:30:52 > git branch -a # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 > > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 > > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 > > 19:30:53 Fetching changes from the remote Git repository > > 19:30:53 > git config remote.origin.url > > git://gerrit.ovirt.org/jenkins.git # timeout=10 > > 19:30:53 Cleaning workspace > > 19:30:53 > git rev-parse --verify HEAD # timeout=10 > > 19:30:53 Resetting working tree > > 19:30:53 > git reset --hard # timeout=10 > > 19:30:53 > git clean -fdx # timeout=10 > > 19:30:53 Pruning obsolete local branches > > 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git > > 19:30:53 > git --version # timeout=10 > > 19:30:53 > git -c core.askpass=true fetch --tags --progress > > git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > > --prune > > 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 > > 19:30:53 Checking out Revision > > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) > > 19:30:53 > git config core.sparsecheckout # timeout=10 > > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 > > 19:30:53 > git branch -a # timeout=10 > > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 > > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > /tmp/hudson8265618520208925180.sh > > 19:30:53 shell-scripts/cleanup_slave.sh > > 19:30:53
###################################################################
> > 19:30:53 # Cleaning up slave # > > 19:30:53
###################################################################
> > 19:30:53 Filesystem Size Used Avail > > Use% Mounted on > > 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev > > 19:30:53 tmpfs 3.9G 0 3.9G > > 0% /dev/shm > > 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run > > 19:30:53 tmpfs 3.9G 0 3.9G > > 0% /sys/fs/cgroup > > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > > 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp > > 19:30:53 /dev/vda1 988M 94M 828M 11% /boot > > 19:30:53 tmpfs 799M 0 799M > > 0% /run/user/1000 > > 19:30:53 --------------------------------------------------------------- > > 19:30:53 Cleaning up postgres databases > > 19:30:53 Postgres installation not found, skipping > > 19:30:53 Cleaning up journal logs (if any) > > 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service > > 19:30:54 Cleaning up /var/tmp > > 19:30:54 done > > 19:30:54 Emptying some common logs > > 19:30:54 /var/log/wtmp > > 19:30:54 Done > > 19:30:54 Making sure there are no device mappings... > > 19:30:54 Removing the used loop devices... > > 19:30:54 Redirecting to /bin/systemctl restart
> > 19:30:54 --------------------------------------------------------------- > > 19:30:54 Filesystem Size Used Avail > > Use% Mounted on > > 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev > > 19:30:54 tmpfs 3.9G 0 3.9G > > 0% /dev/shm > > 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run > > 19:30:54 tmpfs 3.9G 0 3.9G > > 0% /sys/fs/cgroup > > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > > 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp > > 19:30:54 /dev/vda1 988M 94M 828M 11% /boot > > 19:30:54 tmpfs 799M 0 799M > > 0% /run/user/1000 > > 19:30:54
###################################################################
> > 19:30:54 # Slave cleanup done # > > 19:30:54
###################################################################
> > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > /tmp/hudson7354644880288923842.sh > > 19:30:54 + echo shell-scripts/global_setup.sh > > 19:30:54 shell-scripts/global_setup.sh > > 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > 19:30:54 + mkdir -p > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > /tmp/hudson4023353013258685526.sh > > 19:30:54 + echo shell-scripts/mock_setup.sh > > 19:30:54 shell-scripts/mock_setup.sh > > 19:30:54 + shopt -s nullglob > > 19:30:54 + [[ '' == \t\r\u\e ]] > > 19:30:54 + failed=false > > 19:30:54 +
mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock
> > 19:30:54 + chroots=("$WORKSPACE"/mock/*) > > 19:30:54 + false > > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts > > 19:30:54 + mkdir -p mock exported-artifacts > > 19:30:54 + chgrp mock mock > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
> > 19:30:54 + chmod g+rws mock > > 19:30:54 + sudo touch > > 19:30:54 + : > > 19:30:54 + sudo yum clean all > > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf > > clean all'. > > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. > > 19:30:54 To transfer transaction metadata from yum to DNF, run: > > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' > > 19:30:54 > > 19:30:55 Cleaning repos: updates ci-tools fedora > > 19:30:55 Cleaning up Everything > > 19:30:55 + exit 0 > > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > > /tmp/hudson4405890361943185034.sh > > 19:30:55 shell-scripts/standard_check_patch.sh > > 19:30:55 ########################################################## > > 19:30:55 ########################################################## > > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 > > 19:30:55 ########################################################## > > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for script: > > automation/check-patch.sh > > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > 19:30:55 Using base mock conf > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
> > 19:30:55 Using proxified config > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
> > 19:30:55 Generating temporary mock conf > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
> > 19:30:55 Skipping mount points > > 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:30:55 Adding repo ovirt-snapshot -> > > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 > > 19:30:55 Adding repo ovirt-snapshot-static -> > > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 > > 19:30:55 Adding repo centos-glusterfs -> > > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ > > 19:30:55 Adding repo centos-ovirt40-candidate -> > > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ > > 19:30:55 Adding repo lago -> > > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 > > 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 > > 19:30:55 ========== Initializing chroot > > 19:30:55 mock \ > > 19:30:55 > >
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> > \ > > 19:30:55 --root="mocker-epel-7-x86_64.el7" \ > > 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ > > 19:30:55 --init > > 19:30:55 WARNING: Could not find required logging config file: > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> > Using default... > > 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... > > 19:30:55 Start: init plugins > > 19:30:55 INFO: selinux enabled > > 19:30:55 Finish: init plugins > > 19:30:55 Start: run > > 19:30:55 Start: clean chroot > > 19:30:55 Finish: clean chroot > > 19:30:55 Start: chroot init > > 19:30:55 INFO: calling preinit hooks > > 19:30:55 INFO: enabled root cache > > 19:30:55 INFO: enabled yum cache > > 19:30:55 Start: cleaning yum metadata > > 19:30:55 Finish: cleaning yum metadata > > 19:30:55 INFO: enabled ccache > > 19:30:55 Mock Version: 1.2.14 > > 19:30:55 INFO: Mock Version: 1.2.14 > > 19:30:56 Start: yum install > > 19:31:03 ERROR: Command failed. See logs for output. > > 19:31:03 # /usr/bin/yum-deprecated --installroot > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ > > --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts > > 19:31:03 Init took 8 seconds > > 19:31:03 ============================ > > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh > > chroot finished > > 19:31:03 @@ took 8 seconds > > 19:31:03 @@ rc = 1 > > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > 19:31:03 ########################################################## > > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 > > 19:31:03 ## took 8 seconds > > 19:31:03 ## rc = 1 > > 19:31:03 ########################################################## > > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > > 19:31:03 ##! Last 20 log enties: > > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > 19:31:03 ##! > > 19:31:03 ========== Initializing chroot > > 19:31:03 mock \ > > 19:31:03 > >
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> > \ > > 19:31:03 --root="mocker-epel-7-x86_64.el7" \ > > 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ > > 19:31:03 --init > > 19:31:03 Init took 8 seconds > > 19:31:03 ============================ > > 19:31:03 ##! > > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > 19:31:03 ##!######################################################## > > 19:31:03 Build step 'Execute shell' marked build as failure > > 19:31:03 Performing Post build task... > > 19:31:03 Match found for :.* : True > > 19:31:03 Logical operation result is TRUE > > 19:31:03 Running script : #!/bin/bash -xe > > 19:31:03 echo "shell-scripts/mock_cleanup.sh" > > 19:31:03 > > 19:31:03 shopt -s nullglob > > 19:31:03 > > 19:31:03 > > 19:31:03 WORKSPACE="$PWD" > > 19:31:03 > > 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs > > 19:31:03 cat <<EOC > > 19:31:03
> > 19:31:03
#######################################################################
> > 19:31:03 # # > > 19:31:03 # CLEANUP # > > 19:31:03 # # > > 19:31:03
#######################################################################
> > 19:31:03 EOC > > 19:31:03 > > 19:31:03 > > 19:31:03 # Archive the logs, we want them anyway > > 19:31:03 logs=( > > 19:31:03 ./*log > > 19:31:03 ./*/logs > > 19:31:03 ) > > 19:31:03 if [[ "$logs" ]]; then > > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > > 19:31:03 rm -rf "${logs[@]}" > > 19:31:03 fi > > 19:31:03 > > 19:31:03 # stop any processes running inside the chroot > > 19:31:03 failed=false > > 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) > > 19:31:03 # Clean current jobs mockroot if any > > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do > > 19:31:03 [[ "$mock_conf_file" ]] || continue > > 19:31:03 echo "Cleaning up mock $mock_conf" > > 19:31:03 mock_root="${mock_conf_file##*/}" > > 19:31:03 mock_root="${mock_root%.*}" > > 19:31:03 my_mock="/usr/bin/mock" > > 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" > > 19:31:03 my_mock+=" --root=${mock_root}" > > 19:31:03 my_mock+=" --resultdir=$WORKSPACE" > > 19:31:03 > > 19:31:03 #TODO: investigate why mock --clean fails to umount > > certain dirs sometimes, > > 19:31:03 #so we can use it instead of manually doing all
like tried like those libvirtd.service this.
> > 19:31:03 echo "Killing all mock orphan processes, if any." > > 19:31:03 $my_mock \ > > 19:31:03 --orphanskill \ > > 19:31:03 || { > > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." > > 19:31:03 failed=true > > 19:31:03 } > > 19:31:03 > > 19:31:03 mock_root="$(\ > > 19:31:03 grep \ > > 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ > > 19:31:03 "$mock_conf_file" \ > > 19:31:03 )" || : > > 19:31:03 [[ "$mock_root" ]] || continue > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > > 19:31:03 if [[ "$mounts" ]]; then > > 19:31:03 echo "Found mounted dirs inside the chroot $chroot. > > Trying to umount." > > 19:31:03 fi > > 19:31:03 for mount in "${mounts[@]}"; do > > 19:31:03 sudo umount "$mount" \ > > 19:31:03 || { > > 19:31:03 echo "ERROR: Failed to umount $mount." > > 19:31:03 failed=true > > 19:31:03 } > > 19:31:03 done > > 19:31:03 done > > 19:31:03 > > 19:31:03 # Clean any leftover chroot from other jobs > > 19:31:03 for mock_root in /var/lib/mock/*; do > > 19:31:03 this_chroot_failed=false > > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > > 19:31:03 if [[ "$mounts" ]]; then > > 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ > > 19:31:03 "Trying to umount." > > 19:31:03 fi > > 19:31:03 for mount in "${mounts[@]}"; do > > 19:31:03 sudo umount "$mount" \ > > 19:31:03 || { > > 19:31:03 echo "ERROR: Failed to umount $mount." > > 19:31:03 failed=true > > 19:31:03 this_chroot_failed=true > > 19:31:03 } > > 19:31:03 done > > 19:31:03 if ! $this_chroot_failed; then > > 19:31:03 sudo rm -rf "$mock_root" > > 19:31:03 fi > > 19:31:03 done > > 19:31:03 > > 19:31:03 if $failed; then > > 19:31:03 echo "Aborting." > > 19:31:03 exit 1 > > 19:31:03 fi > > 19:31:03 > > 19:31:03 # remove mock system cache, we will setup proxies to do the > > caching and this > > 19:31:03 # takes lots of space between runs > > 19:31:03 shopt -u nullglob > > 19:31:03 sudo rm -Rf /var/cache/mock/* > > 19:31:03 > > 19:31:03 # restore the permissions in the working dir, as sometimes it > > leaves files > > 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins > > job fails to > > 19:31:03 # clean and breaks the jobs > > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" > > 19:31:03 > > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > > /tmp/hudson3681642653853563663.sh > > 19:31:03 + echo shell-scripts/mock_cleanup.sh > > 19:31:03 shell-scripts/mock_cleanup.sh > > 19:31:03 + shopt -s nullglob > > 19:31:03 +
WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
> > 19:31:03 + cat > > 19:31:03
> > 19:31:03
#######################################################################
> > 19:31:03 # # > > 19:31:03 # CLEANUP # > > 19:31:03 # # > > 19:31:03
#######################################################################
> > 19:31:03 + logs=(./*log ./*/logs) > > 19:31:03 + [[ -n ./vdsm/logs ]] > > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs > > 19:31:03 ./vdsm/logs/ > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log > > 19:31:03 + rm -rf ./vdsm/logs > > 19:31:03 + failed=false > > 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) > > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' > > 19:31:03 + [[ -n > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> > ]] > > 19:31:03 + echo 'Cleaning up mock ' > > 19:31:03 Cleaning up mock > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg > > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 > > 19:31:03 + my_mock=/usr/bin/mock > > 19:31:03 + my_mock+=' > >
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
> > 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' > > 19:31:03 + my_mock+=' > >
--resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
> > 19:31:03 + echo 'Killing all mock orphan processes, if any.' > > 19:31:03 Killing all mock orphan processes, if any. > > 19:31:03 + /usr/bin/mock > >
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
> > --root=mocker-epel-7-x86_64.el7 > > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > --orphanskill > > 19:31:04 WARNING: Could not find required logging config file: > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> > Using default... > > 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... > > 19:31:04 Start: init plugins > > 19:31:04 INFO: selinux enabled > > 19:31:04 Finish: init plugins > > 19:31:04 Start: run > > 19:31:04 Finish: run > > 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' > >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> > 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > > 19:31:04 ++ mount > > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:31:04 ++ awk '{print $3}' > > 19:31:04 + : > > 19:31:04 + [[ -n '' ]] > > 19:31:04 + for mock_root in '/var/lib/mock/*' > > 19:31:04 + this_chroot_failed=false > > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > > 19:31:04 ++ mount > > 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:31:04 ++ awk '{print $3}' > > 19:31:04 + : > > 19:31:04 + [[ -n '' ]] > > 19:31:04 + false > > 19:31:04 + sudo rm -rf > > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:31:04 + false > > 19:31:04 + shopt -u nullglob > > 19:31:04 + sudo rm -Rf > > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > > 19:31:04 + sudo chown -R jenkins > > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > 19:31:04 POST BUILD TASK : SUCCESS > > 19:31:04 END OF POST BUILD TASK : 0 > > 19:31:04 Archiving artifacts > > 19:31:04 Build step 'Groovy Postbuild' marked build as failure > > 19:31:04 Started calculate disk usage of build > > 19:31:04 Finished Calculation of disk usage of build in 0 seconds > > 19:31:04 Started calculate disk usage of workspace > > 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds > > 19:31:04 Finished: FAILURE > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--t5NgoZwlhlUmGr82 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/17 11:20, dcaro wrote:
Hi, =20 I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I trie= d to look for logs under the workspace and they missing probably because a n= ewer build is being built on the same slave. =20 How did you find out about that? That has happened before when the same slave was added twice on jenkins w= ith different names, but the behavior was way different (you'd get things lik= e the
On 05/17 10:36, Gil Shinar wrote: dir does not exist and such, not errors when running yum inside mock)
=20
=20 Gil =20 On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote: =20
Can we increase the history for builds on that job? We should be abl= e to debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common c= auses are:
* The repos were actually down (mirrors fail once a day usually, whi= le syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs= =2Etgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213= /console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209= /console
19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output.
Including the to make sure it will not disapper ----
19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nest=
ed) in
workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64 19:30:48 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:49 Fetching changes from the remote Git repository 19:30:49 > git config remote.origin.url git://gerrit.ovirt.org/vdsm.git # timeout=3D10 19:30:49 Cleaning workspace 19:30:49 > git rev-parse --verify HEAD # timeout=3D10 19:30:49 Resetting working tree 19:30:49 > git reset --hard # timeout=3D10 19:30:49 > git clean -fdx # timeout=3D10 19:30:49 Pruning obsolete local branches 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git 19:30:49 > git --version # timeout=3D10 19:30:49 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune 19:30:52 > git rev-parse 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=3D10 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) 19:30:52 > git config core.sparsecheckout # timeout=3D10 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23be= cdc0 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=3D10 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0= d # timeout=3D10 19:30:52 > git branch -a # timeout=3D10 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeo= ut=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{comm= it} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=3D10 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=3D10 19:30:53 > git rev-parse --is-inside-work-tree # timeout=3D10 19:30:53 Fetching changes from the remote Git repository 19:30:53 > git config remote.origin.url git://gerrit.ovirt.org/jenkins.git # timeout=3D10 19:30:53 Cleaning workspace 19:30:53 > git rev-parse --verify HEAD # timeout=3D10 19:30:53 Resetting working tree 19:30:53 > git reset --hard # timeout=3D10 19:30:53 > git clean -fdx # timeout=3D10 19:30:53 Pruning obsolete local branches 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git 19:30:53 > git --version # timeout=3D10 19:30:53 > git -c core.askpass=3Dtrue fetch --tags --progress git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune 19:30:53 > git rev-parse origin/master^{commit} # timeout=3D10 19:30:53 Checking out Revision 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) 19:30:53 > git config core.sparsecheckout # timeout=3D10 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22d= dab9 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab= 9 # timeout=3D10 19:30:53 > git branch -a # timeout=3D10 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeo= ut=3D10 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson8265618520208925180.sh 19:30:53 shell-scripts/cleanup_slave.sh 19:30:53 ################################################################### 19:30:53 # Cleaning up slave # 19:30:53 ################################################################### 19:30:53 Filesystem Size Used = Avail Use% Mounted on 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:53 tmpfs 3.9G 0 = 3.9G 0% /dev/shm 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:53 tmpfs 3.9G 0 = 3.9G 0% /sys/fs/cgroup 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:53 /dev/vda1 988M 94M 828M 11% /boot 19:30:53 tmpfs 799M 0 = 799M 0% /run/user/1000 19:30:53
19:30:53 Cleaning up postgres databases 19:30:53 Postgres installation not found, skipping 19:30:53 Cleaning up journal logs (if any) 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service 19:30:54 Cleaning up /var/tmp 19:30:54 done 19:30:54 Emptying some common logs 19:30:54 /var/log/wtmp 19:30:54 Done 19:30:54 Making sure there are no device mappings... 19:30:54 Removing the used loop devices... 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service 19:30:54
19:30:54 Filesystem Size Used = Avail Use% Mounted on 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev 19:30:54 tmpfs 3.9G 0 = 3.9G 0% /dev/shm 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run 19:30:54 tmpfs 3.9G 0 = 3.9G 0% /sys/fs/cgroup 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp 19:30:54 /dev/vda1 988M 94M 828M 11% /boot 19:30:54 tmpfs 799M 0 = 799M 0% /run/user/1000 19:30:54 ################################################################### 19:30:54 # Slave cleanup done # 19:30:54 ################################################################### 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson7354644880288923842.sh 19:30:54 + echo shell-scripts/global_setup.sh 19:30:54 shell-scripts/global_setup.sh 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 + mkdir -p /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson4023353013258685526.sh 19:30:54 + echo shell-scripts/mock_setup.sh 19:30:54 shell-scripts/mock_setup.sh 19:30:54 + shopt -s nullglob 19:30:54 + [[ '' =3D=3D \t\r\u\e ]] 19:30:54 + failed=3Dfalse 19:30:54 + mock_dir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_6= 4/mock 19:30:54 + chroots=3D("$WORKSPACE"/mock/*) 19:30:54 + false 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts 19:30:54 + mkdir -p mock exported-artifacts 19:30:54 + chgrp mock mock /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-= artifacts
19:30:54 + chmod g+rws mock 19:30:54 + sudo touch 19:30:54 + : 19:30:54 + sudo yum clean all 19:30:54 Yum command has been deprecated, redirecting to '/usr/b= in/dnf clean all'. 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. 19:30:54 To transfer transaction metadata from yum to DNF, run: 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' 19:30:54 19:30:55 Cleaning repos: updates ci-tools fedora 19:30:55 Cleaning up Everything 19:30:55 + exit 0 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e /tmp/hudson4405890361943185034.sh 19:30:55 shell-scripts/standard_check_patch.sh 19:30:55 #######################################################= ### 19:30:55 #######################################################= ### 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 19:30:55 #######################################################= ### 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for scri=
An example of yum failure: http://jenkins.ovirt.org/view/Master%20branch%20per%20project/view/vdsm/job= /vdsm_master_check-patch-el7-x86_64/1523/ You can see there the log in the console: 03:00:58 Start: yum install 03:01:07 ERROR: Command failed. See logs for output. 03:01:07 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_= 64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-= build --setopt=3Dtsflags=3Dnocontexts Then, in the archived artifacts logs.tgz, under vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log, you can see the error log: DEBUG util.py:474: Executing command: ['/usr/bin/yum-deprecated', '--insta= llroot', '/var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root= /', '--releasever', '7', 'install', '@buildsys-build', '--setopt=3Dtsflags= =3Dnocontexts'] with env {'SHELL': '/bin/bash', 'CCACHE_DIR': '/tmp/ccache'= , 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'CCACHE_UMA= SK': '002', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chr= oot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'HOSTNAME'= : 'mock', 'LD_PRELOAD': '/tmp/tmpxqactgv2/$LIB/nosync.so'} and shell False DEBUG util.py:399: Yum command has been deprecated, use dnf instead. DEBUG util.py:399: See 'man dnf' and 'man yum2dnf' for more information. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/bbf11fe5f13b4c6d0a9a242b422885c6ed41e05ae718d4ea3abfc3550ef4f243-comp= s-epel7.xml.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/4e6f7cf18ae8bc3553da486c7847ac8f9a50671406d26c72a4d0765f914c5c76-upda= teinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: One of the configured repositories failed ("Custom epe= l"), DEBUG util.py:399: and yum doesn't have enough cached data to continue. A= t this point the only DEBUG util.py:399: safe thing yum can do is fail. There are a few ways to= work "fix" this: DEBUG util.py:399: 1. Contact the upstream for the repository and get= them to fix the problem. DEBUG util.py:399: 2. Reconfigure the baseurl/etc. for the repository= , to point to a working DEBUG util.py:399: upstream. This is most often useful if you are = using a newer DEBUG util.py:399: distribution release than is supported by the r= epository (and the DEBUG util.py:399: packages for the previous distribution release = still work). DEBUG util.py:399: 3. Disable the repository, so yum won't use it by = default. Yum will then DEBUG util.py:399: just ignore the repository until you permanentl= y enable it again or use DEBUG util.py:399: --enablerepo for temporary usage: DEBUG util.py:399: yum-config-manager --disable epel DEBUG util.py:399: 4. Configure the failing repository to be skipped,= if it is unavailable. DEBUG util.py:399: Note that yum will try to contact the repo. whe= n it runs most commands, DEBUG util.py:399: so will have to try and fail each time (and thu= s. yum will be be much DEBUG util.py:399: slower). If it is a very temporary problem thou= gh, this is often a nice DEBUG util.py:399: compromise: DEBUG util.py:399: yum-config-manager --save --setopt=3Depel.s= kip_if_unavailable=3Dtrue DEBUG util.py:399: failure: repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f= 4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz from epel: [Errno 256] No mor= e mirrors to try. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/rep= odata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-prim= ary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:521: Child return code was: 1 That usually means that the mirrors were in the middle of an update, and the md5sums in the metadata were not consistent at that point it time (happens = =66rom time to time). We could try to minimize that issue handling a mirror ourselves, though that would mean another service to maintain (with it's own potential issues, like not being updated and such, but that's another subject) pt:
automation/check-patch.sh 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:30:55 Using base mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m= ock_configs/epel-7-x86_64.cfg
19:30:55 Using proxified config
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/m= ock_configs/epel-7-x86_64_proxied.cfg
19:30:55 Generating temporary mock conf
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7
19:30:55 Skipping mount points 19:30:55 Using temp chroot =3D epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:30:55 Adding repo ovirt-snapshot -> http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 19:30:55 Adding repo ovirt-snapshot-static -> http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/= el7 19:30:55 Adding repo centos-glusterfs -> http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ 19:30:55 Adding repo centos-ovirt40-candidate -> http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ 19:30:55 Adding repo lago -> http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 19:30:55 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:30:55 mock \ 19:30:55
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x= 86_64/vdsm"
\ 19:30:55 --root=3D"mocker-epel-7-x86_64.el7" \ 19:30:55 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.in= it" \ 19:30:55 --init 19:30:55 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg= ing.ini.
Using default... 19:30:55 INFO: mock.py version 1.2.14 starting (python version = =3D 3.4.3)... 19:30:55 Start: init plugins 19:30:55 INFO: selinux enabled 19:30:55 Finish: init plugins 19:30:55 Start: run 19:30:55 Start: clean chroot 19:30:55 Finish: clean chroot 19:30:55 Start: chroot init 19:30:55 INFO: calling preinit hooks 19:30:55 INFO: enabled root cache 19:30:55 INFO: enabled yum cache 19:30:55 Start: cleaning yum metadata 19:30:55 Finish: cleaning yum metadata 19:30:55 INFO: enabled ccache 19:30:55 Mock Version: 1.2.14 19:30:55 INFO: Mock Version: 1.2.14 19:30:56 Start: yum install 19:31:03 ERROR: Command failed. See logs for output. 19:31:03 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/roo= t/ --releasever 7 install @buildsys-build --setopt=3Dtsflags=3Dnoco= ntexts 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.= sh chroot finished 19:31:03 @@ took 8 seconds 19:31:03 @@ rc =3D 1 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19:31:03 #######################################################= ### 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 19:31:03 ## took 8 seconds 19:31:03 ## rc =3D 1 19:31:03 #######################################################= ### 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 19:31:03 ##! Last 20 log enties: logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log 19:31:03 ##! 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initializing chroot 19:31:03 mock \ 19:31:03
--configdir=3D"/home/jenkins/workspace/vdsm_master_check-patch-el7-x= 86_64/vdsm"
\ 19:31:03 --root=3D"mocker-epel-7-x86_64.el7" \ 19:31:03 --resultdir=3D"logs/mocker-epel-7-x86_64.el7.in= it" \ 19:31:03 --init 19:31:03 Init took 8 seconds 19:31:03 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 19:31:03 ##! 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 19:31:03 ##!####################################################= #### 19:31:03 Build step 'Execute shell' marked build as failure 19:31:03 Performing Post build task... 19:31:03 Match found for :.* : True 19:31:03 Logical operation result is TRUE 19:31:03 Running script : #!/bin/bash -xe 19:31:03 echo "shell-scripts/mock_cleanup.sh" 19:31:03 19:31:03 shopt -s nullglob 19:31:03 19:31:03 19:31:03 WORKSPACE=3D"$PWD" 19:31:03 19:31:03 # Make clear this is the cleanup, helps reading the jen= kins logs 19:31:03 cat <<EOC 19:31:03 ____________________________________________________________________=
19:31:03 ####################################################################= ### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################= ### 19:31:03 EOC 19:31:03 19:31:03 19:31:03 # Archive the logs, we want them anyway 19:31:03 logs=3D( 19:31:03 ./*log 19:31:03 ./*/logs 19:31:03 ) 19:31:03 if [[ "$logs" ]]; then 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" 19:31:03 rm -rf "${logs[@]}" 19:31:03 fi 19:31:03 19:31:03 # stop any processes running inside the chroot 19:31:03 failed=3Dfalse 19:31:03 mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 # Clean current jobs mockroot if any 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do 19:31:03 [[ "$mock_conf_file" ]] || continue 19:31:03 echo "Cleaning up mock $mock_conf" 19:31:03 mock_root=3D"${mock_conf_file##*/}" 19:31:03 mock_root=3D"${mock_root%.*}" 19:31:03 my_mock=3D"/usr/bin/mock" 19:31:03 my_mock+=3D" --configdir=3D${mock_conf_file%/*}" 19:31:03 my_mock+=3D" --root=3D${mock_root}" 19:31:03 my_mock+=3D" --resultdir=3D$WORKSPACE" 19:31:03 19:31:03 #TODO: investigate why mock --clean fails to umount certain dirs sometimes, 19:31:03 #so we can use it instead of manually doing all thi= s. 19:31:03 echo "Killing all mock orphan processes, if any." 19:31:03 $my_mock \ 19:31:03 --orphanskill \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to kill orphans on $chroot= =2E" 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 19:31:03 mock_root=3D"$(\ 19:31:03 grep \ 19:31:03 -Po "(?<=3Dconfig_opts\['root'\] =3D ')[^']= *" \ 19:31:03 "$mock_conf_file" \ 19:31:03 )" || : 19:31:03 [[ "$mock_root" ]] || continue 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_= root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $chr= oot. Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 done 19:31:03 19:31:03 # Clean any leftover chroot from other jobs 19:31:03 for mock_root in /var/lib/mock/*; do 19:31:03 this_chroot_failed=3Dfalse 19:31:03 mounts=3D($(mount | awk '{print $3}' | grep "$mock_= root")) || : 19:31:03 if [[ "$mounts" ]]; then 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ 19:31:03 "Trying to umount." 19:31:03 fi 19:31:03 for mount in "${mounts[@]}"; do 19:31:03 sudo umount "$mount" \ 19:31:03 || { 19:31:03 echo "ERROR: Failed to umount $mount." 19:31:03 failed=3Dtrue 19:31:03 this_chroot_failed=3Dtrue 19:31:03 } 19:31:03 done 19:31:03 if ! $this_chroot_failed; then 19:31:03 sudo rm -rf "$mock_root" 19:31:03 fi 19:31:03 done 19:31:03 19:31:03 if $failed; then 19:31:03 echo "Aborting." 19:31:03 exit 1 19:31:03 fi 19:31:03 19:31:03 # remove mock system cache, we will setup proxies to do= the caching and this 19:31:03 # takes lots of space between runs 19:31:03 shopt -u nullglob 19:31:03 sudo rm -Rf /var/cache/mock/* 19:31:03 19:31:03 # restore the permissions in the working dir, as someti= mes it leaves files 19:31:03 # owned by root and then the 'cleanup workspace' from j= enkins job fails to 19:31:03 # clean and breaks the jobs 19:31:03 sudo chown -R "$USER" "$WORKSPACE" 19:31:03 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe /tmp/hudson3681642653853563663.sh 19:31:03 + echo shell-scripts/mock_cleanup.sh 19:31:03 shell-scripts/mock_cleanup.sh 19:31:03 + shopt -s nullglob 19:31:03 + WORKSPACE=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_= 64 19:31:03 + cat 19:31:03 ____________________________________________________________________=
19:31:03 ####################################################################= ### 19:31:03 # # 19:31:03 # CLEANUP # 19:31:03 # # 19:31:03 ####################################################################= ### 19:31:03 + logs=3D(./*log ./*/logs) 19:31:03 + [[ -n ./vdsm/logs ]] 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs 19:31:03 ./vdsm/logs/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr= =2Elog 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log 19:31:03 + rm -rf ./vdsm/logs 19:31:03 + failed=3Dfalse 19:31:03 + mock_confs=3D("$WORKSPACE"/*/mocker*) 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' 19:31:03 + [[ -n
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7.cfg
]] 19:31:03 + echo 'Cleaning up mock ' 19:31:03 Cleaning up mock 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7.cfg 19:31:03 + mock_root=3Dmocker-epel-7-x86_64.el7 19:31:03 + my_mock=3D/usr/bin/mock 19:31:03 + my_mock+=3D'
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64/vdsm'
19:31:03 + my_mock+=3D' --root=3Dmocker-epel-7-x86_64.el7' 19:31:03 + my_mock+=3D'
--resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64'
19:31:03 + echo 'Killing all mock orphan processes, if any.' 19:31:03 Killing all mock orphan processes, if any. 19:31:03 + /usr/bin/mock
--configdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el7-x8= 6_64/vdsm
--root=3Dmocker-epel-7-x86_64.el7 --resultdir=3D/home/jenkins/workspace/vdsm_master_check-patch-el= 7-x86_64 --orphanskill 19:31:04 WARNING: Could not find required logging config file:
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logg= ing.ini.
Using default... 19:31:04 INFO: mock.py version 1.2.14 starting (python version = =3D 3.4.3)... 19:31:04 Start: init plugins 19:31:04 INFO: selinux enabled 19:31:04 Finish: init plugins 19:31:04 Start: run 19:31:04 Finish: run 19:31:04 ++ grep -Po '(?<=3Dconfig_opts\['\''root'\''\] =3D '\''= )[^'\'']*'
19:31:04 + mock_root=3Depel-7-x86_64-46ef12ce4362729a0f4c411e00e=
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mock= er-epel-7-x86_64.el7.cfg dd8fc
19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc = ]] 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) 19:31:04 ++ mount 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + for mock_root in '/var/lib/mock/*' 19:31:04 + this_chroot_failed=3Dfalse 19:31:04 + mounts=3D($(mount | awk '{print $3}' | grep "$mock_ro= ot")) 19:31:04 ++ mount 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 ++ awk '{print $3}' 19:31:04 + : 19:31:04 + [[ -n '' ]] 19:31:04 + false 19:31:04 + sudo rm -rf /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + false 19:31:04 + shopt -u nullglob 19:31:04 + sudo rm -Rf /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc 19:31:04 + sudo chown -R jenkins /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 19:31:04 POST BUILD TASK : SUCCESS 19:31:04 END OF POST BUILD TASK : 0 19:31:04 Archiving artifacts 19:31:04 Build step 'Groovy Postbuild' marked build as failure 19:31:04 Started calculate disk usage of build 19:31:04 Finished Calculation of disk usage of build in 0 seconds 19:31:04 Started calculate disk usage of workspace 19:31:04 Finished Calculation of disk usage of workspace in 0 se= conds 19:31:04 Finished: FAILURE
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
=20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --t5NgoZwlhlUmGr82 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXOuUyAAoJEEBxx+HSYmnDQAQH/Ax1p+bYMX9BXpNUewDwHsbm cfDpl2FOYO37OpFCWpnnrWzoZByq1pgpaaub43efKodCbfTzgX1B7Yo9nw/zI1eA unxBBsnNHHt9K/4dR2ybgCKTR10tlb7ejCaODwIYv7SWGnepFZGlzKQxr7OTQP2I UrjBmU58Z1+hW/3YsGkHqB/ZQcD0lcYzBigtwoj0Ce3QZn43508B7uBVZgCVKXmM Vt04CtYQEB79aCdNCU/iYBUxg11bQCjjJntXaOXfFlcYXYQ7DTIxnZhXoXPFHyCj gkWBTZEbyCoZfjI1868l2jMyR3hoZWPEdrK3v9SkF5CJo8EyUxzeWE3rbSTB9R8= =6ebz -----END PGP SIGNATURE----- --t5NgoZwlhlUmGr82--
Hi David, I'll write this as an AI to discuss in the next rhev-ci meeting Thanks Gil On Tue, May 17, 2016 at 12:32 PM, dcaro <dcaro@redhat.com> wrote:
On 05/17 10:36, Gil Shinar wrote:
Hi,
I have tried to debug this issue yesterday with the help of Barak. I don't think that the problem is history as I saw another failure like this yesterday (it happens from time to time). Problem is the slave it builds on being taken by another build and cleans the workspace. I
look for logs under the workspace and they missing probably because a newer build is being built on the same slave.
How did you find out about that? That has happened before when the same slave was added twice on jenkins with different names, but the behavior was way different (you'd get things
On 05/17 11:20, dcaro wrote: tried to like the
dir does not exist and such, not errors when running yum inside mock)
An example of yum failure:
http://jenkins.ovirt.org/view/Master%20branch%20per%20project/view/vdsm/job/...
You can see there the log in the console:
03:00:58 Start: yum install 03:01:07 ERROR: Command failed. See logs for output. 03:01:07 # /usr/bin/yum-deprecated --installroot /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts
Then, in the archived artifacts logs.tgz, under vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log, you can see the error log:
DEBUG util.py:474: Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/', '--releasever', '7', 'install', '@buildsys-build', '--setopt=tsflags=nocontexts'] with env {'SHELL': '/bin/bash', 'CCACHE_DIR': '/tmp/ccache', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'CCACHE_UMASK': '002', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'HOSTNAME': 'mock', 'LD_PRELOAD': '/tmp/tmpxqactgv2/$LIB/nosync.so'} and shell False DEBUG util.py:399: Yum command has been deprecated, use dnf instead. DEBUG util.py:399: See 'man dnf' and 'man yum2dnf' for more information. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/bbf11fe5f13b4c6...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/4e6f7cf18ae8bc3...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:399: Trying other mirror. DEBUG util.py:399: One of the configured repositories failed ("Custom epel"), DEBUG util.py:399: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:399: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:399: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:399: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:399: upstream. This is most often useful if you are using a newer DEBUG util.py:399: distribution release than is supported by the repository (and the DEBUG util.py:399: packages for the previous distribution release still work). DEBUG util.py:399: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:399: just ignore the repository until you permanently enable it again or use DEBUG util.py:399: --enablerepo for temporary usage: DEBUG util.py:399: yum-config-manager --disable epel DEBUG util.py:399: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:399: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:399: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:399: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:399: compromise: DEBUG util.py:399: yum-config-manager --save --setopt=epel.skip_if_unavailable=true DEBUG util.py:399: failure: repodata/ff94d6a6fd8803f1ba27ab562b8e99c2b5f7f4ffa5d49b97689f3df6ca57e367-primary.sqlite.xz from epel: [Errno 256] No more mirrors to try. DEBUG util.py:399: http://download.fedoraproject.org/pub/epel/7/x86_64/repodata/ff94d6a6fd8803f...: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:521: Child return code was: 1
That usually means that the mirrors were in the middle of an update, and the md5sums in the metadata were not consistent at that point it time (happens from time to time).
We could try to minimize that issue handling a mirror ourselves, though that would mean another service to maintain (with it's own potential issues, like not being updated and such, but that's another subject)
Gil
On Tue, May 17, 2016 at 8:52 AM, Eyal Edri <eedri@redhat.com> wrote:
Can we increase the history for builds on that job? We should be
debug jobs at least a week back. Artifacts are not needed On May 16, 2016 11:42 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/15 23:23, Nir Soffer wrote:
Another instance:
19:38:15 Start: yum install 19:38:22 ERROR: Command failed. See logs for output.
That means that there was an issue with the yum repos, most common causes are:
* The repos were actually down (mirrors fail once a day usually, while syncing the rpms, though the cause is just a guess) * Repoproxy (that we use to cache rpms) was overloaded
Both jobs histories are now gone, in order to be able to debug those issues (for the next time), try looking into the mock logs, under the 'logs.tgz' file that's archived in the job, then under the path: /./vdsm/logs/mocker-epel-7-x86_64.el7.init/ or /./vdsm/logs/mocker-epel-7-x86_64.el7.install_packages/
Usually in the log named 'root.log' (check which one has more size).
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1213/console
On Sun, May 15, 2016 at 11:21 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
>
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1209/console
> > 19:30:56 Start: yum install > 19:31:03 ERROR: Command failed. See logs for output. > > Including the to make sure it will not disapper > ---- > > 19:30:48 Triggered by Gerrit: https://gerrit.ovirt.org/56550 > 19:30:48 Building remotely on fc23-vm07.phx.ovirt.org (fc23 nested) in > workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > 19:30:48 > git rev-parse --is-inside-work-tree # timeout=10 > 19:30:49 Fetching changes from the remote Git repository > 19:30:49 > git config remote.origin.url > git://gerrit.ovirt.org/vdsm.git # timeout=10 > 19:30:49 Cleaning workspace > 19:30:49 > git rev-parse --verify HEAD # timeout=10 > 19:30:49 Resetting working tree > 19:30:49 > git reset --hard # timeout=10 > 19:30:49 > git clean -fdx # timeout=10 > 19:30:49 Pruning obsolete local branches > 19:30:49 Fetching upstream changes from git:// gerrit.ovirt.org/vdsm.git > 19:30:49 > git --version # timeout=10 > 19:30:49 > git -c core.askpass=true fetch --tags --progress > git://gerrit.ovirt.org/vdsm.git refs/changes/50/56550/6 --prune > 19:30:52 > git rev-parse > 0940208483f3a21261eb5d725348e65c23becdc0^{commit} # timeout=10 > 19:30:52 Checking out Revision 0940208483f3a21261eb5d725348e65c23becdc0 (master) > 19:30:52 > git config core.sparsecheckout # timeout=10 > 19:30:52 > git checkout -f 0940208483f3a21261eb5d725348e65c23becdc0 > 19:30:52 > git rev-parse FETCH_HEAD^{commit} # timeout=10 > 19:30:52 > git rev-list a09c577837b939096f97108fdbbcafe5980d4a0d # timeout=10 > 19:30:52 > git branch -a # timeout=10 > 19:30:52 > git rev-parse remotes/origin/master^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.1^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.2^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.3.0^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.4^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5-gluster^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.0^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.2^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.4^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.5.6^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.0^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.1^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.2^{commit} # timeout=10 > 19:30:52 > git rev-parse remotes/origin/ovirt-3.6.3^{commit} # timeout=10 > 19:30:53 > git rev-parse --is-inside-work-tree # timeout=10 > 19:30:53 Fetching changes from the remote Git repository > 19:30:53 > git config remote.origin.url > git://gerrit.ovirt.org/jenkins.git # timeout=10 > 19:30:53 Cleaning workspace > 19:30:53 > git rev-parse --verify HEAD # timeout=10 > 19:30:53 Resetting working tree > 19:30:53 > git reset --hard # timeout=10 > 19:30:53 > git clean -fdx # timeout=10 > 19:30:53 Pruning obsolete local branches > 19:30:53 Fetching upstream changes from git:// gerrit.ovirt.org/jenkins.git > 19:30:53 > git --version # timeout=10 > 19:30:53 > git -c core.askpass=true fetch --tags --progress > git://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > --prune > 19:30:53 > git rev-parse origin/master^{commit} # timeout=10 > 19:30:53 Checking out Revision > 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 (origin/master) > 19:30:53 > git config core.sparsecheckout # timeout=10 > 19:30:53 > git checkout -f 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 > 19:30:53 > git rev-list 68a517c3ccc1fe58a6c52646e3a99f3ab22ddab9 # timeout=10 > 19:30:53 > git branch -a # timeout=10 > 19:30:53 > git rev-parse remotes/origin/master^{commit} # timeout=10 > 19:30:53 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > /tmp/hudson8265618520208925180.sh > 19:30:53 shell-scripts/cleanup_slave.sh > 19:30:53 ################################################################### > 19:30:53 # Cleaning up slave # > 19:30:53 ################################################################### > 19:30:53 Filesystem Size Used Avail > Use% Mounted on > 19:30:53 devtmpfs 3.9G 0 3.9G 0% /dev > 19:30:53 tmpfs 3.9G 0 3.9G > 0% /dev/shm > 19:30:53 tmpfs 3.9G 2.1M 3.9G 1% /run > 19:30:53 tmpfs 3.9G 0 3.9G > 0% /sys/fs/cgroup > 19:30:53 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > 19:30:53 tmpfs 3.9G 244K 3.9G 1% /tmp > 19:30:53 /dev/vda1 988M 94M 828M 11% /boot > 19:30:53 tmpfs 799M 0 799M > 0% /run/user/1000 > 19:30:53
> 19:30:53 Cleaning up postgres databases > 19:30:53 Postgres installation not found, skipping > 19:30:53 Cleaning up journal logs (if any) > 19:30:53 Redirecting to /bin/systemctl restart systemd-journald.service > 19:30:54 Cleaning up /var/tmp > 19:30:54 done > 19:30:54 Emptying some common logs > 19:30:54 /var/log/wtmp > 19:30:54 Done > 19:30:54 Making sure there are no device mappings... > 19:30:54 Removing the used loop devices... > 19:30:54 Redirecting to /bin/systemctl restart libvirtd.service > 19:30:54
> 19:30:54 Filesystem Size Used Avail > Use% Mounted on > 19:30:54 devtmpfs 3.9G 0 3.9G 0% /dev > 19:30:54 tmpfs 3.9G 0 3.9G > 0% /dev/shm > 19:30:54 tmpfs 3.9G 2.1M 3.9G 1% /run > 19:30:54 tmpfs 3.9G 0 3.9G > 0% /sys/fs/cgroup > 19:30:54 /dev/mapper/fedora--server_fc23--base-root 35G 6.4G 27G 20% / > 19:30:54 tmpfs 3.9G 244K 3.9G 1% /tmp > 19:30:54 /dev/vda1 988M 94M 828M 11% /boot > 19:30:54 tmpfs 799M 0 799M > 0% /run/user/1000 > 19:30:54 ################################################################### > 19:30:54 # Slave cleanup done # > 19:30:54 ################################################################### > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson7354644880288923842.sh > 19:30:54 + echo shell-scripts/global_setup.sh > 19:30:54 shell-scripts/global_setup.sh > 19:30:54 + rm -rf /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > 19:30:54 + mkdir -p > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/tmp > 19:30:54 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson4023353013258685526.sh > 19:30:54 + echo shell-scripts/mock_setup.sh > 19:30:54 shell-scripts/mock_setup.sh > 19:30:54 + shopt -s nullglob > 19:30:54 + [[ '' == \t\r\u\e ]] > 19:30:54 + failed=false > 19:30:54 +
mock_dir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/mock
> 19:30:54 + chroots=("$WORKSPACE"/mock/*) > 19:30:54 + false > 19:30:54 + sudo rm -Rf mock mock-cache exported-artifacts > 19:30:54 + mkdir -p mock exported-artifacts > 19:30:54 + chgrp mock mock > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/exported-artifacts
> 19:30:54 + chmod g+rws mock > 19:30:54 + sudo touch > 19:30:54 + : > 19:30:54 + sudo yum clean all > 19:30:54 Yum command has been deprecated, redirecting to '/usr/bin/dnf > clean all'. > 19:30:54 See 'man dnf' and 'man yum2dnf' for more information. > 19:30:54 To transfer transaction metadata from yum to DNF, run: > 19:30:54 'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate' > 19:30:54 > 19:30:55 Cleaning repos: updates ci-tools fedora > 19:30:55 Cleaning up Everything > 19:30:55 + exit 0 > 19:30:55 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -e > /tmp/hudson4405890361943185034.sh > 19:30:55 shell-scripts/standard_check_patch.sh > 19:30:55 ########################################################## > 19:30:55 ########################################################## > 19:30:55 ## Sun May 15 15:30:55 EDT 2016 Running env: el7:epel-7-x86_64 > 19:30:55 ########################################################## > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:30:55 @@ Sun May 15 15:30:55 EDT 2016 Running chroot for
able to script:
> automation/check-patch.sh > 19:30:55 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:30:55 Using base mock conf >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64.cfg
> 19:30:55 Using proxified config >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/jenkins/mock_configs/epel-7-x86_64_proxied.cfg
> 19:30:55 Generating temporary mock conf >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7
> 19:30:55 Skipping mount points > 19:30:55 Using temp chroot = epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:30:55 Adding repo ovirt-snapshot -> > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7 > 19:30:55 Adding repo ovirt-snapshot-static -> > http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/el7 > 19:30:55 Adding repo centos-glusterfs -> > http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.7/ > 19:30:55 Adding repo centos-ovirt40-candidate -> > http://cbs.centos.org/repos/virt7-ovirt-40-candidate/x86_64/os/ > 19:30:55 Adding repo lago -> > http://resources.ovirt.org/repos/lago/stable/0.0/rpm/el7 > 19:30:55 Adding repo epel -> http://download.fedoraproject.org/pub/epel/7/x86_64 > 19:30:55 ========== Initializing chroot > 19:30:55 mock \ > 19:30:55 >
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> \ > 19:30:55 --root="mocker-epel-7-x86_64.el7" \ > 19:30:55 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ > 19:30:55 --init > 19:30:55 WARNING: Could not find required logging config file: >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> Using default... > 19:30:55 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... > 19:30:55 Start: init plugins > 19:30:55 INFO: selinux enabled > 19:30:55 Finish: init plugins > 19:30:55 Start: run > 19:30:55 Start: clean chroot > 19:30:55 Finish: clean chroot > 19:30:55 Start: chroot init > 19:30:55 INFO: calling preinit hooks > 19:30:55 INFO: enabled root cache > 19:30:55 INFO: enabled yum cache > 19:30:55 Start: cleaning yum metadata > 19:30:55 Finish: cleaning yum metadata > 19:30:55 INFO: enabled ccache > 19:30:55 Mock Version: 1.2.14 > 19:30:55 INFO: Mock Version: 1.2.14 > 19:30:56 Start: yum install > 19:31:03 ERROR: Command failed. See logs for output. > 19:31:03 # /usr/bin/yum-deprecated --installroot > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc/root/ > --releasever 7 install @buildsys-build --setopt=tsflags=nocontexts > 19:31:03 Init took 8 seconds > 19:31:03 ============================ > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:31:03 @@ Sun May 15 15:31:03 EDT 2016 automation/check-patch.sh > chroot finished > 19:31:03 @@ took 8 seconds > 19:31:03 @@ rc = 1 > 19:31:03 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > 19:31:03 ########################################################## > 19:31:03 ## Sun May 15 15:31:03 EDT 2016 Finished env: el7:epel-7-x86_64 > 19:31:03 ## took 8 seconds > 19:31:03 ## rc = 1 > 19:31:03 ########################################################## > 19:31:03 ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > 19:31:03 ##! Last 20 log enties: > logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > 19:31:03 ##! > 19:31:03 ========== Initializing chroot > 19:31:03 mock \ > 19:31:03 >
--configdir="/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm"
> \ > 19:31:03 --root="mocker-epel-7-x86_64.el7" \ > 19:31:03 --resultdir="logs/mocker-epel-7-x86_64.el7.init" \ > 19:31:03 --init > 19:31:03 Init took 8 seconds > 19:31:03 ============================ > 19:31:03 ##! > 19:31:03 ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 19:31:03 ##!######################################################## > 19:31:03 Build step 'Execute shell' marked build as failure > 19:31:03 Performing Post build task... > 19:31:03 Match found for :.* : True > 19:31:03 Logical operation result is TRUE > 19:31:03 Running script : #!/bin/bash -xe > 19:31:03 echo "shell-scripts/mock_cleanup.sh" > 19:31:03 > 19:31:03 shopt -s nullglob > 19:31:03 > 19:31:03 > 19:31:03 WORKSPACE="$PWD" > 19:31:03 > 19:31:03 # Make clear this is the cleanup, helps reading the jenkins logs > 19:31:03 cat <<EOC > 19:31:03
> 19:31:03
#######################################################################
> 19:31:03 # # > 19:31:03 # CLEANUP # > 19:31:03 # # > 19:31:03
> 19:31:03 EOC > 19:31:03 > 19:31:03 > 19:31:03 # Archive the logs, we want them anyway > 19:31:03 logs=( > 19:31:03 ./*log > 19:31:03 ./*/logs > 19:31:03 ) > 19:31:03 if [[ "$logs" ]]; then > 19:31:03 tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > 19:31:03 rm -rf "${logs[@]}" > 19:31:03 fi > 19:31:03 > 19:31:03 # stop any processes running inside the chroot > 19:31:03 failed=false > 19:31:03 mock_confs=("$WORKSPACE"/*/mocker*) > 19:31:03 # Clean current jobs mockroot if any > 19:31:03 for mock_conf_file in "${mock_confs[@]}"; do > 19:31:03 [[ "$mock_conf_file" ]] || continue > 19:31:03 echo "Cleaning up mock $mock_conf" > 19:31:03 mock_root="${mock_conf_file##*/}" > 19:31:03 mock_root="${mock_root%.*}" > 19:31:03 my_mock="/usr/bin/mock" > 19:31:03 my_mock+=" --configdir=${mock_conf_file%/*}" > 19:31:03 my_mock+=" --root=${mock_root}" > 19:31:03 my_mock+=" --resultdir=$WORKSPACE" > 19:31:03 > 19:31:03 #TODO: investigate why mock --clean fails to umount > certain dirs sometimes, > 19:31:03 #so we can use it instead of manually doing all
####################################################################### this.
> 19:31:03 echo "Killing all mock orphan processes, if any." > 19:31:03 $my_mock \ > 19:31:03 --orphanskill \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to kill orphans on $chroot." > 19:31:03 failed=true > 19:31:03 } > 19:31:03 > 19:31:03 mock_root="$(\ > 19:31:03 grep \ > 19:31:03 -Po "(?<=config_opts\['root'\] = ')[^']*" \ > 19:31:03 "$mock_conf_file" \ > 19:31:03 )" || : > 19:31:03 [[ "$mock_root" ]] || continue > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > 19:31:03 if [[ "$mounts" ]]; then > 19:31:03 echo "Found mounted dirs inside the chroot $chroot. > Trying to umount." > 19:31:03 fi > 19:31:03 for mount in "${mounts[@]}"; do > 19:31:03 sudo umount "$mount" \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to umount $mount." > 19:31:03 failed=true > 19:31:03 } > 19:31:03 done > 19:31:03 done > 19:31:03 > 19:31:03 # Clean any leftover chroot from other jobs > 19:31:03 for mock_root in /var/lib/mock/*; do > 19:31:03 this_chroot_failed=false > 19:31:03 mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > 19:31:03 if [[ "$mounts" ]]; then > 19:31:03 echo "Found mounted dirs inside the chroot $mock_root." \ > 19:31:03 "Trying to umount." > 19:31:03 fi > 19:31:03 for mount in "${mounts[@]}"; do > 19:31:03 sudo umount "$mount" \ > 19:31:03 || { > 19:31:03 echo "ERROR: Failed to umount $mount." > 19:31:03 failed=true > 19:31:03 this_chroot_failed=true > 19:31:03 } > 19:31:03 done > 19:31:03 if ! $this_chroot_failed; then > 19:31:03 sudo rm -rf "$mock_root" > 19:31:03 fi > 19:31:03 done > 19:31:03 > 19:31:03 if $failed; then > 19:31:03 echo "Aborting." > 19:31:03 exit 1 > 19:31:03 fi > 19:31:03 > 19:31:03 # remove mock system cache, we will setup proxies to do the > caching and this > 19:31:03 # takes lots of space between runs > 19:31:03 shopt -u nullglob > 19:31:03 sudo rm -Rf /var/cache/mock/* > 19:31:03 > 19:31:03 # restore the permissions in the working dir, as sometimes it > leaves files > 19:31:03 # owned by root and then the 'cleanup workspace' from jenkins > job fails to > 19:31:03 # clean and breaks the jobs > 19:31:03 sudo chown -R "$USER" "$WORKSPACE" > 19:31:03 > 19:31:03 [vdsm_master_check-patch-el7-x86_64] $ /bin/bash -xe > /tmp/hudson3681642653853563663.sh > 19:31:03 + echo shell-scripts/mock_cleanup.sh > 19:31:03 shell-scripts/mock_cleanup.sh > 19:31:03 + shopt -s nullglob > 19:31:03 + WORKSPACE=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > 19:31:03 + cat > 19:31:03
> 19:31:03
#######################################################################
> 19:31:03 # # > 19:31:03 # CLEANUP # > 19:31:03 # # > 19:31:03
#######################################################################
> 19:31:03 + logs=(./*log ./*/logs) > 19:31:03 + [[ -n ./vdsm/logs ]] > 19:31:03 + tar cvzf exported-artifacts/logs.tgz ./vdsm/logs > 19:31:03 ./vdsm/logs/ > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/ > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/root.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/state.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/stdout_stderr.log > 19:31:03 ./vdsm/logs/mocker-epel-7-x86_64.el7.init/build.log > 19:31:03 + rm -rf ./vdsm/logs > 19:31:03 + failed=false > 19:31:03 + mock_confs=("$WORKSPACE"/*/mocker*) > 19:31:03 + for mock_conf_file in '"${mock_confs[@]}"' > 19:31:03 + [[ -n >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> ]] > 19:31:03 + echo 'Cleaning up mock ' > 19:31:03 Cleaning up mock > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7.cfg > 19:31:03 + mock_root=mocker-epel-7-x86_64.el7 > 19:31:03 + my_mock=/usr/bin/mock > 19:31:03 + my_mock+=' >
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'
> 19:31:03 + my_mock+=' --root=mocker-epel-7-x86_64.el7' > 19:31:03 + my_mock+=' >
--resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64'
> 19:31:03 + echo 'Killing all mock orphan processes, if any.' > 19:31:03 Killing all mock orphan processes, if any. > 19:31:03 + /usr/bin/mock >
--configdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm
> --root=mocker-epel-7-x86_64.el7 > --resultdir=/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > --orphanskill > 19:31:04 WARNING: Could not find required logging config file: >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/logging.ini.
> Using default... > 19:31:04 INFO: mock.py version 1.2.14 starting (python version = 3.4.3)... > 19:31:04 Start: init plugins > 19:31:04 INFO: selinux enabled > 19:31:04 Finish: init plugins > 19:31:04 Start: run > 19:31:04 Finish: run > 19:31:04 ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' >
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/mocker-epel-7-x86_64.el7.cfg
> 19:31:04 + mock_root=epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + [[ -n epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc ]] > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > 19:31:04 ++ mount > 19:31:04 ++ grep epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 ++ awk '{print $3}' > 19:31:04 + : > 19:31:04 + [[ -n '' ]] > 19:31:04 + for mock_root in '/var/lib/mock/*' > 19:31:04 + this_chroot_failed=false > 19:31:04 + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > 19:31:04 ++ mount > 19:31:04 ++ grep /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 ++ awk '{print $3}' > 19:31:04 + : > 19:31:04 + [[ -n '' ]] > 19:31:04 + false > 19:31:04 + sudo rm -rf > /var/lib/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + false > 19:31:04 + shopt -u nullglob > 19:31:04 + sudo rm -Rf > /var/cache/mock/epel-7-x86_64-46ef12ce4362729a0f4c411e00edd8fc > 19:31:04 + sudo chown -R jenkins > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > 19:31:04 POST BUILD TASK : SUCCESS > 19:31:04 END OF POST BUILD TASK : 0 > 19:31:04 Archiving artifacts > 19:31:04 Build step 'Groovy Postbuild' marked build as failure > 19:31:04 Started calculate disk usage of build > 19:31:04 Finished Calculation of disk usage of build in 0 seconds > 19:31:04 Started calculate disk usage of workspace > 19:31:04 Finished Calculation of disk usage of workspace in 0 seconds > 19:31:04 Finished: FAILURE _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
participants (5)
-
David Caro -
dcaro
-
Eyal Edri -
Gil Shinar -
Nir Soffer